Writing Tests

1. Write your plan#

The first thing you should do when writing new tests is add/edit the Plan file. An AG350 Plan is a json file that details how and in what environment a test suite should be run on the AG350 Athena build system. In other words, the Plan is the recipe that tells AG350's build system how to execute the test suite.

The crux of the Plan is the "tests" field which specifies what types of test should be run and their respective configurations. Each test has a name, a type, and options (configuration) that depend on the test type chosen. The combination of the Plan and Tests documentation should guide you through setting up the right Plan for your test suite.

The content of the plan should be located in the test suite repository in the .ag350/plan.json file and committed to the working branch of your choosing. Full documentation on the Plan file can be found here.

2. Write your tests#

Once the Plan is settled you're ready to write your tests. Depending on the type of test you may have to write multiple files or none at all! For declarative tests, all necessary information is specified in the Plan and no further action is necessary. For IO tests, the input and output files referenced the Plan must be filled out. For library tests, the source code of the tests themselves must be written and the format of which differs by language. For a good starting point reference our Examples Repository which includes base examples for all test types!

3. Run checks#

Safety Checks

Once you have committed and pushed your plan and tests you should be able to run safety checks to check the validity of the tests. Safety checks verify that your solution and tests are working properly with each other. We run a proofread to ensure that your tests have a valid plan.json running and a build to check that your solution passes all of your tests. Safety checks are not necessary to link and deploy your tests to students, but are suggested to make sure everything is working as expected before available to students.

4. Link to deploy!#

Link Tests

Once you are satisfied with the state of the tests, link them in using teh UI. Once linked the new tests will be run against any incoming submissions.