Test Types

This document is a reference for the supported test types in the V0.1.0 AG350 Plan. In AG350 each test has a name and a type, as well as some additional options that help specify how the test should be run. The name of the test is a string used to uniquely identify the test while the type specifies how the test should be run.

Philosophy (Test Types)#

In AG350 there are 4 fundamental ways to test student code: 1. IO 2. Library 3. Script 4. Declarative. Each supported test type falls underneath one of these broader categories.

Input-output ("IO") Testing: In an IO test the student code runs as a process. The student code is provided input via stdin and the output of the process is compared to an expected output for correctness.
Library Testing: Library testing, as the name suggests, treats student code as a library. For this test type test suite code calls student code as a library within the same process and uses the output of the student library to determine correctness.
Script Testing: Script testing gives professors fine-grained control over the test suite execution. Professors write scripts in a variety of supported scripting languages (bash, javascript, etc.) that produce a reliable output format (see Pragma documentation) on standard out that the AG platform can parse.
Declarative Testing: Similar to library testing, declarative testing treats the student code as a library. However, for declarative testing no code is necessary. Rather, inputs and expected outputs of the student function are specified in configuration and the necessary test suite code is completely scaffolded by AG350.

Test Reference#

Table of Contents#

IO
- JavaIO
- PythonIO
Library
- PythonLibrary
Scripting
- JavascriptPragmaScript
- BashPragmaScript
Declarative
- DeclarativePython

IO Tests#

JavaIO#

A JavaIO test runs a student's java code as a process with stdin supplied by the test suite repository. The result of the test is a single assertion with either a PASS or FAIL status depending on whether or not the stdout of the student process matches the expected output specified in the test suite repository. The student code must contain a main function, which will be run during test execution in the build.

Options:

source (string: required): A relative path from the base of the student repository to the java file that should be compiled and run.
className (string: required): The name of the class containing the main method in the student java file.
input (string: required): A relative path from the base of the test suite repository that contains the contents that will be passed to the java process via stdin.
output (string: required): A relative path from the base of the test suite repository that contains the expected stdout from the java process. At the completion of the java process the output of the process will be compared to the expected output on a line-by-line basis. If any of the lines in the student's output differs from the expected output then the output of the test will be a single FAIL assertion. If the output of the process does indeed match the expected output then the output of the test will be a single PASS assertion.

Example:

{
    "type": "JavaIO",
    "name": "TestMultiplication",
    "options": {
        "source": "run.java",
        "className": "Run",
        "input": "input.txt",
        "expected": "output.txt"
    }
}

PythonIO#

A PythonIO test runs a student's python code as a process with stdin supplied by the test suite repository. The result of the test is a single assertion with either a PASS or FAIL status depending on whether or not the stdout of the student process matches the expected output specified in the test suite repository.

Options:

source (string: required): A relative path from the base of the student repository to the python file that should be run.
input (string: required): A relative path from the base of the test suite repository that contains the contents that will be passed to the python process via stdin.
output (string: required): A relative path from the base of the test suite repository that contains the expected stdout from the python process. At the completion of the python process the output of the process will be compared to the expected output on a line-by-line basis. If any of the lines in the student's output differs from the expected output then the output of the test will be a single FAIL assertion. If the output of the process does indeed match the expected output then the output of the test will be a single PASS assertion.

Example:

The following test object indicates that a build will run the run.py script at the base of the student repository with the contents of input.txt in the test suite repository piped to stdin. The output of running run.py will be compared to the contents of output.txt at the base of the test suite repository.

{
    "type": "PythonIO",
    "name": "TestMultiplication",
    "options": {
        "source": "run.py",
        "input": "input.txt",
        "expected": "output.txt"
    }
}

Library Tests#

PythonLibrary#

A PythonLibrary test runs a test file written in python.

Options:

source (string: required): A relative path from the base of the test suite repository to the python test file that should be run. In the test suite file any method starting with "test" will be run. For each method that is run an assertion will be added to the test result - if the method runs to completion a _PASS assertion will be added, if the method throws a FAIL assertion will be added. Therefore, its recommended that writers of library tests invoke the student library in each method and throw if the output or expected state is not what is expected.

Example:

During the build an method starting with "test" in the test_negatives.py file at the base of the test_suite repository will be run and correspond to either a _PASS or FAIL assertion in the test results.

{
    "type": "PythonLibrary",
    "name": "MyPyTest",
    "options": {
        "source": "test_negatives.py"
    }
}

Scripting Tests#

JavascriptPragmaScript#

A JavascriptPragmaScript test executes a javascript script in the test suite repository. The script should produce Pragma format on stdout. The pragmas produced on standard out are parsed by AG to determine the assertions, PASS or FAIL, associated with the test. The javascript script will be executed using the version of node associated with the images of the plan.

Options:

source (string: required): The name of the javascript file at the base of the test suite repository that will be run.

Example:

{
    "name": "HelloWorld",
    "type": "JavascriptPragmaScript",
    "options": {
        "source": "./helloWorldScript.js"
    }
}

BashPragmaScript#

A BashPragmaScript test executes a bash script in the test suite repository. The script should produce Pragma format on stdout. The pragmas produced on standard out are parsed by AG to determine the assertions, PASS or FAIL, associated with the test.

Options:

source (string: required): The name of the bash file at the base of the test suite repository that will be run.

Example:

{
    "name": "HelloWorld",
    "type": "BashPragmaScript",
    "options": {
        "source": "./helloWorldScript.sh"
    }
}

Declarative Tests#

DeclarativePython#

A DeclarativePython test scaffolds code to test a specified python method written by the student.

Options:

source (string: required): The name of the python file at the base of the student repository that contains the method that should be run (no extension included).
method (string: required): The name of the method written by the student.
signature (string: required): The signature of the method written by the student. This should be written in the format (arg1Type arg1Name, arg2Type arg2Name, ...) => outputType. The following types are currently supported: int, int[], string, string[], float.
assertions (object[]: required): An array of object that specify the inputs and expected outputs that should be used to test the student library. Each object corresponds to one trial of the student method and should contain an "inputs" and "output" field. The inputs field is an ordered array where each entry corresponds the arguments of the method, and the output field is the expected output for those inputs. As the name suggests the results of the test corresponds directly to an entry in the assertion array.

Example:

The following test will result in a single PASS or FAIL assertion depending on whether or not the falling method in the Gravity.py file in the base of the student repository produces 59.1 when given the inputs time=3, velo=5.

{
    "name": "GravityTest",
    "type": "DeclarativePython",
    "options": {
        "source": "Gravity",
        "method": "falling",
        "signature": "(float time, float velo) => float",
        "assertions": [
            {
                "inputs": [
                    3,
                    5
                ],
                "output": 59.1
            }
        ]
    }
}