Test Pragmas

This document is reference for AG350 Test Pragmas. The AG350 Test Pragma format is a mechanism for communication information about a test such as events, assertions, and additional data. This pragma format can be consumed by a variety of AG350 consumers, most notably the Athena build system.

Pragma Types

@ag350:test:start

Indicates the start of a test.

@ag350:test:end

Indicates the end of a test.

@ag350:assertion:start

Indicates a start of an assertion. A start assertion pragma must be between a test start and test end pragma.

@ag350:assertion:end

Indicates the end of an assertion. A end assertion pragma must be between a test start and test end pragma and after an assertion start pragma.

@ag350:assertion:data

Indicates assertion data. An assertion data pragma must be between an assertion start and assertion end pragma. An assertion data pragma must include a valid json object following the "@ag350:assertion:data" text. This json object must have a name for the assertion (preferably unique within the test) and a status (either "PASS" or "FAIL"). In addition to these required fields this assertion data can include any desired key-value pairs that may be useful information to associate with the assertion.

Ex:

@ag350:assertion:data { "name": "Example assertion", "status": "FAIL", "expected": 1, "value": 2, "time": "2s" }

Pragma Format

Pragmas can be read from any text output or form. When a collection of pragmas are included in an output the output is considered to be in a "pragma format."

The pragma format have the following properties:

1. Each pragma must be on its own line.
2. The pragma format is forgiving - non-pragma data can be included amongst pragmas so long as pragmas are on their own line.

Examples

Correct

1. A test with a single pass assertion.

@ag350:test:start

@ag350:assertion:start

@ag350:assertion:data { "name": "assertion_1", "status": "PASS", "expected": 1, "value": 1 }

@ag350:assertion:end

@ag350:test:end

2. A test with two assertions (one pass, one fail). Notice, the pragma format is forgiving, allowing text that are not pragmas.

@ag350:test:start

Compiling

@ag350:assertion:start

Running the student code

@ag350:assertion:data { "name": "assertion_1", "status": "PASS", "expected": 1, "value": 1 }

@ag350:assertion:end

Running some more...

@ag350:assertion:start

@ag350:assertion:data { "name": "assertion_2", "status": "FAIL", "expected": \3, "value": 2 }

@ag350:assertion:end

Completed!

@ag350:test:end

Incorrect

1. No test start pragma.

@ag350:assertion:start

@ag350:assertion:data { "name": "assertion_1", "status": "PASS", "expected": 1, "value": 1 }

@ag350:assertion:end

@ag350:test:end

2. No test end pragma.

@ag350:test:start

@ag350:assertion:start

@ag350:assertion:data { "name": "assertion_1", "status": "PASS", "expected": 1, "value": 1 }

@ag350:assertion:end

3. Assertion data is not valid json.

@ag350:test:start

@ag350:assertion:start

@ag350:assertion:data value = 1, expected = 2

@ag350:assertion:end

@ag350:test:end

4. No assertion start and end pragma.

@ag350:test:start

@ag350:assertion:data { "name": "assertion_1", "status": "PASS", "expected": 1, "value": 1 }

@ag350:test:end