72 lines
2.8 KiB
Plaintext
72 lines
2.8 KiB
Plaintext
Litani End-to-End Test Suite
|
|
============================
|
|
|
|
|
|
Each file in the `tests` subdirectory contains a test definition. The `run`
|
|
script imports a single one of those files, and uses the test definition to
|
|
execute a complete Litani run. Once the run completes, the `run` script then
|
|
loads the resulting `run.json` file and checks that it conforms to the test
|
|
definition.
|
|
|
|
Each "test definition" is a Python module that is expected to contain certain
|
|
methods; the `run` script calls into each of these methods in order to set up,
|
|
execute, and check the Litani run. Those methods are as follows:
|
|
|
|
|
|
get_init_args()
|
|
Returns a dict containing command-line arguments to be passed to `litani
|
|
init`. The dict must contain two keys, `args` and `kwargs`. `args` is a
|
|
list of switches (command-line options that don't take an argument),
|
|
while `kwargs` is a dict mapping command-line options to the value(s)
|
|
that they take as an argument. The keys or list items should be equal to
|
|
the name of the command-line switch, without the leading `--`. Values of
|
|
the `kwargs` dict can be a string or a list of strings. If a list of
|
|
strings, these will be joined together with whitespace.
|
|
|
|
For example, to make the run script run
|
|
|
|
`litani init --pools foo:1 bar:2 --project-name baz`
|
|
|
|
this method could return the dict
|
|
|
|
{
|
|
"kwargs": {
|
|
"pools": [ "foo:1", "bar:2" ],
|
|
"project-name": "baz",
|
|
}
|
|
}
|
|
|
|
The `kwargs` dict must not contain "output-symlink",
|
|
"output-directory", or "output-prefix", as the `run` script will
|
|
supply its own value for these.
|
|
|
|
|
|
get_jobs()
|
|
Returns a list of jobs that will be added to Litani. Each job is similar
|
|
to the return of `get_init_args()` above, i.e. it's a dict containing
|
|
command-line arguments. The run script will add each job before
|
|
executing the run. For example, to make the run script run
|
|
|
|
`litani add-job --command true --interleave-stdout-stderr`
|
|
`litani add-job --command false --description "oh no!"`
|
|
|
|
this method should return
|
|
|
|
[{
|
|
"args": [ "interleave-stdout-stderr" ],
|
|
"kwargs": { "command": "true" }
|
|
}, {
|
|
"kwargs": { "command": "false", "description": "oh no!" }
|
|
}]
|
|
|
|
|
|
def get_run_build_args()
|
|
Returns a dict containing the arguments to `litani run-build`, in the
|
|
same format as `get_init_args()` above.
|
|
|
|
|
|
check_run(run: dict)
|
|
Takes a deserialized run.json file and returns True iff the Litani run
|
|
was as expected. This method can check that particular keys of the run
|
|
have certain values, etc.
|