CL 3: Quantitative Verification

For credibility assessment level 3, the goal is to verify the behavior of a model. A specific test folder (test/behavior) contains the tests for CL3. The model is employed in a co-simulation with a trace file player. The input trace file can either be placed in the test folder (as shown here) or be generated by a python script directly in the CI pipeline (as shown here). During the simulation, an output trace file is generated containing the model output of every simulation time step. This is done by connecting a trace file writer to the model output in the SSP specification of the test. The trace file writer is pre-built in every test environment. The FMU is located at /tmp/tracefile_writer_fmu/osi-trace-file-writer.fmu. After the co-simulation is finished, the GitHub action scans the behavior test folder for file named analyze.py. If it is found, the python script is called with the path to the just generated output trace file as an argument.

All tests in CL 3 are considered behavior tests, since they all verify the implemented behavior of the model. Therefore, every test is located in an individual folder in test/behavior. The individual test folder SHALL follow the naming scheme "xxx_short_description", where xxx is a three digit consecutive number. The folder SHALL contain a system structure definition file (.ssd). In this file, the utilized model input (trace file player or esmini) as well as the trace file writer are specified. The test folder SHALL additionally contain a README.md file, which describes the test system, scenario and pass/fail criterion. This README.md SHALL follow the same template as the integration tests. Other simulation artefacts such as the trace file or scenario to be played as well as the python analysis script are also to be placed in that test folder. Example implementations in the test/behavior folder can be found in the sensor model template repository.

The purpose is to test the proper implementation of the model behavior in terms of its quantifiable requirements. E.g. the sensor is rotated around all 3 axis, and it is evaluated, if the coordinate transformations in the model are correct. Or sensor model is detecting objects in its field of view correctly. This can be achieved in two ways:

  1. The simulation trace file is analyzed by a custom standalone python script.

  2. The simulation trace file is compared to an expectation trace file with a python script.

The general setup of this test level is depicted in the following image and both approaches are described in more detail in the following sections.

Test setup for credibility assessment level 3
Figure 1. Test setup for credibility assessment level 3

CL 3.1 Standalone Trace File Analysis

The OSI trace file generated by the co-simulation is read in by a python script after the simulation has finished. The python script can perform a variety of tests on the trace file, depending on the specific scenario and use-case. For example, the ground truth contained in the input trace file places an object at a certain distance inside the field of view of a sensor. The python script can analyze the sensor model output and check, if the object was detected at the expected distance. Another example would be to place an object outside of the field of view and check, that it was not detected. Furthermore, distribution functions for detections or expected number of detections on an object can be checked.

An example implementation of this test can be found in the sensor model template repository.

CL 3.2 Trace File Comparison

(not yet implemented)
This test is similar to 3.1. But instead of analyzing the simulation trace file with certain criteria, the python script compares the model output to an expectation trace files. The expectation trace file can either contain other simulated data or even measurement data from a real system to compare the model under test to. This comparison is done with appropriate metrics. Required thresholds for these metrics determine the pass/fail criterion of these tests.