Assurance cases are used not only during system development but also during operation. For many systems, Safety Performance Indicators (SPIs) are used in this context. They are used to monitor and report the system safety performance over time. They help to identify whether safety goals are being achieved and provide early warning signals of emerging risks. They are also useful in identifying weaknesses in the safety management system.
Assurance cases should not ignore SPIs, as they provide essential information about the system safety during its operation. However the nature of these indicators is different from regular assurance case evidence. SPI values change over time, so the argument must be able to handle this variability. We need dynamic assurance cases that can react to changes of the reported SPI.
Assurance case process to incorporate SPI data
There are three steps of this process that can be performed automatically:
- Transfer the SPI data. The goal is to connect to the SPI measurement system and transfer SPI data into evidence references in the assurance case. SPI data is maintained as dynamic evidence within the argument. This means that data in the argument can be updated daily or more frequently when needed. We use a simple generic data format for SPI values in PREMIS that can represent any type of values, including textual and numerical information.
- Interpret the SPI data. This is done by updating the base claim based on the supported SPI evidence. PREMIS allows to define conditions on SPI values to determine the resulting claim assessment. There are three possible statuses for SPI indicators: the SPI status can be OK (acceptable risk level), Warning (risk soon) and Alert (at risk).
- Propagate the SPI status information. PREMIS assessment propagation mechanism automatically updates the assessment in the assurance case. Depending on the argument structure, changes of the SPI indicators values may affect the assessment of related claims in the argument, including the top claim. Exceeding the threshold of a single critical SPI indicator may even result in the top claim being rejected.
This process is fully automated when PREMIS arguments are connected to source systems that provide SPI values. This could be a single system integrating all SPIs, or it could be multiple source systems reporting individual metrics. It could be any system that provides network access to its data (an API) or has the ability to transmit the data in any other way, for example by email. We use Jira integrated with PREMIS as the main source of SPI data.
Dynamic evidence, such as SPIs, changes the way arguments are handled. The point is that, most of the time, dynamic evidence works like regular evidence, but it can occasionally turn into counter-evidence. This happens when the indicator’s value indicates that some security requirements are not met. The system’s safety or security argument can only be accepted if the measurement data confirms that the system actually meets the safety and security goals. When a dynamic proof indicates that this is not the case, a response is required, not by changing the argument, as is the case with defeaters, but by changing the value of the evidence – in this case, the SPI indicator. It’s the system operation that needs to be corrected. The argument needs any changes only when the system or the operation process is modified. If the correction involves design changes or the release of a new system version, the relevant parts of the argument should also be updated. Regardless of the associated argumentation changes, it is essential that the dynamic evidence reverts to its accepted state, so that it no longer acts as counter-evidence.
How does can work in practice?
A simple example of a Safety Performance Indicator is the number of open safety issues in Jira. We can have an epic in Jira for collecting safety issues in one place, or a filter to select issues covered by the indicator. Let’s say we have a safety epic and all safety issues should be linked to it as sub-issues. Some issues may not be relevant and we define the SPI as a number of open security issues with major or critical priority. If the indicator value exceeds zero, it should report an ALARM. This can be represented by the following rules in PREMIS:
In some cases we may want to add a WARNING state to be used when the indicator value is close to the limit, but the limit is not exceeded yet.
Depending on the settings of your assurance case the result of the SPI value verification can affect the status of the supported base claim and other claims up the argument structure to the top claim. The diagram below shows an example where the SPI measurement implemented as argument evidence resulted in a WARNING status of the supported base claim, which is then propagated up to the top claim. The top claim can only be accepted when all SPI indicators report the status OK.
Although assurance cases are not intended for tracking safety issues and managing their resolution, they should maintain the current status of system safety. Assurance cases are live artefacts, and one aspect of this involves the use of dynamic evidence, such as performance indicators.
Dynamic evidence, including SPI measurement and the use of rules when an alarm, a warning or the OK status should be reported, can be integrated into the assurance case argument in PREMIS. Contact us for more information about how you can use them in your assurance cases.