An assurance case used in a systematic and professional manner can not only be useful for system certification, but also as a communication tool in the system assurance process helping in early detection of problems and decision-making. Its effectiveness stems from its focus on goals and building confidence. However, this can be lost if you don’t work with assurance cases in the right way. It can become a costly and useless artifact. You risk failure during qualification or certification of the system.
We usually advise on how to build a trustworthy argument, but today we’ll take a a different approach and discuss how you can effectively ruin an assurance case. The following list presents eight of the worst practices for working with arguments. There are probably more, but these are definitely damaging. If you recognise any of these in your own work, you should expect your argument to be unusable.
Programmers use the concept of code smells, or poor programming practices. When they notice them in their code, they should perform appropriate refactoring to improve the code structure and quality. Below is a list of practices we can call argumentation smells.
Let’s start with the assurance case goals.
1. Define a very general goal
Why spend time precisely defining the top claim of the argument? Give flexibility with a general definition of the top claim and its context, especially the system environment and conditions of use. The claimed properties of the system don’t need to be measurable to start real work.
In reality, imprecise goals are a recipe for failure. Different stakeholders can interpret them differently. Even if you do a good engineering job and provide high-quality evidence, the overall result will be deemed untrustworthy. The only solution is to precisely define the goals, context, and acceptance criteria from the outset.
2. Develop argument when the system is already completed
Why bother with arguments when you’re busy developing the system? You could simply gather the documentation you’ve created at the end and build the argument around it. Build the system first, then create GSN diagrams.
Assurance cases are about reasoning, not just diagrams. The real goal is to build confidence as early as possible. The required evidence should be identified early to ensure that satisfies the requirements when produced. Later it will be too expensive to produce or fix it. Late argument development can generally lead to some risks being discovered late, when the design is locked, or some relevant evidence will not be available. The only way to avoid this is to start working on the assurance case early on, working iteratively through the phases of the system life cycle and using the argument in the decision-making process to produce the right evidence.
3. Simply decompose claims
Simply decompose claims into other claims without any explanation about the reasoning. No strategies, no justifications. When needed just use simple strategie like “Argue by testing / analysis / process compliance” avoiding analysis if this sufficient and adequate. Another simple strategy you may use is for example “we followed ISO 26262 / DO-178C / IEC 61508, therefore the system is safe”. Use possible simplifications when decomposing the argument.
An unexplained claim decomposition does not constitute an argument. Creating an argument means defining a strategy with explicit inference rules and their justification, and, if necessary, also providing a confidence argument. The reasoning must cover the full scope and context of the claim, and any exceptions, conditions, assumptions, uncertainties, or limitations should be clearly stated. Each step of the reasoning should increase the level of confidence in achieving safety and other required system characteristics. Constantly ask, “Does this really build confidence?” Also, ask experts for verification.
4. Ignore reported assurance or confidence issues
Assign responsibility for the assurance case to someone who has little influence on project decisions. Treat the assurance case as administrative and documentation work for certification purposes. Do not involve decision-makers in this work.
The reality is that if we are to truly reduce risk in a system, an assurance case is a tool that allows us to quickly identify necessary decisions. Therefore, the decision-maker should be responsible for developing the arguments. Typically, projects employ two main roles: one responsible for development and the other for verification and quality control. We can recommend a third management role responsible for assurance to be added to these roles, such as a Lead Assurance Architect, for example. Whatever is the role name, it’s crucial that assurance case outcome should have influence on project decisions. Any assurance case warnings, risks and evidence requests should not be ignored. Any issue arising from assurance cases should result in appropriate actions to ensure the required confidence in the assurance case.
5. Use evidence at hand
When you argue by testing, simply use a test report. Don’t waste time on checking if it’s sufficient to fully support the claim. Use evidence that fits. Trust that the evidence you have is appropriate for the purpose of your argument.
A cheap and misleading approach. The need for evidence stems from the argument, and it is the argument that generates the requirements for evidence. First, look at the argument for the evidence needed, and only then for the documentation available, not the other way around. If evidence doesn’t fully meet the requirements, either try to strengthen or supplement it, or explicitly describe the gaps and deviations. Each piece of evidence should be carefully assessed. This builds trust in the argument.
It’s important to note that claims and evidence operate at different levels of abstraction. Evidence typically represents technical data, such as analyses, modelling or testing, that we aggregate to form claims about the properties of a system. Technical competence is also important to properly interpret the proofs, understanding all constraints and the context. Trustworthy arguments require trustworthy interpretations of evidence.
6. Ignore assumptions
Specify assumptions only when and where it’s convenient. They can be also mentioned in descriptions of claims, not as separate elements. Focus on the main argument, not details such as assumptions. It’s assumed they are true, so it doesn’t matter is they are specified or not.
If assumptions aren’t made explicit, readers may assume stronger system properties than actually hold. The system may appear to be safe or secure for scenarios when, in reality, it’s not. Avoiding the discussion on assumptions during assurance case development may lead to late problem identification. Missing assumptions is like a misleading marketing: someone claims the product is very attractive, but forgets to mention limitations and situations where it doesn’t work.
7. Assume certainty and zero residual risk
Focus on the main line reasoning, without going into details of the coverage limits, confidence levels, or known gaps. Assume everything is fully effective. Say you mitigate all risks and skip the problem if the remaining residual risk. Say you fully testand verify the system. Skip properties that cannot be tested and require other ways of verification.
This is a very naïve approach. No system is completely safe or secure. The purpose of an argument is to explicitly demonstrate the remaining level of uncertainty, the limitations, and whether the residual risk is acceptable. System approval should be a well-informed decision that takes all limitations and uncertainties into account.
8. Don’t invest time in updating the argument
The target is to have the assurance case approved. When it’s achieved, treat it as a formal document rather than a technical artefact that requires updates like all system configuration items.
Assurance cases that are not consistent with the current system version become useless for engineers. The argument has value only when it is updated in line with all system artefacts. Otherwise, it would just become a record of the past. To avoid this, you need an effective configuration and change management process. You should maintain explicit status information on whether a given argument fragment is up to date.
Summary
It’s worth checking whether the issues mentioned above are present in your projects. We must respond quickly to them to avoid damaging our arguments and the effectiveness of the process. Proper application of assurance cases requires appropriate preparation, organizational conditions, and technical skills. We should not only ensure that the GSN diagram looks good, but above all, ensure the quality of the assurance case process.