System Qualification Test – SYS.5 in Automotive SPICE^®

Since this process is carried out shortly before delivery, there are close relationships to processes such as Project Management (MAN.3), Configuration Management (SUP.8), Product Release (SPL.2) and of course System Requirements Analysis (SYS.2). 

The following are the most important aspects of System Qualification Test in Automotive SPICE®:  

• You must have a system qualification test strategy. Like all testing and supporting processes, system qualification testing requires the development and definition of a testing strategy. You may have a separate test strategy for each test level, but it is better to develop and coordinate the test strategy across all test levels. This verifies that all requirements are covered and redundancies are avoided. The test strategy should cover the following topics: 

1. the test object in question 
2. the methods for developing test cases and test data (e.g. development of positive/negative tests, equivalence partitioning). 
3. a regression test strategy, which in Automotive SPICE® terminology means that you define how you want to re-test after a bug fix or change request 
4. the test environment  
5. test coverage in relation to the project plan and release plan 
6. entry and exit criteria and test interruption criteria 

Of course, this process has a strong connection to SUP.9 Problem Resolution Management, so you can use either the test strategy or the defect management strategy on how to deal with failed tests. 

Part of the strategy is also to define coverage targets based on the type of delivery. So, here is a strong link to the project plan and the demand plan.

When you document a test strategy, parts of it can be generic and covered in the internal process description and guidelines. Again, Automotive SPICE® does not define how and where you document your test strategy. It can be distributed over several documents.

The test strategy should also cover your approach to test automation. Generally, it can be said that test automation is always better than manual testing, but there may be tests that are too expensive to automate or the test environment does not allow this such as in the case of vehicle maneuvers.

• Selection of test cases. This process also expects a corresponding selection of test cases to be performed for the various tests. The aim and expectation is that for the different deliveries the system is properly tested on the basis of the aforementioned test strategy.  

The idea is that you can have deliveries with different expectations. A possible strategy could be that you have complete coverage of all implemented systems requirements for the important deliveries.  

For smaller deliveries, only the delta of implemented requirements since the last delivery is tested.

For this approach, of course, the right test cases must be selected. 

Another possible situation for test case selection would be the regression test, which covers change requests and/or bug fixes. Here, test cases are selected that cover the change request or bug, and the effects that they can have. That means that dependencies on requirements that can be affected by the change request or bug fix are also tested. 

• Traceability and consistency. This process also requires that you verify traceability between your system test cases and system requirements. Traceability can be established through hyperlinks like in DOORS, through specific traceability tools like Rectify, through traceability matrices or through other manageable means which are supported by your tool landscape. The purpose of traceability is: 

1. supports consistency checks, i.e. checking the completeness and accuracy of the coverage of system requirements. 
2. supports the impact assessment in case of change requests or bugs. 
3. supports the reporting of stakeholder expectations and identify which requirements have been tested, often referred to as coverage reports 

The second part of this aspect is about consistency. 

Consistency can only be proven in a review where you show that you covered the system requirements completely and correctly. 

If you skip this review you may have incomplete or faulty system test cases. The worst part is that you may not even notice the defects in the system qualification test because this test is performed against your system requirements. If these are faulty, your test may not show false behavior.  

What is the benefit of a System Qualification Test?

These tests are carried out after the system has been integrated from hardware, software, mechanics and possibly other components. These tests confirm that the complete system works as expected and meets the system requirements. 

What is the content of the System Qualification Test Process? 

• The system test strategy for the tests of the overall system is defined (BP1). This supports all parties involved (testers, test managers, requirements engineers, project management, etc.) to develop a common understanding of how to accomplish the work (BP1). 
• It must be prevented that changes to the system have negative side effects on already successfully tested requirements. For this purpose a regression strategy needs to be provided (BP1). 
• The tests that demonstrate the implementation of the system requirements are to be specified in detail (BP2). 
• Test cases are selected for execution in line with the test strategy, the regression strategy, and the release plan (BP3). Tests are performed and results are recorded (BP4). 
• There needs to be bidirectional traceability between system requirements, test cases and test results (BP5). The system requirements and the test cases need to be consistent (BP6). Consistency means that the traceability links are correct and complete and that the test cases are correct with respect to the requirements. This is typically checked through reviews. 
• A test summary report is produced and disseminated to the relevant people (like project management and customer) (BP7). 

Bidirectional traceability between system requirements and system qualification tests

Experiences, problems and hints

• The system qualification test process (SYS.5) basically expects the same activities as the software qualification test process (SWE.6). The difference is just the object that is tested. In practice, both test processes are often combined on the target hardware. This is acceptable as long as one can demonstrate that the system requirements have been tested completely. 
• The system qualification test is closely connected to the system requirements activities (SYS.2). The requirements engineers develop the verification criteria which contain their expertise and hints on what should be observed when developing the tests. 
• Traceability between requirements and tests is essential to be able to demonstrate coverage of requirements by tests and to monitor the test progress. This requires automated traceability using a highly integrated requirements tool and test tool. 
• Consistency checking is often underestimated and poorly implemented. This requires reviews which check whether the tests are correct, and the requirements are covered completely. 
• If the project uses a platform system, the tests against platform requirements and those against the project-specific requirements should be seamlessly adjusted.  

If you’re serious about learning Automotive SPICE®,  participate in one of our Automotive SPICE® trainings.  

About the author 

Bhaskar Vanamali has been working on process improvement for nearly 20 years and served as secretary of Working Group 13 of the VDA QMCHe is Principal Assessor and Trainer for Automotive SPICE® and has performed more than 140 assessments and trained more than 250 assessors. 

Read our related content

System Qualification Test (SYS.5) resource guide

Services for Automotive SPICE and Extensions