ISO 26262 Software Compliance in the Automotive Industry

System testing tests the system as a whole. Once all the components are integrated, the entire system is tested rigorously to verify it meets the specified functional, safety, security, and other nonfunctional requirements.

Specialized testing teams perform this type of testing in safety-critical software. System testing falls within the scope of black box testing. As such, it shouldn’t require any knowledge of the inner design of the code or logic.

An important distinction with system level testing is the system is tested in an environment that is close to the production environment where the application will be deployed. At this stage, specific safety functions are validated, and system wide security testing is run.

Automotive System Testing at the Service Level

Individual systems within an automobile may not be considered part of a service. However, connectivity into larger systems means they should be. For example, in an automobile, the role of the engine control unit (ECU) alone is to ensure proper combustion and emissions in the engine, but the car is tracking fuel economy, using the ECU, and reporting it to a central server over a wireless connection.

This mileage data is then used to plan routes and estimate operating costs. Suddenly, the ECU is a critical leaf node in a business decision making process.

Instead of viewing system quality in terms of meeting individual device requirements, the scope is broadened to consider the quality of the services provided. Testing at the service level ensures nonfunctional requirements are met. For example, performance and reliability are difficult to assess at the device level or during software unit testing. Service based testing can simulate the operational environment of a device to provide realistic loads. In the HVAC example, the new temperature sensor can be tested with varying request rates to see if it meets performance requirements.

Security is a significant concern in automotive systems. Cyber attacks most likely originate from the network itself by attacking the exposed APIs. Service based testing can create simulated environments for robust security testing, either through fuzzing (random and erroneous data inputs) or denial-of-service attacks. A new temperature sensor in the HVAC example might operate correctly with expected requests, but crash when overloaded. An attacker might be able to exploit this to overload the system and cause an outage.