Cost-Effective Testing of IoT Devices With Virtual Labs

Even the smallest Internet of things (IoT) device lives in a complex environment that may not be fully understood at the time of development. There are security risks associated with devices being connected to the internet for the first time and benefits to service orientation for design, development, and testing. In this post, we’ll take service-based testing and service virtualization to the next step: virtual labs.

Building a realistic physical testing lab environment is difficult. Even when complete, it becomes the main bottleneck in system testing. Virtual labs remove this bottleneck while providing new benefits to service-based IoT device testing.

Many IoT Devices Are Not Ready for Primetime

A recent study found that 80% of IoT apps are not being tested for security flaws. The Barr Group found that 56% of embedded device developers don’t review source code for security vulnerabilities and 37% don’t have a written coding standard. These are not encouraging statistics. It’s clear that IoT device manufacturers need to take quality, safety, and security more seriously.

Test automation is one important step in order to make sure testing is being done more rigorously, consistently, and thoroughly. Testing, especially for security vulnerabilities, is often seen as too costly and complex and is therefore rushed or overlooked altogether. But it’s an expensive mistake to let your customers (and attackers) test your IoT device security for you.

The Ingredients of the IoT Virtual Lab

A real test lab requires the closest physical manifestation of the environment an IoT device is planned to work. But even in the most sophisticated lab, it’s difficult to scale to a realistic environment. A virtual lab fixes this problem. Virtual labs evolve past the need for hard-to-find or nonexistent hardware dependencies. Combine virtual labs with key test automation tools:

• Service virtualization simulates all of the dependencies needed by the device under test in order to perform full system testing. This includes all connections and protocols used by the device with realistic responses to communication. For example, service virtualization can simulate an enterprise server backend that an IoT device communicates with to provide periodic sensor readings. Similarly, virtualization can control the IoT device in a realistic manner.
• Service and API testing provide a way to drive the device under test in a manner that ensures the services and APIs provided are performing flawlessly. These tests can be manipulated via the automation platform to perform performance and security tests as needed.
• Runtime monitoring detects errors in real time on the device under test and captures important trace information. For example, memory leaks, which can remain undetected in a finished product, can be caught and resolved early and cheaply.
• Test lab management and analytics provide overarching control of the virtual labs. Once virtualized, the entire lab setup can be replicated as needed and test runs can be automated and repeated. Analytics provide the necessary summary of activities and outcomes.

The edge computing IoT ecosystem below depicts a typical environment in which embedded IoT devices are deployed. Sensors and control devices communicate information to the edge, which is a series of appliances or applications that can receive information and use logic to communicate back to a device or up to the cloud.

The cloud then has higher level logic that allows it to act upon that information. The cloud is a set of services like microservices, connections to databases, additional logic, or third-party services. It’s a complex web of functional building blocks, shown below to the right.

The Role of Parasoft’s Virtualize & SOAtest in Creating a Virtual Lab Environment

To validate the complexity of IoT environments, Parasoft Virtualize simulates required dependencies and Parasoft SOAtest drives tests that emulate these inputs. These tools enable performing realistic test scenarios across all devices over the network, covering generic protocols like REST/HTTP and popular IoT protocols such as CoAP, XMPP, and MQTT.

Additionally, Parasoft verifies that the device under test, which in this example is the gateway, effectively communicates with cloud services by validating the responses received from SOAtest. The image below illustrates how a virtual lab environment can be established for testing edge devices.

If there are external ways of communicating information into that gateway, those calls can be simulated as well. Parasoft Virtualize is designed to stabilize the testing environment to create predictable responses to requests that leverage test data from SOAtest, fully testing the gateway and services. 

Finally, the top-level services might be communicating back to the edge and back to other sensors and external actors. It might be important to know that the flow from your inputs is making its way through the environment back to the backend systems. Parasoft Virtualize simulates the receiving of those calls down to the edge—down to the IoT devices—and then relays that information back to SOAtest to confirm that the call made the round trip and behaved as expected inside the IoT ecosystem. The combination of Parasoft Virtualize and SOAtest provides full control to test the whole environment, even within the complexities of an IoT ecosystem.

Improve Quality and Security While Reducing Time, Costs, and Risks

Normal test environments are expensive, probably more so than most development managers forecast. A study by voke Research found that the average investment in a pre-production lab was $12 million. In terms of time, the average time to provision the lab was 18 days and a further 12-14 days were spent on configuration.

These labs take lots of time and money to set up. After that, they act as the bottleneck for testing due to limited access. Further, the day-to-day operational costs of physical labs are significant. In most cases, duplicating a physical lab to increase test throughput is cost-prohibitive.

Let’s break down the benefits of the virtual IoT test lab.

• Improve quality through better and more complete testing. Service-based testing ensures that key use cases are exercised and perfected. Automated performance tests ensure stability and reliability under heavy loads. In addition, runtime monitoring ensures that hard-to-find bugs are detected and traced.
• Improve security with automated penetration tests that simulate malformed data. Load testing can simulate denial of service attacks and runtime monitoring can detect security vulnerabilities. Test repeatability ensures that each iteration, patch, or release is tested in the exact same manner. In addition, test development and manipulation, such as improving and creating new tests, is simplified.
• Reduce testing time, risk, and cost by removing the need for expensive dependencies required for complete systems testing. Automation provides repeatability and consistency that manual testing cannot. It also provides better and more complete testing. Virtual labs greatly reduce the provision time needed for physical lab setups, impacting total test time.

Conclusion

Given the state of IoT device development, changes need to be made to development and testing processes. Test automation is a proven approach to reduce costs and risk. The next big step in quality and security improvement for IoT devices is using virtual labs that combine service virtualization, service-based testing, virtual lab management, and runtime monitoring. This reduces the provisioning and configuration costs while increasing the quality of the testing being performed.