Best Practices for Building Safe Autonomous Systems: Takeaways From ADAS & AV Technology Expo

Applied Intuition was proud to participate in this year’s ADAS and Autonomous Vehicle Technology Expo in San Jose, California. Our team gave a presentation covering best practices for building safe autonomous systems (Figure 1). The following blog post summarizes key takeaways from our presentation for those who could not attend this year’s event.

Best Practices for Building Safe Autonomous Systems

Autonomy programs typically encounter three challenges when building and validating advanced driver assistance systems (ADAS) or automated driving systems (ADS): Programs might find it difficult to build test coverage for an infinite space, assess whether they have conducted enough testing, and scale simulation tests (Figure 2). Applied’s best practices for building safe ADS thus include three major pillars: A scalable test library, thorough coverage analysis, and automated testing.

Building a test library

Testing is the foundation of safe ADAS and ADS development. There are four common ways to create tests (Figure 3). Each of them comes with its benefits and challenges, and each of them can help autonomy teams support specific development needs.

1. From drive data: Autonomy programs can create scenarios from real-world drive data. When choosing this method, programs should annotate behaviors automatically during log ingestion, query the drive data for scenes of interest​, create test scenarios directly from those scenes, and re-simulate variations of those scenarios to scale their scenario library.
2. Abstract and logical scenarios: Another way to scale scenario creation is by creating synthetic (abstract or logical) scenarios. Abstract scenarios cover as large a parameter space as possible​, while logical scenarios define a range of realistic parameters. Autonomy programs should leverage tools that can generate large numbers of scenario variations programmatically.
3. From maps: Autonomy programs can expand their map coverage by creating scenarios based on maps. When choosing this method, programs should query their maps for locations that are difficult to navigate and leverage tools that can seamlessly create simulation scenarios at those locations of interest.‍
4. Environmental parameterization: Environmental parameterization allows autonomy programs to generate realistic 3D worlds and validate their perception system in varying environmental conditions. This can help programs reproduce certain edge cases deterministically and discover unsupported edge cases.

Coverage analysis

Based on their test library, autonomy programs should conduct coverage analysis to determine whether they have carried out enough tests. There are four types of coverage that autonomy programs should track: Requirements coverage, parameter coverage, map coverage, and ODD coverage (Figure 4).

Automated testing

Once they have built their test library and implemented a process for coverage analysis, autonomy programs should leverage cloud simulation to scale their scenario-based testing. This allows programs to trigger tests automatically, run thousands of tests in parallel, configure various test suites, analyze terabytes of test data, identify issues, and resolve them rapidly.

Key takeaways

Building safe autonomous systems is no easy task, but with certain best practices, programs can alleviate the challenges of ADS validation. First, programs can expand coverage effectively by using a combination of scenario creation approaches. Second, teams can guide their scenario creation efforts with coverage analysis to identify untested and low-performing areas of their ODD. Third, autonomy programs can leverage automated testing to conduct simulation tests at scale.

Applied’s Approach

Applied Intuition offers a number of solutions that help autonomy programs with their scenario creation, coverage analysis, and automated testing efforts. Over the years, our team has acquired extensive industry experience, worked with customers to help them progress towards their validation and safety goals, tracked the latest research, collaborated with regulators and standardization committees, and developed the best tools and processes to support our customers.

‍

‍Contact us to learn more about our solutions for safe ADS development and validation.