Methods for Map Validation: Takeaways From IEEE IV 2022

June 16, 2022

Applied Intuition participated in the Online Map Validation and Road Model Creation (MaVRoC) workshop at the Institute of Electrical and Electronics Engineers (IEEE) Intelligent Vehicles (IV) Symposium this year. Our team led a discussion to share challenges and jointly gain new perspectives on offline map and online road model creation and validation for autonomous driving (Figure 1).

Figure 1: Applied Intuition presented to academia and industry stakeholders at IEEE IV 2022.

Many autonomy programs face map curation challenges that map validation can help mitigate:

Maps are difficult to create and expensive to purchase, as their creation requires large amounts of data and a combination of classical and machine learning (ML) algorithms.
Maps go stale quickly. Changes in real-world conditions require teams to update autonomous driving maps constantly.
Scaling map development is costly. Teams are able to hand-curate maps if their operational design domain (ODD) is small and spans thousands of miles. However, scaling these labor-intensive operations for cross-country maps covering up to millions of miles results in significant costs.

To help autonomy programs address the above challenges, Applied Intuition’s workshop presentation focused on the following key topics.

Methods for Map Validation

Maps are a key component of localization systems. They provide a priori information about landmarks that enable an autonomous vehicle (AV) to place itself in the world. Incorrect maps can cause the AV stack to provide poor localization estimates, which can cause disengagements during on-road testing or, in the worst case, accidents. Teams need to validate their stack continuously and update their maps whenever they receive new data, therefore addressing and mitigating the challenges of map curation.

Different map validation methods exist—each with its own advantages regarding speed and effectiveness and different levels of fidelity and cost:

Algorithmic map validation

Algorithmic map validation requires teams to write unit tests that verify map data correctness and completeness. This method is relatively inexpensive and fast to carry out. It catches errors that may result from improper data cleaning or missing data. Teams can also use algorithmic validation to check for semantic relationships and geometric issues that may exist in the map (e.g., invalid lane widths).

Simulation-based map validation

Simulation-based map validation requires teams to define map-agnostic test cases relative to the AV, map points, and actors. Teams will also query the map to find various road features and generate scenarios at scale. This method helps teams find errors in the interaction between the map and the autonomous planning system.

Data-in-the-loop map validation

Data-in-the-loop map validation (or real-world data validation) requires teams to compare ground truth maps to information that fleets of AVs collect on the road. This method uses ML to perform feature extraction from sensor data, creating a “lightweight” map. Teams can perform “diffs” between this lightweight map and the real world to determine where the real world has changed. This can happen using ID-based checks or geometric heuristics. Real-world data validation is usually sufficient to find most changes, and it is especially effective with large fleets.

Applied Intuition’s Approach

Applied Intuition’s map editing and validation tool Map Toolset* enables autonomy programs to create maps, validate them in the real world, and address the key challenges of map validation by combining all of the methods mentioned above. Applied Intuition’s platform automates geometric and semantic checks and uses continuous integration and continuous deployment (CI/CD) processes to help prevent regressions. Furthermore, Applied Intuition’s solutions integrate with a variety of different map formats to ensure that errors found in the validation process relate to true map errors (Figure 2).