Log data management is one of the most important tasks that every autonomy program needs to master. Test fleets collect on average four terabytes of log data per vehicle per day, while production fleets (i.e., vehicles purchased by individual consumers) can generate millions of events per day. This firehose of data has enormous potential to power an autonomy program’s development efforts.
Due to the costs and risks involved in real-world testing, it is crucial that autonomy programs collect and manage their log data effectively. For example, to operate a test fleet, autonomy programs must purchase and maintain vehicles and sensors and pay a team of safety operators. Additionally, just one critical mistake during real-world testing can put human lives at risk. Autonomy programs should thus implement practices to scale their data collection efficiently, create a pipeline for effective log data processing, and build scalable workflows that extract the maximum value from all collected data.
Applied Intuition’s log management handbook discusses the technical building blocks, ideal workflows, and cost management strategies of an expansive log management process. This blog post is the first in a three-part series providing a short introduction to these topics. The full-length handbook is available for download below.
In autonomous systems development, log data is any real-world data collected on the system corresponding to the autonomous task at hand. For autonomous vehicles, log data is collected during a drive and ranges from raw sensor inputs to pedal or wheel actuation commands.
The concepts, principles, and approaches laid out in our log management handbook apply to autonomy programs of all sizes and across industries. Most metrics and examples concern SAE Level 2-4 systems in automotive, but the contents of our handbook are equally relevant to autonomous trucking, construction, mining, and agriculture as well as warehouse robots, unmanned aerial systems, and other types of autonomous systems.
Our handbook’s structure follows the journey of a log file from inception to long-term storage (Figure 1). First, an autonomous system collects the log file. Next, data processing pipelines distribute the log file, and different teams explore it according to their specific use case. Finally, the log file lands in long-term storage. Our handbook discusses each of these steps in detail.
Log data powers various workflows for different teams within autonomous systems development. Our handbook covers the following workflows:
Log data is one of the most essential building blocks of autonomous systems development. Our log management handbook explains common log data management challenges and lays out recommended practices for autonomy programs across industries. The next part of this blog post series will summarize the handbook’s key insights regarding an important step in the log management life cycle: Log exploration. Stay tuned for the next blog post, or download the entire handbook today.
Contact the Applied team with your questions about this handbook and learn more about Applied’s log management solutions on our website.
