Read the ISOCC 2021 paper that uses my work!
Human-Inspired Camera: A Novel Camera System for Computer Vision
Unity3D Game Engine as a Versatile Synthetic Data Generation Platform
One of the heaviest impacting factors on resources when developing Advanced Driver Assistance Systems is the process of real data collection. A fully-equipped data rig must be deployed on streets for extended periods; this proves to be expensive in areas of hardware maintenance, data post-processing, and manual annotation subject to human error. Moreover, diversity is constrained to geography - in other words, there is only so much that can be captured within the 24 hours of a day.
We propose an automated pipeline to leverage the scripting and physics capabilities of the Unity game engine to generate synthetic data sets cost-effectively. A wide range of data formats can be produced including RGB-D to Velodyne-like point clouds. Total control of the environment via generation parameters means synthetic augmentations such as lighting, weather, and scene context can be manipulated. The effects of these augmentations on neural network performance on crucial scene-understanding tasks like 2D/3D object detection or instance segmentation can therefore be rigorously studied. Preliminary results show that our synthetic data holds the potential to boost 2D object detection performance and model robustness.
Aside from being a means of data generation, our Unity platform can serve as a safe testbed for collaborative algorithms. A complex road network utilizing V2X techniques can be adequately simulated while taking into account bandwidth constraints.
Powerful Unity3D Foundation
Built on top of Unity3D, our platform operates within a virtual world rich with high-fidelity models and textures. Plus, Unity's C# scripting API enables the full automation of the data collection process. We can cost-effectively generate vast amounts of data with the added benefit of ensured variability. With a single RTX 2080 Ti, an image generation rate of 5 Hz is achievable.
Accurate Ground Truth
By using a simulation, we inherently have access to the variables that define the virtual world - every generated dataset comes with complementary ground truth accurate down to the pixel and virtually for free. Realtime ground truth can also be simulated, such as pseudo car odometry and camera ego-motion.
Deterministic Control of Conditions
A wide range of weather conditions can be simulated through techniques used in the game industry. From sunlight direction to harsh snow, the inclusion of adverse conditions serves as a deterministic method to benchmark neural network performance.
Diverse Datasets
Manipulation of conditions allows us to generate datasets of high variability to avoid data bias. Not only can diversity be applied stylistically, but ground truth such as 2D bounding boxes can also exhibit randomness through changing poses and/or occlusion amount. Preliminary experiments demonstrate that our synthetic data provides object detection boost when trained alongside real data. Moreover, our synthetic data alone holds the same generalizing capabilities as other real data sets.
Collaborative Testbed
The creation of a real collaborative dataset proves to be difficult because of technicalities. Our platform eliminates issues of inaccurate pose data, unstable hardware synchronization, and laborious point cloud annotation. We can generate collaborative data sets of 2D and 3D formats tailored for specific applications such as feature matching and V2X edge computing.