Self-driving from VISION ONLY - Tesla's self-driving progress by Andrej Karpathy (Talk Analysis)

Tesla is pushing the state-of-the-art in full self-driving, and interestingly, they explicitly switch from having multiple different sensors to a vision-only system. We discuss the highlights of Andrej Karpathy’s talk about Tesla’s FSD system, how to label petabytes of data, how to sample edge-cases, how to train a neural network that has to work in real-time, and why moving to having only cameras is superior to multi-sensor approaches.

OUTLINE:
0:00 - Intro & Overview
1:55 - Current Auto-Breaking system
3:20 - Full Self-Driving from vision only
4:55 - Auto-Labelling for collecting data
8:45 - How to get diverse data from edge-cases
12:15 - Neural network architecture
16:05 - Tesla’s in-house supercomputer
17:00 - Owning the whole pipeline
18:20 - Example results from vision only
23:10 - Conclusion & Comments