History of Computer Vision
Six decades of teaching computers to see
The Big Picture: 60 Years
Computer Vision (CV) is the field of getting computers to understand images. It began in 1966 as a "summer assignment," passed through the era of geometric models and the era of hand-crafted features, and then developed into the deep-learning revolution of the 2010s, the neural 3D representations of the 2020s, and on to foundation models and generative AI. Let's start by taking in the whole 60-year story in a single picture.
The 1960s: An Optimistic Start
In 1966, MIT launched a research plan called the "Summer Vision Project." At the time, people were optimistic enough to think that a single summer might be enough to give a computer sight, and the implementation was handed to a student. It ended in failure. "Seeing" turned out to be a far harder problem than anyone had assumed at the time.
Lesson of the era: Why is something humans do in an instant so hard for a computer? This question became the starting point that drove the next 60 years of research.
The 1980s: Marr's Framework
In his book Vision (1982), David Marr organized vision as information processing that builds up representations in stages. It was a landmark work that gave computational vision a theoretical framework, and it remains a basic reference point in vision research today.
Marr's three levels of representation
- Primal sketch: basic features obtained directly from the image, such as edges and contours.
- 2.5-D sketch: the orientation and depth of surfaces as seen from the viewer (a viewpoint-dependent representation).
- 3-D model: a viewpoint-independent representation of the object's own shape.
The 1990s-2000s: Features and Engineering
In 1999, David Lowe's SIFT (Scale-Invariant Feature Transform) appeared. It detects feature points that can be found reliably even when scale or rotation changes, and describes their surroundings as a numeric vector. It became the foundation for object recognition, panorama stitching, and Structure from Motion (3D reconstruction from multiple views).
Around the same time, methods based on geometric models were systematized, including Harris corner detection, the Hough transform, and epipolar geometry. This was the era of engineering, in which humans designed good features by hand.
Key tools of the era
- SIFT / SURF: local features robust to scale and rotation.
- Harris corners: detection of corners in an image.
- Hough transform: detection of lines and circles.
- Epipolar geometry: the basis for matching two images and for 3D reconstruction.
The 2010s: The Deep Learning Revolution
In 2012, AlexNet cut the error rate (top-5 error) in the ImageNet image-recognition contest from about 26% to about 15%, stunning researchers. Deep convolutional neural networks (CNNs) rapidly replaced the earlier hand-crafted-feature approaches.
After that, deep networks such as VGG and ResNet, and then ViT (Vision Transformer), which applies the Transformer to images, pushed accuracy on some tasks beyond human levels.
Turning point: a major shift from "humans design the features" to "let the model learn the features from data." This became the mainstream of today's CV.
The 2020s: 3D and Neural Representations
In 2020, NeRF (Neural Radiance Fields) appeared. A neural network itself implicitly learns "the 3D structure and appearance of a scene" and generates images from any viewpoint.
In 2023, 3D Gaussian Splatting (3DGS) emerged, representing a scene explicitly with many Gaussians (blurred points) to achieve fast, high-quality rendering. NeRF and 3DGS are the state of the art in 3D representation, reshaping CG, VR, and self-driving. And from 2024 onward, CV faces even bigger changes.
2024 Onward: Foundation Models and Generative AI
Progress did not stop after 2023. If anything, the previously separate threads of "recognition," "generation," "3D," and "language" are increasingly merging into large foundation models (huge AIs that can be reused for many different tasks).
Multimodal foundation models: building on CLIP (2021), which linked images and language, huge models that handle images and text together, such as GPT-4o and Gemini, have become practical. Rather than building a dedicated recognition model from scratch, we are moving toward reusing a general model that "sees, reads, and answers." In segmentation too, we went from SAM (Segment Anything, 2023) to SAM 2 (2024), extending from still images to video.
Video generation and world models: Diffusion Transformers, exemplified by Sora (2024), can now generate high-quality video from text. The goal is broadening from merely "seeing" to simulating a world that includes physics and time.
Feed-forward 3D reconstruction: DUSt3R / MASt3R (2024) estimate dense 3D in a single pass with a network, from multiple images whose camera positions are unknown. Depth Anything (2024) is a foundation model that estimates depth from a single image. 3D Gaussian Splatting is also being extended to moving scenes (4D).
A recent trend: from an era of designing a small model per task to an era of reusing huge foundation models across many tasks. These models keep growing larger, while at the same time becoming faster (real-time) and lighter for edge devices. CV keeps evolving while merging with language, generation, and 3D.
Summary
The 60 years of computer vision are a history of humanity's ongoing challenge with "the difficulty of seeing." The central idea has shifted era by era, as follows.
| Era | Representative technology | Central idea |
|---|---|---|
| 1966 | MIT Summer Vision Project | Optimism about "solving vision," and its setback |
| 1980s | Marr's framework | Organizing vision as the staged construction of representations |
| 1999 | SIFT | Robust, human-designed hand-crafted features |
| 2012 | AlexNet | Learning features from data (the DL revolution) |
| 2020 / 2023 | NeRF / 3DGS | 3D scene representation by neural networks |
| 2024- | Foundation models / generative AI | Fusion of recognition, generation, 3D, and language (SAM 2, Sora, DUSt3R) |
- The starting point was the 1966 MIT Summer Vision Project.
- Marr provided a theoretical framework, and SIFT built the era of features.
- The 2012 AlexNet marked the shift from design to "learning."
- The 2020s advanced 3D representation with NeRF and 3DGS, and from 2024 onward the fusion into foundation models and generative AI is accelerating.