History of Computer Vision
Six decades of teaching computers to see
The Big Picture: 60 Years
Computer Vision (CV) is the field of getting computers to understand images. Starting from the often-cited 1966 "summer assignment" (there was earlier work too, such as on recognizing 3D solids), it passed through the era of geometric models and the era of hand-crafted features, and then developed into the deep-learning revolution of the 2010s, the neural 3D representations of the 2020s, and on to foundation models and generative AI. Let's start by taking in the whole 60-year story in a single picture.
The 1960s: An Optimistic Start
In 1966, MIT launched a research plan called the "Summer Vision Project." At the time, people were optimistic enough to think that a single summer might be enough to give a computer sight, and the implementation was handed to a student. It fell well short of its goals. "Seeing" turned out to be a far harder problem than anyone had assumed at the time.
Lesson of the era: Why is something humans do in an instant so hard for a computer? This question became the starting point that drove the next 60 years of research.
The 1980s: Marr's Framework
In his book Vision (1982), David Marr organized vision as information processing that builds up representations in stages. It was a landmark work that gave computational vision a theoretical framework, and it remains a basic reference point in vision research today.
Marr's three levels of representation
- Primal sketch: basic features obtained directly from the image, such as edges and contours.
- 2.5-D sketch: the orientation and depth of surfaces as seen from the viewer (a viewpoint-dependent representation; "2.5" is not a number of dimensions but means depth of the visible surfaces only).
- 3-D model: a viewpoint-independent representation of the object's own shape.
The 1990s-2000s: Features and Engineering
In 1999, David Lowe's SIFT (Scale-Invariant Feature Transform) appeared. It detects feature points that can be found reliably even when scale or rotation changes, and describes their surroundings as a numeric vector. It became the foundation for object recognition, panorama stitching, and Structure from Motion (3D reconstruction from multiple views).
Around the same time, methods based on geometric models were systematized, including earlier inventions such as Harris corner detection (1988), the Hough transform (1962), and epipolar geometry. This was the era of engineering, in which humans designed good features by hand.
Key tools of the era
- SIFT / SURF: local features robust to scale and rotation.
- Harris corners: detection of corners in an image.
- Hough transform: detection of lines and circles.
- Epipolar geometry: the basis for matching two images and for 3D reconstruction.
The 2010s: The Deep Learning Revolution
In 2012, AlexNet achieved a top-5 error rate of about 15% in the image-recognition contest built on ImageNet data (ILSVRC), far below the runner-up (a hand-crafted-feature method) at about 26%, stunning researchers. Deep convolutional neural networks (CNNs) rapidly replaced the earlier hand-crafted-feature approaches.
After that came deep networks such as VGG and ResNet; on ImageNet classification, ResNet (2015) reached a top-5 error of about 3.6%, below a human estimate (about 5%, a rough figure from one person's experiment). In 2020, ViT (Vision Transformer), which applies the Transformer to images, appeared.
Turning point: a major shift from "humans design the features" to "let the model learn the features from data." Feature learning had been studied earlier, for example in CNNs by LeCun et al. (1998), but after AlexNet it became the mainstream of today's CV.
The 2020s: 3D and Neural Representations
In 2020, NeRF (Neural Radiance Fields) appeared. From many photos with known camera positions and orientations, a neural network itself implicitly learns "the 3D structure and appearance of a scene" and generates images from new viewpoints (it cannot correctly reproduce parts that were never photographed).
In 2023, 3D Gaussian Splatting (3DGS) emerged, representing a scene explicitly with many Gaussians (blurred points) to achieve fast, high-quality rendering. NeRF and 3DGS are the state of the art in 3D representation, reshaping CG, VR, and self-driving. And from 2024 onward, CV faces even bigger changes.
2024 Onward: Foundation Models and Generative AI
Progress did not stop after 2023. If anything, the previously separate threads of "recognition," "generation," "3D," and "language" are increasingly merging into large foundation models (models pre-trained on broad data that can be adapted, for example by further training, to many different tasks).
Multimodal foundation models: building on CLIP (2021), which linked images and language, huge models that handle images and text together, such as GPT-4o and Gemini, have become practical. Rather than building a dedicated recognition model from scratch, we are moving toward reusing a general model that "sees, reads, and answers." In segmentation too, we went from SAM (Segment Anything, 2023) to SAM 2 (2024), extending from still images to video.
Video generation and world models: Diffusion Transformers, exemplified by Sora (2024), can now generate high-quality video from text. Research is broadening from merely "seeing" to world models that predict how things change over time (though a natural-looking video does not guarantee that the laws of physics are reproduced correctly).
Feed-forward 3D reconstruction: DUSt3R / MASt3R (2024) estimate dense 3D in a single pass with a network, from multiple images whose camera positions are unknown. Depth Anything (2024) is a foundation model that estimates relative depth from a single image (metric distances come from fine-tuned versions). 3D Gaussian Splatting is also being extended to moving scenes (4D).
A recent trend: from an era of designing a small model per task to an era of reusing huge foundation models across many tasks. These models keep growing larger, while at the same time becoming faster (real-time) and lighter for edge devices. CV keeps evolving while merging with language, generation, and 3D.
Summary
The 60 years of computer vision are a history of humanity's ongoing challenge with "the difficulty of seeing." The central idea has shifted era by era, as follows.
| Era | Representative technology | Central idea |
|---|---|---|
| 1966 | MIT Summer Vision Project | Optimism about "solving vision," and its setback |
| 1980s | Marr's framework | Organizing vision as the staged construction of representations |
| 1999 | SIFT | Robust, human-designed hand-crafted features |
| 2012 | AlexNet | Learning features from data (the DL revolution) |
| 2020 / 2023 | NeRF / 3DGS | 3D scene representations optimized from images (NeRF: implicit, by a neural network; 3DGS: explicit, by Gaussians) |
| 2024- | Foundation models / generative AI | Fusion of recognition, generation, 3D, and language (SAM 2, Sora, DUSt3R) |
- The starting point was the 1966 MIT Summer Vision Project.
- Marr provided a theoretical framework, and SIFT built the era of features.
- The 2012 AlexNet marked the shift of the mainstream from design to "learning" (research on feature learning predates it).
- The 2020s advanced 3D representation with NeRF and 3DGS, and from 2024 onward the fusion into foundation models and generative AI is accelerating.
Frequently Asked Questions
Is SIFT already outdated?
It is being overtaken by learned deep features, but because it is robust to scale and rotation and needs no training data, it is still used for 3D reconstruction (structure from motion) and image matching. Real-time SLAM and AR mainly use faster features such as ORB.
What is the difference between CV and image processing?
There is no strict boundary, but image processing tends to deal with image-to-image tasks (filtering, correction), while CV deals with image-to-meaning tasks (recognition, 3D reconstruction).
How will CV develop from here?
Larger foundation models (CLIP, Segment Anything, etc.), 3D reconstruction and novel-view synthesis (evolution of NeRF/3DGS), and greater efficiency (edge AI) are expected.