Camera Matrix

Overview of the camera model

The process of projecting a point in 3D space onto a 2D image is expressed as a chain of several coordinate transformations.

Intrinsic matrix K

The intrinsic parameters describe characteristics that are specific to the camera itself.

Extrinsic parameters [R|t]

The extrinsic parameters describe the position and orientation of the camera relative to the world coordinate system.

Major coordinate-system conventions (OpenCV, OpenGL, DirectX/Unity)

This article's convention of $X_c$ right, $Y_c$ down, $Z_c$ forward is the OpenCV convention. Other major conventions exist, and which is the majority or minority changes from field to field. The handedness (right- or left-handed) also differs between conventions.

$X$ right, $Y$ up, $Z$ into the scene (forward) is left-handed, which is the DirectX / Unity convention. Starting from the OpenCV convention ($Y$ down) and flipping only $Y$ to point up flips a single axis, turning right-handed into left-handed and yielding this convention (Unreal is also left-handed, but assigns the axes differently, with $X$ forward and $Z$ up). By contrast, the OpenCV and OpenGL conventions are related by flipping $Y$ and $Z$ together (a 180° rotation about the $X$ axis), so both are right-handed. Because this article is in a computer-vision context, it adopts the OpenCV convention.

Even for the same $[R|\mathbf{t}]$, the picture looks different depending on which coordinate system you hold fixed while viewing it. Figure 3 was world-anchored (hold the world fixed and draw the camera's pose within it; this is the view for tasks that deal with camera pose and trajectory). Conversely, if you fix the camera at the origin, then the world coordinate system is placed by rotating and translating it via $[R|\mathbf{t}]$. This is exactly the projection equation $\mathbf{P}_c = R\,\mathbf{P}_w + \mathbf{t}$ (world→camera), which corresponds to the view matrix in rendering.

Camera matrix P

The camera matrix (projection matrix) is the combination of the intrinsic and extrinsic parameters.

Expanding:

Common points of confusion (pitfalls)

The same symbol or name often refers to different things in different sources, and this is one of the biggest factors that make learning computer vision hard. Online sources appear to contradict one another mostly because many of them do not state which convention they adopt. First, the names themselves clash across fields:

Because the same name can refer to different matrices, it is safest to first check the definition used in that particular source. Next come mismatches in how equations are written and in conventions:

• Row vectors vs. column vectors: the column-vector convention $\mathbf{p}' = R\,\mathbf{p}$ (this article, mathematics, OpenCV) or the row-vector convention $\mathbf{p}' = \mathbf{p}\,R$ (DirectX, etc.)? A transpose swaps the whole equation. On top of that, whether the matrix memory layout is row-major (C / NumPy) or column-major (OpenGL / Eigen default) is yet another separate issue.
• Direction of the transform: world→camera ($R,\ \mathbf{t}$) or camera→world ($R^{\top},\ \mathbf{C} = -R^{\top}\mathbf{t}$)? The same "extrinsic parameters" can refer to opposite directions (Figures 3 and 4).
• Active vs. passive rotation: rotating the point (active) or rotating the coordinate system (passive)? $R$ and $R^{\top}$ swap.
• Handedness and axes of the coordinate system: right- / left-handed, $Y$ up / down, $Z$ forward / backward (the comparison table above).
• Image coordinates: origin at the top-left (CV) or the bottom-left (OpenGL)? Is the pixel center at integer coordinates or at $+0.5$?
• Representation of rotation: the rotation order of Euler angles; whether a quaternion has $w$ first or last; Hamilton / JPL convention.

Summary of coordinate systems

Code example (Python)

import numpy as np

# Intrinsic parameters
fx, fy = 800, 800
cx, cy = 320, 240
K = np.array([
    [fx,  0, cx],
    [ 0, fy, cy],
    [ 0,  0,  1]
])

# Extrinsic parameters (e.g. rotate 45 deg about the Z axis, move to (1,0,5))
theta = np.radians(45)
R = np.array([
    [np.cos(theta), -np.sin(theta), 0],
    [np.sin(theta),  np.cos(theta), 0],
    [0,              0,             1]
])
t = np.array([[1], [0], [5]])

# Extrinsic matrix [R|t]
Rt = np.hstack([R, t])

# Camera matrix P = K[R|t]
P = K @ Rt
print("Camera Matrix P:")
print(P)

# Project a 3D point
P_world = np.array([0, 0, 0, 1])  # world origin
p_homogeneous = P @ P_world
p = p_homogeneous[:2] / p_homogeneous[2]
print(f"Projected point: ({p[0]:.1f}, {p[1]:.1f})")

# Camera position (world coordinates)
camera_position = -R.T @ t.flatten()
print(f"Camera position: {camera_position}")

# Using OpenCV
import cv2
# When using cv2.projectPoints()
rvec, _ = cv2.Rodrigues(R)  # rotation matrix -> rotation vector
tvec = t  # tvec is not the camera position; it is the world origin expressed in the camera frame (= t)
points_3d = np.array([[0, 0, 0]], dtype=np.float32)
points_2d, _ = cv2.projectPoints(points_3d, rvec, tvec, K, None)

Summary

• The camera matrix $P = K[R|\mathbf{t}]$ is a $3 \times 4$ matrix representing the 3D→2D projection.
• Intrinsic parameters $K$: focal length and principal point (camera-specific).
• Extrinsic parameters $[R|\mathbf{t}]$: the camera's position and orientation.
• The projection can be expressed linearly in homogeneous coordinates.
• The intrinsic matrix $K$ is found by camera calibration (covered in a later chapter); to begin with, check your understanding with the exercises in the next chapter.