Camera Matrix

Overview of the camera model

The process of projecting a point in 3D space onto a 2D image is expressed as a chain of several coordinate transformations.

Intrinsic matrix K

The intrinsic parameters describe characteristics that are specific to the camera itself.

Extrinsic parameters [R|t]

The extrinsic parameters describe the position and orientation of the camera relative to the world coordinate system.

Major coordinate-system conventions (OpenCV, OpenGL, DirectX/Unity)

This article's convention of $X_c$ right, $Y_c$ down, $Z_c$ forward is the OpenCV convention. Other major conventions exist, and which is the majority or minority changes from field to field. The handedness (right- or left-handed) also differs between conventions.

$X$ right, $Y$ up, $Z$ into the scene (forward) is left-handed, which is the DirectX / Unity convention. Starting from the OpenCV convention ($Y$ down) and flipping only $Y$ to point up flips a single axis, turning right-handed into left-handed and yielding this convention (Unreal is also left-handed, but assigns the axes differently, with $X$ forward and $Z$ up). By contrast, the OpenCV and OpenGL conventions are related by flipping $Y$ and $Z$ together (a 180° rotation about the $X$ axis), so both are right-handed. Because this article is in a computer-vision context, it adopts the OpenCV convention.

Even for the same $[R|\mathbf{t}]$, the picture looks different depending on which coordinate system you hold fixed while viewing it. Figure 3 was world-anchored (hold the world fixed and draw the camera's pose within it; this is the view for tasks that deal with camera pose and trajectory). Conversely, if you fix the camera at the origin, then the world coordinate system is placed by rotating and translating it via $[R|\mathbf{t}]$. This is exactly the coordinate-transform equation $\mathbf{P}_c = R\,\mathbf{P}_w + \mathbf{t}$ (world→camera), which corresponds to the view matrix in rendering (the projection to 2D is the later step of dividing by $Z_c$).

Camera matrix P

The camera matrix (projection matrix) is the combination of the intrinsic and extrinsic parameters.

Expanding:

Common points of confusion (pitfalls)

The same symbol or name often refers to different things in different sources, and this is one of the biggest factors that make learning computer vision hard. Online sources appear to contradict one another mostly because many of them do not state which convention they adopt. First, the names themselves clash across fields:

Because the same name can refer to different matrices, it is safest to first check the definition used in that particular source. Next come mismatches in how equations are written and in conventions:

• Row vectors vs. column vectors: the column-vector convention $\mathbf{p}' = R\,\mathbf{p}$ (this article, mathematics, OpenCV) or the row-vector convention $\mathbf{p}'^{\top} = \mathbf{p}^{\top} R^{\top}$ (writing the same transform with row vectors transposes the matrix too; DirectX, etc.)? A transpose swaps the whole equation. On top of that, whether the matrix memory layout is row-major (C / NumPy) or column-major (OpenGL / Eigen default) is yet another separate issue.
• Direction of the transform: world→camera ($R,\ \mathbf{t}$) or camera→world ($R^{\top},\ \mathbf{C} = -R^{\top}\mathbf{t}$)? The same "extrinsic parameters" can refer to opposite directions. (Figures 3 and 4 draw the same world→camera transform with a different side held fixed; they are not inverses of each other.)
• Active vs. passive rotation: rotating the point (active) or rotating the coordinate system (passive)? $R$ and $R^{\top}$ swap.
• Handedness and axes of the coordinate system: right- / left-handed, $Y$ up / down, $Z$ forward / backward (the comparison table above).
• Image coordinates: origin at the top-left (CV) or the bottom-left (OpenGL)? Is the pixel center at integer coordinates or at $+0.5$?
• Representation of rotation: the rotation order of Euler angles; whether a quaternion has $w$ first or last; Hamilton / JPL convention.

Summary of coordinate systems

Code example (Python)

import numpy as np

# Intrinsic parameters
fx, fy = 800, 800
cx, cy = 320, 240
K = np.array([
    [fx,  0, cx],
    [ 0, fy, cy],
    [ 0,  0,  1]
], dtype=float)

# Extrinsic parameters (a world→camera example: R = 45 deg rotation about Z, t = (1, 0, 5); the camera position is -R^T t, not t)
theta = np.radians(45)
R = np.array([
    [np.cos(theta), -np.sin(theta), 0],
    [np.sin(theta),  np.cos(theta), 0],
    [0,              0,             1]
])
t = np.array([[1.0], [0.0], [5.0]])

# Extrinsic matrix [R|t]
Rt = np.hstack([R, t])

# Camera matrix P = K[R|t]
P = K @ Rt
print("Camera Matrix P:")
print(P)

# Project a 3D point
P_world = np.array([0, 0, 0, 1])  # world origin
p_homogeneous = P @ P_world
p = p_homogeneous[:2] / p_homogeneous[2]
print(f"Projected point: ({p[0]:.1f}, {p[1]:.1f})")

# Camera position (world coordinates)
camera_position = -R.T @ t.flatten()
print(f"Camera position: {camera_position}")

# Using OpenCV
import cv2
# When using cv2.projectPoints()
rvec, _ = cv2.Rodrigues(R)  # rotation matrix -> rotation vector
tvec = t  # tvec is not the camera position; it is the world origin expressed in the camera frame (= t)
points_3d = np.array([[0, 0, 0], [1, 0, 0]], dtype=np.float32)  # the origin and a point that shows the rotation
points_2d, _ = cv2.projectPoints(points_3d, rvec, tvec, K, None)
print(points_2d.reshape(-1, 2))  # [[480. 240.] [593.1 353.1]] (the origin matches the matrix computation above)

Summary

• The camera matrix $P = K[R|\mathbf{t}]$ is a $3 \times 4$ matrix representing the 3D→2D projection.
• Intrinsic parameters $K$: focal length and principal point (camera-specific).
• Extrinsic parameters $[R|\mathbf{t}]$: the camera's position and orientation.
• In homogeneous coordinates the projection is a matrix product followed by division by the third component.
• The intrinsic matrix $K$ is found by camera calibration (covered in a later chapter); the exercises chapter is in preparation.