# OpenCV on a Raspberry Pi: robot vision that actually works

> Install OpenCV, capture frames with Picamera2, detect an object by colour, turn pixels into motor commands, and make vision reliable before clever.

Canonical: https://roboticsengineering.in/opencv-raspberry-pi-robot-vision/ | Published: 2026-10-27 | Updated: 2026-10-28 | Author: Jignesh Panchal

**Robot vision is mostly plumbing.** The algorithm that finds a coloured object is twenty lines. The work is getting a clean frame, consistent lighting, the right colour space and a control loop that does not oscillate. This guide sets up OpenCV on a Raspberry Pi, connects the camera properly, and turns a detection into a motor command you can trust.

## Install OpenCV on Raspberry Pi OS

Use the Debian package for a stable install, or a virtual environment if you need a newer version. The Debian package is compiled for the Pi and avoids long build times.

```
sudo apt update
sudo apt install -y python3-opencv python3-picamera2 python3-numpy
python3 -c "import cv2, numpy; print(cv2.__version__, numpy.__version__)"
```

If you prefer pip, create a virtual environment that can still see the system Picamera2 package:

```
python3 -m venv --system-site-packages ~/robotenv
source ~/robotenv/bin/activate
pip install opencv-python numpy pyserial
```

Do not install both and wonder which one is loading. Run `python3 -c "import cv2; print(cv2.__file__)"` and confirm the path matches the environment you intend to use.

## Get frames the right way

On a Pi with a CSI camera, `cv2.VideoCapture(0)` often fails or gives poor results because the camera stack is libcamera, not V4L2 in the way OpenCV expects. Use Picamera2 and hand the array to OpenCV.

```
from picamera2 import Picamera2
import cv2, time

picam2 = Picamera2()
config = picam2.create_preview_configuration(
    main={"size": (640, 480), "format": "RGB888"})
picam2.configure(config)
picam2.start()
time.sleep(2)

while True:
    rgb = picam2.capture_array()
    frame = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
    cv2.imshow("view", frame)
    if cv2.waitKey(1) == 27:
        break

cv2.destroyAllWindows()
picam2.stop()
```

The colour conversion is not optional. Picamera2 returns RGB; OpenCV assumes BGR. Skip the conversion and every red object looks blue.

## Find an object by colour

HSV separates colour from brightness, which makes thresholds far more stable than RGB when the light changes. This example finds the largest red region and returns its centre.

```
import cv2, numpy as np

def find_red_centroid(frame_bgr):
    hsv = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2HSV)
    lower1 = np.array([0, 120, 70])
    upper1 = np.array([10, 255, 255])
    lower2 = np.array([170, 120, 70])
    upper2 = np.array([180, 255, 255])
    mask = cv2.inRange(hsv, lower1, upper1) | cv2.inRange(hsv, lower2, upper2)
    mask = cv2.erode(mask, None, iterations=2)
    mask = cv2.dilate(mask, None, iterations=2)
    contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL,
                                   cv2.CHAIN_APPROX_SIMPLE)
    if not contours:
        return None
    c = max(contours, key=cv2.contourArea)
    if cv2.contourArea(c) < 800:
        return None
    m = cv2.moments(c)
    if m["m00"] == 0:
        return None
    cx = int(m["m10"] / m["m00"])
    cy = int(m["m01"] / m["m00"])
    return cx, cy
```

Red wraps around the hue circle, which is why there are two ranges. Green and blue need only one. Tune the saturation and value floors on the actual robot, under the actual lighting, with the actual object.

## Turn a pixel into a motor command

For a pan servo or a differential drive, the useful signal is the horizontal error between the object centre and the frame centre. Add a deadband so the robot does not twitch when the object is almost centred, and clamp the output so a bad detection cannot command full speed.

```
FRAME_W = 640
DEADBAND = 20
KP = 0.004

def steering(cx):
    error = cx - FRAME_W // 2
    if abs(error) < DEADBAND:
        return 0.0
    turn = KP * error
    return max(-0.4, min(0.4, turn))

# in the loop
point = find_red_centroid(frame)
if point:
    turn = steering(point[0])
    ser.write(f"M 0.25 {turn:.3f}\n".encode())
else:
    ser.write(b"S\n")
```

The gain value is a starting point, not a universal constant. If the robot weaves, reduce it. If it reacts slowly, increase it slightly. Test at low speed and with the wheels off the ground first.

## Make vision reliable before you make it clever

- **Lock exposure and white balance.** Auto settings change the image between frames and make thresholds drift. Set them once for the room and keep them fixed during a run.
- **Light the object, not the room.** A small LED ring or a shaded enclosure does more for reliability than a better algorithm.
- **Process a smaller frame.** 640×480 is usually enough for tracking. Upscaling costs time and adds no useful information.
- **Work on a region of interest.** If the object can only appear in the lower half of the frame, crop before processing and save half the CPU.
- **Filter in time, not just space.** Require two or three consecutive detections before acting on a target, and keep the last known position for a few frames when detection drops.
- **Log frames when it fails.** Save a short video of every failed run. The cause is almost always visible: glare, motion blur, wrong white balance or a shadow.

## Realistic performance expectations

Numbers depend on resolution, code and cooling, so treat these as planning guides rather than benchmarks.

| Task | Raspberry Pi 4 | Raspberry Pi 5 |
| --- | --- | --- |
| Colour threshold at 640×480 | 15–25 fps | 30+ fps |
| Haar cascade face or marker detection | 5–10 fps | 10–20 fps |
| Small neural network at 320×320 | 2–6 fps | 8–20 fps |
| Two cameras, stereo depth | Difficult | Practical at low resolution |

If your loop runs at 8 fps, do not build a control loop that assumes 30. Design the robot to move slowly enough that a 120 ms reaction time is safe, or move the timing-critical part to a microcontroller.

## When to move beyond colour tracking

Colour tracking is fast, explainable and cheap. It fails when lighting changes, when objects are similar in colour, or when you need to identify a class rather than a blob. Move to ArUco markers when you need a known target with a pose. Move to a small neural network when the object class matters more than its colour. Move to stereo or a depth camera when distance accuracy matters, because a single camera only gives you distance if you know the object’s real size.

Related: [Raspberry Pi 5 robot guide](https://roboticsengineering.in/raspberry-pi-5-robot-guide/), [Raspberry Pi robot guide](https://roboticsengineering.in/raspberry-pi-robot-guide/), [choosing distance sensors](https://roboticsengineering.in/lidar-vs-ultrasonic-vs-ir-robots/), [PID for line followers](https://roboticsengineering.in/pid-control-line-follower-explained/), [I2C vs SPI vs UART](https://roboticsengineering.in/i2c-spi-uart-robotics/).

**Can OpenCV run on a Raspberry Pi 5?**

Yes. The Debian python3-opencv package works on 64-bit Raspberry Pi OS, and pip wheels are also available. Simple colour tracking runs comfortably at 640×480; heavier neural networks need a smaller input size or an accelerator.

**Why does cv2.VideoCapture(0) not work with my Pi camera?**

The CSI camera is driven by libcamera, which OpenCV does not access as a normal V4L2 device on current Raspberry Pi OS. Use Picamera2 to capture frames and pass the NumPy array to OpenCV.

**Should I detect objects in RGB or HSV?**

Use HSV for colour-based detection because hue stays stable when brightness changes. Use RGB or grayscale when you are feeding a neural network or a template matcher that expects that format.

**How do I make colour detection work under different lights?**

Lock exposure and white balance, add controlled lighting near the object, retune the HSV thresholds in the actual environment, and require several consecutive detections before acting on a target.

**How many frames per second do I need for robot vision?**

For slow tracking, 10 to 15 frames per second is enough if the robot moves slowly and the control loop has a deadband. For fast line following or catching moving objects, use a microcontroller for the control loop and keep vision as a supervisor.

If you are building the software yourself, start with [OpenCV on Raspberry Pi](https://roboticsengineering.in/robot-vision-camera-guide-india/) and add an industrial camera later.

Frame rates depend heavily on the board: [Raspberry Pi vs Jetson for robot vision](https://roboticsengineering.in/raspberry-pi-vs-jetson-robotics/) compares real throughput.

Once basic vision works, [a YOLO detector](https://roboticsengineering.in/robot-vision-yolo-object-detection/) is the next step.

Consistent lighting matters more than resolution, as the [machine vision lighting guide](https://roboticsengineering.in/machine-vision-lighting-guide/) explains.

Before any distance maths in a vision pipeline, run a proper [chessboard calibration](https://roboticsengineering.in/camera-calibration-opencv-python/) so pixel coordinates mean something.

Once the camera is calibrated you can [detect ArUco markers and estimate pose](https://roboticsengineering.in/aruco-marker-robot-localization/) on the same Pi.

A Raspberry Pi can host a depth camera, but the range and frame rate limits are set out in the [depth camera guide](https://roboticsengineering.in/depth-camera-robot-vision-guide/).

When the Pi runs out of headroom, move the same OpenCV code to [a Jetson Orin Nano](https://roboticsengineering.in/jetson-orin-nano-robot-vision-setup/) and rebuild the inference engine.

## Sources and further reading

- [OpenCV documentation](https://docs.opencv.org/) — colour spaces, contours and camera calibration.
- [Raspberry Pi camera software](https://www.raspberrypi.com/documentation/computers/camera_software.html) — Picamera2 and libcamera reference.
- [Picamera2 manual](https://datasheets.raspberrypi.com/camera/picamera2-manual.pdf) — capture configuration and camera controls.
- [ROS tutorials](https://docs.ros.org/en/rolling/Tutorials.html) — integrating cameras and vision into a robot stack.