# Depth cameras for robot vision: choosing and using one

> Stereo, structured light and time-of-flight depth cameras compared, with accuracy limits, point cloud filtering, ROS 2 setup and India buying advice.

Canonical: https://roboticsengineering.in/depth-camera-robot-vision-guide/ | Published: 2026-10-28 | Updated: 2026-10-28 | Author: Jignesh Panchal

**A depth camera turns a robot from a machine that sees colour into one that sees distance.** That extra dimension is what makes bin picking, obstacle avoidance and 3D inspection possible on a small budget.

This guide explains how the three main depth technologies work, which specifications predict real-world performance, where each type fails, and how to get usable point clouds in ROS 2. It also covers the surfaces that break every depth sensor and the buying questions that matter in India.

## Three ways to measure depth

| Technology | How it works | Strengths | Weaknesses |
| --- | --- | --- | --- |
| Active stereo | Two cameras plus a projected pattern | Good range outdoors, colour aligned | Struggles in direct sun, needs texture or pattern |
| Structured light | Projected dot pattern, triangulated | High accuracy at close range | Short range, sunlight washes the pattern out |
| Time of flight | Measures light travel time per pixel | Works on plain surfaces, compact | Lower resolution, multi-path errors near corners |

Most affordable robot cameras are active stereo, because the same hardware gives colour, depth and sometimes an onboard processor. Structured light units are the choice for close-range inspection where millimetre accuracy matters more than distance.

## Specifications that predict performance

- **Minimum range.** Often overlooked. Many cameras cannot see closer than 200 to 300 mm, which matters for a gripper-mounted camera.
- **Range at accuracy.** Depth error grows roughly with the square of distance for stereo. A camera quoted at two percent error at one metre may be far worse at four.
- **Field of view.** Wide views see more but spread pixels over a larger area, reducing angular resolution.
- **Depth resolution.** Often lower than colour resolution, and the depth image may be upsampled to match.
- **Interface bandwidth.** Uncompressed depth at 30 frames per second saturates USB 3 quickly; plan the cabling and the host controller.
- **Global shutter.** Matters whenever the robot or the part is moving.

| Task | Typical working distance | Depth accuracy needed |
| --- | --- | --- |
| Bin picking | 400 to 900 mm | 2 to 5 mm |
| Pallet detection | 1 to 3 m | 10 to 30 mm |
| Obstacle avoidance | 0.3 to 4 m | 50 mm is often enough |
| Surface inspection | 150 to 400 mm | Under 1 mm |
| Docking | 0.3 to 1.5 m | 5 to 10 mm |

Write your requirement as a triplet: distance, accuracy and frame rate. A camera that meets two of the three is the wrong camera, and no amount of filtering recovers missing accuracy.

## Surfaces that break depth sensing

| Surface | What happens | Workaround |
| --- | --- | --- |
| Polished metal | Specular reflection returns no depth | Diffuse lighting, matte fixture, different angle |
| Clear plastic or glass | Light passes through, depth is wrong | Detect separately, use structured light at close range |
| Black rubber or felt | Absorbs the projected pattern | Raise exposure, add background contrast |
| Thin edges | Depth bleeds between foreground and background | Use intensity edges, filter flying pixels |
| Direct sunlight | Stereo pattern washed out | Shade the cell, use ToF or lidar |
| Corner or cavity | Multi-path reflections give false depth | Change viewpoint, reject outliers |

Every one of these failures is physical, not a software bug. If a part is shiny, no filter will invent the missing returns; you change the lighting, the angle, or the sensor technology.

## Calibration and alignment

Depth cameras need intrinsic calibration for the depth sensor and the colour sensor, plus the transform between them. Factory calibration is usually good enough to start and not good enough for measurement.

- Verify the factory intrinsics by measuring a known object at a known distance.
- Check the colour-to-depth alignment by looking at a printed checkerboard edge; misalignment appears as a coloured halo.
- Calibrate the camera-to-robot transform separately, ideally with a target on the gripper.
- Recalibrate after a knock; a camera that has taken a hit rarely keeps its alignment.

## From depth image to a usable point cloud

A raw cloud is noisy and large. The processing chain that works in practice is short: crop to the region of interest, downscale, remove statistical outliers, then segment the plane or cluster the objects.

```
import open3d as o3d

cloud = o3d.io.read_point_cloud('scene.ply')

# crop to the bin volume
bbox = o3d.geometry.AxisAlignedBoundingBox([-0.3, -0.3, 0.4], [0.3, 0.3, 1.0])
cloud = cloud.crop(bbox)

# downscale and clean
cloud = cloud.voxel_down_sample(0.003)
cloud, _ = cloud.remove_statistical_outlier(nb_neighbors=20, std_ratio=2.0)

# find the dominant plane, usually the table or bin floor
plane, inliers = cloud.segment_plane(0.01, 3, 200)
objects = cloud.select_by_index(inliers, invert=True)
```

Voxel size is the single most useful parameter. Three millimetres is a sensible start for bin picking at half a metre; larger voxels speed everything up but erase small features.

## ROS 2 integration

Most depth cameras ship a ROS 2 driver that publishes an image, a camera info message and a point cloud. The usual layout is one node for the driver and a second for point cloud conversion, with the transform published by a static transform publisher.

```
ros2 run realsense2_camera realsense2_camera_node --ros-args \
  -p pointcloud.enable:=true \
  -p align_depth.enable:=true

ros2 run tf2_ros static_transform_publisher \
  0 0 0.2 0 0 0 base_link camera_link
```

Set the frame correctly before anything else. A point cloud in the wrong frame produces a scene that looks plausible and is positioned wrongly, which is much harder to debug than an empty cloud.

- Keep depth and colour timestamps aligned; use the driver option rather than resampling later.
- Reduce resolution before raising frame rate; most picking tasks run well at 15 frames per second.
- Use compressed transport only for visualisation, never for control.
- Log one bag per failure case; a bag is worth an hour of remote guessing.

## Mounting and lighting

Mount the camera rigidly. Vibration from a robot arm blurs stereo matching and produces depth holes, and a bracket that flexes by a millimetre moves the depth by centimetres at range.

- Use a metal bracket, not a printed one, for any camera above a few hundred grams.
- Add a light shield if the cell has windows or bright overhead lamps.
- Mount the camera at an angle to shiny parts rather than straight on.
- Keep the camera cable away from motor phase wires.

## Buying in India

Depth cameras are imported, so price is driven by duty, warranty and support as much as by the sensor. Buy from a seller who will replace a unit without a six-week round trip, and check that the ROS 2 driver version you need is supported by the firmware shipped.

Two practical alternatives: a pair of synchronised global shutter cameras with a stereo library, which is cheaper and much more work, or a 2D lidar for planar obstacle avoidance, which solves a different problem but often enough of it.

Related: [robot vision camera guide](https://roboticsengineering.in/robot-vision-camera-guide-india/), [OpenCV camera calibration](https://roboticsengineering.in/camera-calibration-opencv-python/), [vision guided robotics](https://roboticsengineering.in/vision-guided-robotics-india/), [OpenCV on Raspberry Pi](https://roboticsengineering.in/opencv-raspberry-pi-robot-vision/), [machine vision lighting](https://roboticsengineering.in/machine-vision-lighting-guide/), [Raspberry Pi versus Jetson](https://roboticsengineering.in/raspberry-pi-vs-jetson-robotics/).

**What is the difference between a depth camera and a lidar?**

A depth camera gives a dense 2D grid of distances over a wide field of view and usually includes colour. A lidar scans a plane or a volume with very accurate ranges but far fewer points and no colour. They solve different problems and often work together.

**Why is my point cloud full of holes?**

Usually the surface is shiny, transparent or very dark, or the scene has direct sunlight. Change the lighting or angle, or use a sensor technology that suits the surface, before adding filters.

**Do I need to calibrate a depth camera?**

Check the factory calibration against a known distance, and always calibrate the camera-to-robot transform. Factory intrinsics are usually close but not accurate enough for measurement.

**How much depth accuracy do I need for bin picking?**

Around two to five millimetres at the working distance for typical parts. Check the manufacturer accuracy curve at your actual distance, not the headline figure at one metre.

**Can I build a depth camera from two webcams?**

Yes, with synchronised global shutter cameras and a stereo matching library. It is cheaper in hardware and much more expensive in engineering time, and rolling shutter webcams will produce poor results on a moving robot.

Depth streams are heavy, so the host matters; [setting up a Jetson Orin Nano](https://roboticsengineering.in/jetson-orin-nano-robot-vision-setup/) covers capture and TensorRT on-device.

Depth cameras add bandwidth quickly, and the [interface comparison](https://roboticsengineering.in/industrial-camera-interfaces-guide/) shows where USB3 stops being enough.

## Sources and further reading

- [Intel RealSense documentation](https://dev.intelrealsense.com/docs) — depth camera specifications and SDK reference.
- [Open3D documentation](https://www.open3d.org/docs/release/) — point cloud filtering and segmentation.
- [ROS 2 documentation](https://docs.ros.org/en/rolling/) — image transport and point cloud message conventions.