← All chapters

Cookbook · Chapter 8

Camera and computer vision

Reading a live webcam, working its pixels, and simple thresholding and color tracking.

What’s realistic in Umfeld: there is a built-in Capture class for live cameras, and full access to pixels — so brightness thresholding and color tracking are a few loops away. Umfeld does not bundle OpenCV, blob detection, or face detection. If you need those, the patterns below (working directly on pixels[]) are the supported path; reach for an external C++ vision library only when you truly need it.

Using a webcam

Capture::list() enumerates devices; init() selects one with a resolution, frame rate, and pixel format. Call reload() each frame before drawing it.

#include "Capture.h"

Capture* cam;

void setup() {
    for (const auto& device : Capture::list()) console(device);

    cam = new Capture();
    cam->init("FaceTime HD Camera", "1280x720", "30", "nv12");
    cam->start();
}

void draw() {
    background(0);
    cam->reload();
    image(cam, 0, 0, width, height);
}

Reading the camera’s pixels

A Capture behaves like a PImage: after reload(), call loadPixels() and read pixels[]. Pull channels out of each pixel with red(), green(), blue().

cam->reload();
cam->loadPixels();
color_t c = cam->pixels[y * cam->width + x];
float brightness = (red(c) + green(c) + blue(c)) / 3.0f;

Thresholding video

Turn the frame into pure black and white at a brightness cut-off — the classic first step of any vision pipeline. Draw into the window’s own pixel buffer:

void draw() {
    cam->reload();
    cam->loadPixels();
    loadPixels();

    const float threshold = map(mouseX, 0, width, 0, 255);
    for (int i = 0; i < width * height; i++) {
        color_t c = cam->pixels[i];
        float   b = (red(c) + green(c) + blue(c)) / 3.0f;
        pixels[i] = (b > threshold) ? color(255) : color(0);
    }
    updatePixels();
}

Color tracking

Find the average position of every pixel near a target color — enough to follow a brightly colored object without any library:

void draw() {
    background(0);
    cam->reload();
    cam->loadPixels();
    image(cam, 0, 0, width, height);

    const color_t target = color(255, 0, 0);   // tracking red
    long sumX = 0, sumY = 0, count = 0;

    for (int y = 0; y < cam->height; y++) {
        for (int x = 0; x < cam->width; x++) {
            color_t c = cam->pixels[y * cam->width + x];
            float dr = red(c)   - red(target);
            float dg = green(c) - green(target);
            float db = blue(c)  - blue(target);
            if (dr*dr + dg*dg + db*db < 60*60) {   // within tolerance
                sumX += x; sumY += y; count++;
            }
        }
    }

    if (count > 0) {
        float cx = (float) sumX / count * width  / cam->width;
        float cy = (float) sumY / count * height / cam->height;
        noFill();
        stroke(0, 255, 0);
        strokeWeight(4);
        circle(cx, cy, 40);
    }
}

The same idea — scanning pixels[] and collecting matching points — is the basis of a simple blob tracker. Build up the bounding box of the matching pixels instead of just their average to draw a box around the object.