Real-Time Detection & Tracking on the Edge

The problem

The detection model was accurate but ran at X ms on the target hardware — far too slow to be used by the controller, which meant the robot reacted to obstacles a full cycle late.

What I built

A detection and tracking pipeline rebuilt around the latency budget: a YOLO-family detector converted to TensorRT with INT8 quantization, feeding a ByteTrack association stage that keeps identities stable across frames.

How it works

• PyTorch training, exported through ONNX to a TensorRT engine built for the specific Jetson target.
• INT8 post-training quantization with a calibration set drawn from real deployment footage rather than the training set.
• ByteTrack for association; CUDA-side pre-processing to keep the image off the CPU.
• Latency measured end-to-end (capture to published detection), not just model forward time.

How it was tested

A benchmark harness pins the engine to the target device and fails CI if p99 latency regresses. Accuracy is re-checked against the held-out set after every quantization change, since quantization can silently cost recall on small objects.

Results

• XX FPS sustained on the Jetson, inside the control loop.
• Latency reduced from X ms to Y ms.
• Under Z% mAP loss from INT8 quantization.

What I learned

• Quantization error was not uniform — small distant objects lost far more recall than large ones. Averaged mAP hid this completely until I broke the metric down by object size.
• Calibrating on training data gave misleadingly good numbers. Using real deployment footage for calibration mattered more than the quantization scheme itself.