Language-Conditioned Robot Manipulation

The problem

Teaching a manipulator a new task meant writing a new script and collecting new training data for every object. Operators on the floor could describe what they wanted in plain language, but nothing in the stack could act on that — so every small change needed an engineer.

What I built

A pipeline that turns a free-form instruction into collision-aware motion. An LLM decomposes the sentence into ordered subgoals, an open-vocabulary detector grounds each referenced object in the live scene, and MoveIt 2 plans and executes the motion with a Cartesian servoing fallback when the plan is refused.

How it works

• LLM planner prompted to emit a strict subgoal schema, so a malformed response is rejected rather than executed.
• Open-vocabulary grounding with CLIP-style embeddings, matching detected regions against the phrase rather than a fixed class list.
• Depth-camera point cloud segmentation (PCL, Open3D) to recover a 6-DoF grasp pose per candidate object.
• ROS 2 action server wrapping MoveIt 2; the planner and the perception stack run as separate composable nodes to keep the control path real-time.
• PyTorch for the grasp ranker, exported to TensorRT for on-robot inference.

How it was tested

Instruction parsing is covered by unit tests against a fixture set of phrasings. The full pipeline runs in Gazebo in CI on every merge, and a nightly job replays recorded scenes through the perception stack to catch grounding regressions.

Results

• XX% task success across N unseen instruction phrasings, with no per-object training.
• Grounding accuracy of XX% on a held-out set of N cluttered scenes.
• End-to-end latency from instruction to first motion: X seconds.

What I learned

• The LLM was the least fragile part. Grounding was where things broke — ambiguous references like "the one on the left" need the scene geometry, not a better prompt.
• I initially let the planner retry on failure without bound. A refused plan would loop until the operator intervened; capping retries and surfacing the failure was better than hiding it.
• Next time I would build the evaluation set of phrasings before the pipeline, not after. It would have shown much earlier which failures were grounding and which were planning.

My role

Solo project. I designed the subgoal schema, built the grounding and grasp-ranking stack, and did the integration onto the arm.