Multi-Modal Perception and Planning Based Robotic Manipulation for Complex and Dynamic Real-World Tasks

Sammanfattning

This thesis investigates intelligent robotic manipulation in dynamic and unstructured environments by integrating multi-modal perception and large language models (LLMs) for high-level reasoning. While advances in perception, planning, and control have improved robotic autonomy, current systems often struggle with executing complex tasks due to limited adaptability and task understanding. Recent progress in LLMs enables new forms of zero-shot reasoning and planning, without the need to train any models or perform additional programming. This work leverages zero-shot chain-of-thought (CoT) prompting to interpret natural language and convert it into structured symbolic action plans. The system is implemented on a Universal Robots UR5e 6-DOF robotic manipulator. A 3D camera serves as the visual input to detect objects using pre-trained object detection models, while grasp point estimation is handled by AnyGrasp, a learning-based module that predicts robust 6D grasp poses based on visual input. This allows the system to perceive, reason, and act in real time. A graphical user interface is also developed to provide an intuitive way for users to input commands and monitor execution. The framework is evaluated across multiple manipulation tasks, such as moving objects along different axes (x/y/z), sorting, stacking, spatial positioning, and handling multiple objects, using several LLMs (GPT-4o, DeepSeek, Claude, and LLaMA) to compare reasoning quality. Although CoT reasoning enhances the interpretability and modularity of generated plans, ablation studies show that a simplified pipeline without CoT achieves higher task success rates, likely because LLMs occasionally hallucinate or overcomplicate reasoning even for trivial tasks, leading to failures. This research contributes a transparent and extensible architecture for voice-controlled robotic manipulation, demonstrating the promise and limitations of symbolic reasoning via LLMs in real-world robotic systems.