#4640 Programmable Robot Dog w/AI & SLAM Mapping Navigation

#4640 What fun and useful things could I do with PuppyPi Robot Dog with ChatGPT Large AI Models AI Embodied Intelligence ROS Robotic Dog Vision Scene Voice Understanding SLAM Mapping Navigation Bionic Quadruped Robot, Ultimate Kit & RaspberryPi5 8GB?

The Hiwonder PuppyPi Ultimate Kit (powered by a Raspberry Pi 5 with 8GB RAM, ROS2, LiDAR, FPV camera, and integrated LLM voice pipeline) is a phenomenal sandbox for Embodied AI. Because it bridges high-level natural language reasoning with low-level physical inverse kinematics and SLAM, the possibilities go far beyond simple "fetch" routines.

Here is a breakdown of fun, practical, and advanced engineering projects you can execute with this setup.

1. Autonomous Spatial Mapping & Dynamic Patrols

Combining 2D/3D LiDAR, ROS2 (Nav2 package), and the RPi 5’s processing speed gives you full indoor spatial awareness.

Semantic "Guard Dog" Routine: * How it works: Use SLAM (Gmapping/Cartographer) to build a point-cloud map of your office or home. Program PuppyPi to autonomously patrol specified waypoints.
- The Twist: Connect the FPV camera to a local object detection model (YOLOv8/MediaPipe). If it detects a foreign object, a door left open, or an unrecognized human face during non-operational hours, it stops, uses its voice module to challenge the target, and logs a timestamped snapshot.
Dynamic Obstacle Course Navigation: * Set up physical barriers that change position. Use the ROS2 TEB (Timed Elastic Band) Local Planner with real-time dynamic obstacle avoidance to let the robot re-calculate gaits and pathing on the fly without stopping.

2. True Embodied AI (Voice $\rightarrow$ Intent $\rightarrow$ Spatial Action)

Rather than relying on hardcoded triggers, leverage the Raspberry Pi 5’s multi-threading to run a local/hybrid LLM pipeline.

Context-Aware Physical Assistant:
- Concept: Ask the robot, "Find my keys on the coffee table and tell me if you see them."
- Pipeline: The LLM parses the command $\rightarrow$ breaks it down into a multi-step ROS action sequence (Navigate to Coordinate $X,Y$ $\rightarrow$ Pitch camera up $\rightarrow$ Run vision inference) $\rightarrow$ Returns a spoken response using its voice module: "I'm at the coffee table. I see your keys near the edge."
Multimodal Object Sorting / Retrieval (with optional Robotic Arm):
- If using the robotic arm attachment included in the Ultimate Kit, program the robot to interpret continuous context: "Pick up the red block and move it to the blue target."
- The LLM parses color and task intent, OpenCV handles AprilTag or HSV color coordinates, and the inverse kinematics engine handles the leg stance adjustment and micro-arm positioning.

3. Custom Gait Dynamics & Inverse Kinematics (IK)

PuppyPi’s 8-DOF linkage structure offers an ideal physical testbed for robot kinematics and control theory.

Active Terrain Adaptation via IMU Feedback:
- Write custom Python controllers utilizing the onboard 6-axis IMU. Build a closed-loop feedback loop where, if the robot walks over uneven surfaces (like ramps or cables), it continuously adjusts leg extension heights in real time to maintain a level body axis.
Custom Gait Synthesizer:
- Tweak joint stance timing (trot, walk, amble, bound) by adjusting touchdown/lift-off phase matrices.
- Program specialized trick maneuvers: dynamic bowing, terrain probing, or custom body-tilt responses triggered by gesture controls (via MediaPipe hand tracking).

4. Interactive & Experimental Sandbox Ideas

Project Concept	Primary Tech Stack	Functional Outcome
AprilTag Dynamic Following	OpenCV + ROS2 Control	PuppyPi locks onto a moving AprilTag target mounted on a vehicle or person and follows at a precise set distance ( $d$ ).
Digital Twin Simulation	Gazebo / RViz2	Build complex navigation routines and gait modifications safely in a 3D Gazebo environment before pushing code to the physical RPi 5 hardware.
Local LLM Offline Agent	Ollama / Llama.cpp + ROS	Run a quantized local LLM directly on a local node or network server to make PuppyPi fully operational offline without relying on cloud APIs.
Interactive Gestural Dog	MediaPipe + OpenCV	Control directional gait vectors entirely through real-time hand gestures caught on the FPV camera (e.g., Palm = Stop, Point Right = Turn, High Five = Sit).

5. Recommended Getting-Started Roadmap

Hardware & Kinematics Baseline: Verify servo calibration and test basic posture adjustment scripts using Hiwonder’s base Python APIs.
Visual Processing Node: Run local OpenCV loops (face tracking, AprilTag reading) via the FPV camera feed.
LiDAR & SLAM Mapping: Launch RViz on a workstation connected to the same ROS DOMAIN ID as the RPi 5 to map your workspace floor plan.
LLM Agent Integration: Pipeline voice-to-text input into an LLM node that maps intent directly to ROS2 /cmd_vel or goal action servers.

How can Ollama be installed and run on the Pi5 with 8 GB?

Installing and running Ollama on a Raspberry Pi 5 (8 GB) is straightforward because Ollama natively supports ARM64 architecture (aarch64).

On the Pi 5's Cortex-A76 cores, lightweight models run smoothly for robotics, automation pipelines, or local chat.

Prerequisites

64-bit OS: Ensure you are running a 64-bit OS (e.g., Raspberry Pi OS 64-bit or Ubuntu 64-bit). You can check with:
uname -m # Should return 'aarch64'
Active Cooling: Running LLM inference pushes all 4 CPU cores to 100%. An active cooler or thermal fan is strongly recommended to prevent thermal throttling.

Step 1: Install Ollama

Open a terminal on your Pi 5 and update package repositories, then run the official Ollama installer:

# Update system packages
sudo apt update && sudo apt upgrade -y

# Download and execute the official installation script
curl -fsSL https://ollama.com/install.sh | sh

The script will automatically detect the aarch64 ARM architecture, install the Ollama binary, and configure a systemd background service.

Once finished, verify the installation:

ollama --version

Step 2: Test Run Your First Model

Ollama automatically downloads and loads models upon execution. Start with a lightweight model to test inference performance:

ollama run tinyllama

To exit the interactive prompt at any time, type /bye and press Enter.

Recommended Models for Pi 5 (8 GB RAM)

Because the Pi 5 uses CPU RAM as unified memory without dedicated VRAM, selecting the right model size balances generation speed (tokens per second) and reasoning depth:

Model	Memory Footprint	Approximate Speed	Recommended Use Case
qwen2.5:0.5b	~350 MB	~20–30 t/s	Ultra-fast intent parsing, edge robotics, simple classification
`tinyllama` (1.1B)	~630 MB	~15–20 t/s	Lightweight text formatting, fast local testing
llama3.2:1b	~1.3 GB	~10–14 t/s	Great balance of speed and instruction following
llama3.2:3b	~2.0 GB	~4–7 t/s	General conversational assistant & solid reasoning
`phi3:mini` (3.8B)	~2.3 GB	~3–5 t/s	Strong technical and mathematical logic
deepseek-r1:1.5b	~1.1 GB	~8–12 t/s	Compact reasoning/chain-of-thought tasks

Note on 7B/8B Models: You can run quantized 7B models like mistral or deepseek-r1:7b on an 8 GB Pi 5, but expect generation speeds around 1 to 2 tokens/sec. They are best reserved for background or non-realtime batch jobs.

Step 3: Interacting via REST API (Python / Automation)

Ollama runs an HTTP REST server in the background on port 11434. You can send JSON requests programmatically from Python scripts or local applications:

import requests

response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3.2:1b",
        "prompt": "Summarize the primary purpose of ROS2 in one sentence.",
        "stream": False
    }
)

print(response.json()["response"])

Useful System Commands

List locally downloaded models:
ollama list
Remove a downloaded model:
ollama rm model_name
Check Ollama service status:
sudo systemctl status ollama
Restart the background service:
sudo systemctl restart ollama

Would an IBM Granite or Google Gemma model work on this hardware?

Yes, absolutely—specific variants of both IBM Granite and Google Gemma run on a Raspberry Pi 5 (8 GB RAM). However, because the Pi 5 relies entirely on ARM CPU cores and unified system RAM, performance hinges on choosing the smaller, heavily optimized parameter sizes.

1. IBM Granite Models

IBM’s Granite series is built specifically for enterprise tasks, function calling, structured JSON output, and code execution—making it an excellent match for local robotics pipelines.

Best Choice: granite3-moe (1B or 3B)
- Size on Disk: ~1 GB to 2 GB
- RAM Usage: ~1.5 GB
- Speed: ~12–18 tokens/sec
- Why it works: IBM’s Mixture-of-Experts (MoE) architecture only activates a fraction of its total parameters per token, giving you lower latency on CPU-only hardware.
ollama run granite3-moe:3b
Great Choice: granite3-dense:2b
- Size on Disk: ~1.6 GB
- RAM Usage: ~2.0 GB
- Speed: ~8–12 tokens/sec
- Why it works: Highly capable at structured tool-calling and parsing instructions for external APIs or ROS topics.
ollama run granite3-dense:2b
Usable but Slow: granite3-dense:8b
- Size on Disk: ~4.9 GB
- Speed: ~1.5–2.5 tokens/sec
- Verdict: Fits in the 8 GB RAM envelope, but generation speed will introduce noticeable lag in real-time interactions.

2. Google Gemma Models

Google’s Gemma family is known for strong reasoning and conversational quality, but its dense architecture requires careful model selection on single-board computers.

Best Choice: gemma2:2b
- Size on Disk: ~1.6 GB
- RAM Usage: ~2.2 GB
- Speed: ~8–12 tokens/sec
- Why it works: Excellent instruction following and reasoning for its footprint. Fits comfortably alongside background OS tasks and ROS nodes.
ollama run gemma2:2b
Micro Choice: gemma3:1b / functiongemma (270M)
- Size on Disk: ~300 MB to 1 GB
- Speed: ~20+ tokens/sec
- Why it works: Ultra-fast responses for low-overhead embedded tasks.
Usable but Slow: gemma2:9b
- Size on Disk: ~5.4 GB
- Speed: ~1 to 2 tokens/sec
- Verdict: Takes up most of your available 8 GB unified memory once loaded and runs slowly on CPU.
Incompatible: gemma2:27b
- Verdict: Will not run. Exceeds the 8 GB system RAM limit.

Recommended Hardware Pairing Summary

Model	Ollama Command	RAM Footprint	Typical Speed	Ideal Robotics Task
Granite 3 MoE (3B)	ollama run granite3-moe:3b	~1.8 GB	~14 t/s	Low-latency state management & tool activation
Granite 3 Dense (2B)	ollama run granite3-dense:2b	~2.0 GB	~10 t/s	ROS node command conversion & JSON parsing
Gemma 2 (2B)	ollama run gemma2:2b	~2.2 GB	~9 t/s

Could a laptop running a Python application with Ollama control the vehicle via WiFi?

Yes, absolutely. Running Ollama and your primary Python logic on a powerful laptop while communicating with the PuppyPi over Wi-Fi is actually the recommended architectural pattern for robotics development.

Offloading the LLM inference to your laptop frees up the Raspberry Pi 5’s CPU entirely for low-latency tasks like inverse kinematics (IK), ROS2 node management, real-time servo control, and LiDAR SLAM processing.

How the Architecture Works

┌──────────────────────────────┐              Wi-Fi Network              ┌──────────────────────────────┐
│        YOUR LAPTOP           │ ──────────────────────────────────────> │         PUPPYPI (RPi 5)      │
│                              │                                         │                              │
│  [Python Script]             │   Option A: ROS2 DDS Topics             │  [ROS2 Control Nodes]        │
│        │                     │   Option B: HTTP / REST Requests        │        │                     │
│        ▼                     │   Option C: WebSockets                  │        ▼                     │
│  [Ollama Local Engine]       │                                         │  [Servo Control / Kinematics]│
└──────────────────────────────┘                                         └──────────────────────────────┘

3 Ways to Implement the Control Link

Method 1: Native ROS2 DDS over Wi-Fi (Most Robust)

Because PuppyPi runs ROS2 natively, both your laptop and the Pi 5 can exist on the same ROS2 network graph if they are on the same Wi-Fi subnet.

Set standard environment variables on both laptop and Pi 5:
export ROS_DOMAIN_ID=42
Publish directly to motion topics: Your laptop’s Python script queries Ollama, parses the intent into a ROS message (e.g., geometry_msgs/msg/Twist or PuppyPi’s native posture topic), and publishes it directly over Wi-Fi. The Pi 5 receives it with zero added middleware.

Method 2: HTTP / REST API (Easiest to Setup)

You can run a lightweight API server (like FastAPI or Flask) on the Pi 5 that exposes movement endpoints.

On the Pi 5: Expose endpoints mapped to Hiwonder’s Python SDK:

# Simple FastAPI snippet on Pi 5
from fastapi import FastAPI
import puppy_kinematics # PuppyPi SDK

app = FastAPI()

@app.post("/cmd/walk")
def walk(direction: str, duration: float):
    # Trigger leg gait routines
    return {"status": "executing"}

On your Laptop: Run Python + Ollama, use structured outputs (JSON mode), and issue POST requests:

import requests
import ollama

# Get structured action from local Ollama model
response = ollama.chat(
    model="granite3-dense:2b",
    messages=[{"role": "user", "content": "Walk forward for 3 seconds."}],
    format="json"
)

# Send execution command to Pi 5 IP
requests.post("http://192.168.1.150:8000/cmd/walk", json=response.message.content)

Method 3: WebSockets (Best for Low Latency / Continuous Control)

If you want real-time streaming control—such as passing continuous joystick vectors or immediate safety stops—a bi-directional WebSocket connection provides sub-10ms communication over local 5GHz Wi-Fi.

Key Performance Tips

Use 5GHz Wi-Fi: Quadruped balance and obstacle-avoidance loops require reliable timing. A 5GHz connection or dedicated router minimizes packet jitter.
Keep High-Frequency Loops Local: Run safety-critical code (e.g., stopping when the LiDAR detects an obstacle 10cm away) directly on the Pi 5. The laptop should send high-level intent ("Search for the red ball"), while the Pi handles real-time reflexes ("Stop before hitting the wall").

Can the vehicle send SLAM data to the laptop?

Yes, absolutely. Transmitting SLAM and LiDAR data over Wi-Fi is standard practice in ROS2 environments.

Because ROS2 utilizes DDS (Data Distribution Service), any node running on your network can publish topics (like point clouds or occupancy grids), and any other node can subscribe to them across devices seamlessly.

Here is how to stream, visualize, and interact with the PuppyPi's SLAM data on your laptop:

1. ROS2 DDS Network Setup

To allow your laptop to "see" the topic data broadcast by the Pi 5, ensure both devices are on the same Wi-Fi network and share the same domain ID.

Set the Domain ID on both devices:
Add this line to ~/.bashrc on both the Raspberry Pi 5 and your laptop:
export ROS_DOMAIN_ID=42
Verify connectivity from your laptop:
While the SLAM process is active on the PuppyPi, list available topics from your laptop terminal:
ros2 topic list
You should see topics such as /map, /scan (LiDAR), /tf (transforms), and /odom.

2. Visualizing Map & Robot Pose in Real Time (RViz2)

Instead of forcing the Pi 5 to process heavy 3D rendering GUIs, you run RViz2 locally on your laptop GPU to render the map streamed over Wi-Fi.

Launch RViz2 on your laptop:

ros2 run rviz2 rviz2

In the RViz UI, add the following displays:

Map: Set topic to /map. Change durability setting under Topic Quality of Service (QoS) to Transient Local so it receives the complete map buffer.
LaserScan: Set topic to /scan to see live LiDAR point clouds scanning obstacles.
TF / RobotModel: Visualizes the robot's orientation ( $x, y, \theta$ ) relative to the map origin frame.

3. Streaming SLAM Data to Python Apps / LLM Pipelines

If your laptop Python script needs to inspect the map programmatically (e.g., passing obstacle layout data to your local Ollama model or path planning algorithm), you can subscribe directly to /map:

import rclpy
from rclpy.node import Node
from nav_msgs.msg import OccupancyGrid

class MapSubscriber(Node):
    def __init__(self):
        super().__init__('laptop_map_listener')
        self.subscription = self.create_subscription(
            OccupancyGrid,
            '/map',
            self.map_callback,
            10
        )

    def map_callback(self, msg):
        width = msg.info.width
        height = msg.info.height
        resolution = msg.info.resolution  # meters per pixel
        grid_data = msg.data  # 1D array of cell values (-1: unknown, 0: free, 100: occupied)
        
        self.get_logger().info(f"Received {width}x{height} map grid.")

rclpy.init()
node = MapSubscriber()
rclpy.spin(node)

Network Optimization Tip

Because raw high-frequency /scan (LiDAR point clouds) or uncompressed video feeds can clog wireless bandwidth, keep these configurations in mind:

Use FastDDS / CycloneDDS tuning: Standard 2D Occupancy Grid maps (/map) are tiny (published every 1–2 seconds) and consume very little Wi-Fi bandwidth.
5GHz Band: Keep the RPi 5 connected to a 5GHz Wi-Fi band to avoid packet drops when streaming both continuous joint states and camera feeds simultaneously.

From <https://gemini.google.com/app/> Google Gemini (3.6 Flash)

Search This Blog

Learning from ChatGPT