For participants and self-guided learners
Everything you need to go from a fresh machine to running the lab hub — usually under 15 minutes.
Written by Jim SeelanThis is the setup guide for Physical AI Lab — a hands-on session where you build the complete Physical AI pipeline from webcam to robot arm in three hours.
Visit the lab →The short version: double-click setup.bat and wait. The script handles everything and opens the browser when it's done. The steps below are just there in case you want to know what's happening.
Clone it with Git, or download the ZIP from GitHub and unzip it somewhere sensible.
git clone https://github.com/learnermaker/physical-ai-lab.git
cd physical-ai-labsetup.batNo admin rights needed. A terminal window will open and work through the setup automatically. Total time is usually 10–15 minutes on a decent connection.
Here's what it does:
| Stage | What happens |
|---|---|
| [1/9] | Looks for Python 3.12 — downloads and installs it quietly if not found |
| [2/9] | Creates a .venv folder inside the repo (isolated Python environment) |
| [3/9] | Activates the environment |
| [4/9] | Installs PyTorch CPU-only (the big one — about 200 MB) |
| [5/9] | Installs everything else from requirements.txt |
| [6/9] | Registers the kernel so VS Code notebooks can use this environment |
| [7/9] | Downloads models and demo assets: hand_landmarker.task (~8 MB), the pre-trained SAC agent for Module 2 (~3 MB), and the fallback demo video |
| [8/9] | Runs a quick check — prints pass/warn/fail for each package |
| [9/9] | Opens the hub server and launches the browser |
Step 8 prints something like this:
[PASS] numpy
[PASS] mujoco
[PASS] gymnasium
...
[WARN] Camera unavailable -- fallback video will be used
...
Setup complete: 13/14 checks passed[WARN] is fine — the lab has built-in fallbacks for webcam and OpenGL. [FAIL] means something needs fixing; the script prints the exact pip install command to run.
Module 5 can call Google Gemini live to analyse camera frames. You don't need a key — it falls back to pre-recorded responses — but live results are noticeably more interesting.
Get a free key at aistudio.google.com (sign in → Get API key), then:
# In the repo folder — Windows
copy .env.example .env
# Open .env in Notepad and add your key:
# GEMINI_API_KEY=AQ....your_keypython modules/02_simulation/03_pretrained_agent.pysetup.bat again. It detects what's already done and skips those steps.
If you'd rather do it yourself — or if setup.bat is giving you grief — here are the individual steps. You'll need Python 3.12 installed first. Estimated time: 20–30 minutes on a decent connection.
git clone https://github.com/learnermaker/physical-ai-lab.git
cd physical-ai-labThis keeps the lab packages separate from everything else on your machine.
# Create
py -3.12 -m venv .venv
# Activate (Windows)
.venv\Scripts\activate
# Your prompt should now show (.venv).venv\Scripts\activate again before continuing.
python -m pip install --upgrade pipThis needs to go in before the rest so that packages like Stable-Baselines3 link against the right version. It's about 200 MB so give it a minute.
pip install torch==2.14.0+cpu --index-url https://download.pytorch.org/whl/cpu
# Quick check:
python -c "import torch; print(torch.__version__)"
# Should print: 2.14.0+cpupip install -r requirements.txt
# Installs: numpy, mujoco, gymnasium, stable-baselines3,
# opencv-python, mediapipe, google-genai, fastapi, and more.
# Around 500 MB. Takes 3–10 minutes.python -m ipykernel install --user --name physical-ai --display-name "Physical AI Lab"These are gitignored large binaries — download_model.py fetches them in one step. Run it from the activated environment:
python download_model.pyThis downloads:
assets/hand_landmarker.task — MediaPipe hand landmark model (~8 MB)models/sac-HalfCheetah-v5.zip — pre-trained SAC agent for Module 2 (~3 MB, from HuggingFace)assets/fallback_hand_demo.mp4 — demo video for no-webcam participants (copied from the repo)All three are idempotent — safe to re-run, skips anything already present.
python verify_install.pyAll 14 checks should be [PASS] or [WARN]. A warning just means a fallback is active — the lab still works.
copy .env.example .env
# Edit .env and set:
# GEMINI_API_KEY=AQ....your_key
# Get a free key at https://aistudio.google.com/python api/server.pyOpen http://localhost:8000 in Chrome or Edge.
.venv\Scripts\activate
python api/server.pyIf setup is done, two commands get you running:
# Option A — terminal (recommended)
.venv\Scripts\activate
python api/server.py
# Option B — double-click setup.bat again.
# It detects the .venv and skips straight to launching the server.Open http://localhost:8000 in Chrome or Edge. That's the hub — everything runs from there.
| Module | What you build | What you'll see |
|---|---|---|
| 0 — Kickoff | The pipeline diagram. Print your first state vector. | Four numbers. That's the whole robot brain, right now. |
| 1 — Perception | Webcam → 21 landmarks → 4 joint angles | Green dots on your hand in real time. Angles printing as you move. |
| 2 — Simulation | Step CartPole and Reacher environments | The 11 numbers Reacher sees. The 2 torques you'll control in M4. |
| 3 — Reinforcement Learning | Train a PPO agent on CartPole from scratch | Flat reward curve → sharp climb. The agent discovers balance around 20–30k steps. Don't talk over it. |
| 4 — Perception → Action | Wire your hand to the robot arm. One line of code. | Move your finger. The Reacher arm follows. The loop is closed. |
| 5 — Foundation Models | Replace your hand with Gemini as the controller | Camera frame goes in. JSON action comes back. Same pattern as RT-2. |
| 6 — Wrap-up | Map everything to real research tools | MuJoCo, SB3, MediaPipe — not toys. What comes next. |
Every module has an exercise.py with a # TODO START / END block — usually 3–5 lines, 10–15 minutes each. A # SOLUTION block at the bottom can be uncommented if you get stuck. Open it in VS Code alongside the browser.
| Module | What the exercise asks |
|---|---|
| M0 | Print a numpy state vector — np.array([145.2, 163.8, 171.4, 158.0], dtype=np.float32) |
| M1 | Add the ring finger angle (landmarks 13→14) to make a 4-element vector |
| M2 | Print observation_space.low and .high for Reacher — understand the bounds |
| M3 | Change total_timesteps and re-run. Watch the curve shape change. |
| M4 | Add landmark 4 (thumb tip) as a third torque channel via ActionSliceWrapper |
| M5 | Write your own MY_PROMPT. Watch how phrasing changes the model's output. |
Open these in VS Code — File → Open Folder → select the repo, then pick the Physical AI Lab kernel top-right.
| Notebook | When to open it |
|---|---|
modules/00_kickoff/overview.ipynb | First thing — run all cells, draws the full pipeline |
modules/03_rl/01_mdp_concepts.ipynb | M3 — MDP concepts, run alongside the training script |
modules/06_wrapup/concepts_map.ipynb | End — maps every module to real-world equivalents |
There's a setup.sh that mirrors the Windows script. It needs Python 3.12 already installed:
bash setup.shIf Python 3.12 isn't on your machine yet:
# Homebrew first, if you don't have it
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
brew install python@3.12
python3.12 --version # should print 3.12.xsudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.12 python3.12-venv python3.12-dev
python3.12 --versionSame as the Manual Setup tab, with two differences:
# Create and activate the venv
python3.12 -m venv .venv
source .venv/bin/activate # ← not .venv\Scripts\activate
# Everything else is identical
pip install torch==2.14.0+cpu --index-url https://download.pytorch.org/whl/cpu
# If that exact version isn't available yet, omit the ==2.14.0+cpu pin:
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt
python download_model.py
python verify_install.py
python api/server.pystart_hub.sh instead:
bash start_hub.shsource .venv/bin/activate && python api/server.py
| Topic | Notes |
|---|---|
| MuJoCo | Pip wheels exist for macOS (arm64 + x86_64) and Linux x86-64. Installs without issues. |
| PyTorch on Apple Silicon | The CPU wheel works fine. For faster Module 3 training you can skip the --index-url flag and get the native MPS build instead — entirely optional. |
| OpenGL on Linux | If you're on a headless server, make_env automatically falls back from human rendering to rgb_array to None. Nothing to configure. |
| MediaPipe | Wheels exist for macOS arm64/x86_64 and Linux x86_64. Linux aarch64 is not covered. |
| Webcam on Linux | Add yourself to the video group if Chrome can't access the camera: sudo usermod -aG video $USER, then log out and back in. |
| Browser | Chrome or Edge required on all platforms. SharedArrayBuffer (needed by MediaPipe WASM) isn't available in Firefox. |
pip install as normal — pip picks the right one automatically.
Most issues fall into one of these categories. If none of these help, open an issue on the GitHub repository with the exact error message.
Right-click setup.bat → Open (instead of double-clicking) to keep the window visible. Or run it from a terminal:
cd C:\path\to\physical-ai-lab
.\setup.batPython 3.12 couldn't be found or downloaded. Check your connection, then install it manually from python.org — tick Add Python to PATH during setup — then re-run setup.bat.
The CPU wheel is about 200 MB. If pip times out, run it manually in a terminal with the venv active:
.venv\Scripts\activate
pip install torch==2.14.0+cpu --index-url https://download.pytorch.org/whl/cpu[FAIL] mujoco in the verify outputARM64 Windows (Surface Pro X, Snapdragon): MuJoCo doesn't have an ARM64 Windows wheel. Run everything inside WSL2:
# PowerShell, as Administrator
wsl --install
# Restart, open Ubuntu, then run: bash setup.shRegular x86-64 Windows: Usually a missing C++ runtime. Install it:
winget install Microsoft.VCRedist.2015+.x64You need Chrome or Edge. Firefox and Safari don't support SharedArrayBuffer, which MediaPipe's WASM inference requires. Switch browsers and try again.
# Find who's using it
netstat -ano | findstr :8000
# Kill it (replace 12345 with the PID in the last column)
taskkill /PID 12345 /F
python api/server.pyClick the kernel name in the top-right of any notebook → Python Environments… → pick Physical AI Lab. If it's not there, re-register it:
.venv\Scripts\activate
python -m ipykernel install --user --name physical-ai --display-name "Physical AI Lab"Check .env exists in the repo root and has a valid key. Module 5 falls back to cached responses automatically if the key is missing — the lab works either way.
type .env # should show GEMINI_API_KEY=AQ....