no display, no problem

Week 1 was supposed to be about getting the XLerobot moving. Instead it was cables, SSH sessions, and a camera feed that would not show up on my laptop.

I'm a builder by reflex. This summer I'm trying to read more anyway, so I front-loaded the week with that, then spent the rest turning a box of parts into something I could trust on the bench.

reading my way in

Before I touched the hardware again, I wanted a clearer picture of where embodied AI actually is right now. Not the Twitter version. More like: what do people think the stack looks like in 2026?

That meant world models. LeCun's been arguing for years that useful physical intelligence runs through internal simulators, not bigger chatbots. Predict in representation space, plan with a model, act in the real world. I rewatched his JEPA lecture, then followed the thread to V-JEPA 2 and Meta's claims about zero-shot robot planning from video.

On paper it's compelling. A model that learns physics-ish structure from watching the world, then needs only a small amount of robot data to start planning actions. Makes you wonder if you'll have to hand-write every control loop this summer.

I also went through the LeRobot docs, Physical Intelligence's writeup on π0, SmolVLA, OpenVLA, Diffusion Policy, and a TensorRT guide for YOLO on Jetson. Tooling for getting policies onto real hardware, frontier VLAs that already look like products, inference tricks for when a demo has to run at control-loop speed. Useful context before breaking things on my own bench.

In week 0 I called fresh eyes my only real edge. They go stale fast if you don't feed them. Naive questions only help once you know enough to tell which ones are actually naive. The reading isn't me abandoning the builder instinct. It's sharpening the one advantage I've got. Full list at the bottom. I'll keep doing this each week so future-me can trace what shaped the thinking.

ssh, no screen

Then the hardware remembered it exists.

Most of the week was setup: wiring, calibration, getting the Jetson to boot cleanly, making sure the arms move. I landed on SSH from my laptop into the Jetson on the bench. Convenient, except the Jetson had no display attached.

That sounds like a small detail until you try to run the example scripts. Anything that opens a window (cv2.imshow, a live preview, an annotated feed) dies immediately over SSH with no X server and no monitor. The process runs, the camera captures, and then nothing renders. Or worse, the whole thing crashes and you spend an hour wondering if the USB camera is broken when the problem is just display plumbing.

The fix was to split the work. Run the video and display pieces on a machine with a screen attached (the Jetson with an HDMI monitor, or a local terminal on the device itself), and do everything else over SSH from the laptop. Vision on one side, control and logging on the other. Clumsy at first. By Friday it was the only setup that worked.

None of this photographs well. Every robotics project has this week. You tell yourself you'll be training models by Wednesday, and by Wednesday you're still arguing with dmesg and a depth map that doesn't line up with reality. I did make progress: the Jetson boots cleanly, the camera streams, the calibration numbers look believable. The robot still hasn't done anything clever, but it also hasn't caught fire.

the example code can't keep up

Once the feed was visible, I started working through the XLerobot example scripts: how they're structured, what they assume about the hardware, where the sharp edges are. There's a follow demo that points the arm's end-effector at whatever object YOLO sees. On paper it's the fun one.

In practice it ran yolo11x.pt on the CPU, inline, inside the 50 Hz control loop. Every iteration blocked on inference. The arm lurched once every ten seconds, hopelessly behind the cup. The demo looked broken because it was broken. Not because I wired something wrong, but because the reference code put a ten-second synchronous inference call inside a real-time loop.

The catch on Jetson is that the GPU PyTorch wheels are Python 3.10 only, while LeRobot wants 3.12. You can't just move inference to the GPU in the same process. So I wrote a new version: a vision sidecar doing YOLO on the GPU in a 3.10 env, and the control loop in the 3.12 env, talking over localhost UDP (seq, capture_ts, dx, dy). The only thing that changed was where YOLO runs. Everything else (IK, gains, calibration) stayed fixed.

The gap was bigger than I expected.

On this platform the bottleneck wasn't the IK or the arm. It was synchronous CPU inference sitting inside the loop. Move YOLO off the control thread and visual servoing goes from broken to actually tracking the cup.

what I took from it

The example code wasn't buggy. It was a sketch someone wrote to show how the pieces fit together, and I treated it like production code. Reference demos are written to be read, not to hold a 50 Hz deadline. I'd have bet the arm or the IK would be the problem. It was one blocking inference call on the control thread, the kind of thing a demo hides and a real loop can't.

The fix I'm least proud of is probably the one I'll keep. Splitting Python versions (3.10 for the GPU wheels, 3.12 for LeRobot) looked like pure friction. It also forced perception and control into separate processes, which is what you'd want anyway. I wouldn't have chosen that architecture on purpose. Jetson made me.

The gap between the world-model talks and the calibration worksheet is wider than I remembered from 2022. The research stack assumes a working robot. The robot stack assumes you already know which cable is lying to you. Week 1 lived entirely on the robot stack.

Somewhere in the cable-wrangling I had my first conversation with a potential customer. The challenge isn't to make an arm twitch. It's to build something someone will actually pay for, and I don't fully know what that is yet. No MVP, no demo worth showing. Still, saying out loud what I'm building toward, in plain language instead of architecture diagrams, was the harder calibration this week. No worksheet for that one.

I'd rather eat this week now than discover it in September. Next week I want teleop working, a camera feed I trust, and a demo trajectory I can record without improvising fixes mid-session.