Week 1 ended with a promise: teleop working, a camera feed I trust, something I could record without improvising mid-session. Week 2 got me the camera feed and a first VR session. Teleop still isn't clean enough to record. The gripper install ate the rest of the week.
I put on a Meta Quest for the first time and teleoperated a real arm. That part was genuinely sick. Then I spent an afternoon destroying screws, cracking a 3D print, and wondering why nobody warns you that robotics has a torque spec.
first time in vr
I'd never used a Quest before. Strapped it on, opened the teleop stack, squeezed the grip on the right controller, and the arm moved where I moved. Not smoothly at first. The first session was jittery enough that I couldn't pick up a pawn, but it still felt like reaching for things, not clicking through a menu.
The stack runs through telegrip: WebSocket VR input, pybullet IK, and our record loop reading joint targets off the solver and commanding the real XLerobot arm. Getting there took more software archaeology than I'd like to admit: wrong URDF path, units mismatch between degrees and normalized motor ranges, a grip-origin bug that made the arm crawl instead of follow. Once it worked, I understood why people tolerate the cable mess.
That doesn't mean I can pick anything up yet. The arm still jitters on the bench. I spent most of the debugging time on it: EMA smoothing on the five arm joints (gripper left alone so open/close stays sharp), telegrip's IK loop bumped from 20 Hz to 50 Hz, idle-sync fixed so a grip press roots from the real pose instead of a zero pose captured while you're sitting at the setup prompt. Offline sim looked much cleaner. On hardware the arm still hunts around the target. I can reach toward a piece. Closing the gripper on one reliably is still out.
a gripper and a torque spec
The stock gripper wasn't cutting it for chess pieces. Too narrow, too fiddly on small objects. So I designed and printed a custom parallel gripper: two fingers, symmetric stroke, meant to pick up a knight without launching it across the room.
Installing it was the week.
I (over)screwed up. Stripped heads. Snapped a print trying to force alignment. Had to reorder hardware and reprint a mount at 100% infill like that would teach the plastic a lesson. In software, a bad deploy rolls back. In hardware, a bad deploy sends you to the hardware store at 6pm on a Saturday. You don't get this failure mode from a failed CI check.
The gripper works now. Parallel, stable, picks up a rook cleanly. The install cost me more time than every bug I fixed in the teleop stack combined. I'm keeping that ratio in mind.
the chess retention crisis
My closest friend used to live ten minutes away. In 2023 we'd play chess in person roughly three times a week, kitchen table, no clock, trash talk included. Then he moved. Forty-five minutes away now. Different city, longer metro ride. We didn't slowly play less. We mostly stopped.
Being a builder, I did what any reasonable person would do: I pretended this was a metrics problem. Pulled rough session counts from memory and calendar scraps, normalized to monthly active in-person games, ran year-over-year on it like it was a startup metric. The numbers were sloppy. The answer wasn't.
monthly sessions, 2023 → jun 2026
Self-reported in-person games vs. one chess.com trial window (shaded). n ≈ small and biased. emotionally significant anyway.
Kitchen-table games only. chess.com trial shown separately; it spiked, then churned.
Commute time (minutes, one way) overlaid on trailing 3-month average session rate. Correlation ≠ causation. Felt like causation.
We tried chess.com. Played a dozen games over two weeks. Technically fine. Felt wrong without a board between us, without the pause to refill tea, without the post-game autopsy at the same table. We didn't keep playing online.
So I'm building a robot that plays chess with me. My friend can connect remotely, send a move, and the robot executes it on my board. Physical game, distributed players. Over-engineered, probably. Also the kind of thing a summer with a dual-arm platform was made for.
the chess robot (early)
I've just started building toward this. One arm watches the board from a fixed scan pose, the other arm plays. Perception reads the board into FEN; a game loop talks to Stockfish; teleop records demos for an ACT policy later.
Some pieces exist. Scan pose (M0) gets all 64 squares in one wrist-cam frame; the back rank is foreshortened but usable. Board-to-FEN uses homography warp, empty-board reference subtraction, and edge-diff occupancy. Starting position reads correctly after three pivots and one camera re-angle. Move detection passes synthetic tests for e2e4 and Nf3 from saved frames; real scan→move→scan still needs a bench run. The game loop scaffold runs full turn-taking in sim mode with python-chess. Live mode is wired, but the Stockfish binary isn't installed yet. VR teleop runs through telegrip with smoothing and comm retries wired in, but I still can't record a clean pick.
Plenty still missing. I haven't recorded 50+ teleop pick/place episodes (M1). Need clean trajectories before ACT training is real. Perception is clever but brittle: false positives on edge squares, phantom pieces if the board nudges between captures, oblique view misses low-contrast pieces on dark squares. We papered over some of this with a soft start-gate (seed from known starting FEN, detect changes), but a bumped board still hurts. No trained ACT policy (M2) yet, so the robot can't autonomously move a piece. The remote friend path is sketched (friend sends UCI over the network, loop validates and executes) but none of the plumbing exists. Captures and special moves (M5) are deferred. Robot knocking a queen off the board is a problem for future-me.
Lots to figure out. The gap between "correct starting FEN in a saved JPEG" and "robot plays a full game while your friend trash-talks from forty-five minutes away" is still most of the summer.
what I took from it
The Quest moment I keep coming back to: jittery arm, couldn't close on a pawn, and I still didn't want to take the headset off. That's the split this week. One side felt like the embodied-AI demos I'd been reading about. The other was me at the hardware store on a Saturday with stripped screw heads in my pocket.
The chess charts are half a joke. The other half is simpler. My friend moved across town, the games mostly stopped, and chess.com didn't fix it because there was no table between us, no pause to refill tea, no post-game autopsy at the same counter. I'm building a robot to get that back.
Week 1 was infrastructure I could hide behind. Week 2 was the first time I wanted to show someone the robot, and the first time the robot asked for a torque driver instead of a debugger. Next week I want one pick I can actually record, Stockfish on the Jetson, and one honest scan→move→scan on the bench. And maybe a text to my friend about what I'm building.