Task 1. Dataset Exploration & Quality Audit

For this task I used three different datasets: lerobot/aloha_sim_insertion_human and lerobot/aloha_static_coffee for teleoperation and n3puiol/epic_kitchens_100_lerobot for egocentric.

Why two datasets for teleoperation?

I wanted to compare and see the tradeoffs between two different methods of data collection from the same robot platform. (Simulation v. Real)

• Encoder quantum estimation: finds the smallest recurring step in joint values to estimate the sensor's measurement resolution.
• Savitzky-Golay filter: smooths noisy joint-position data with a local polynomial so velocity, acceleration, and jerk aren't dominated by encoder noise.
• Hampel filter: flags points that are unusually far from the local median, making it robust to isolated spikes.
• Bonferroni correction: lowers each joint's significance threshold so testing 14 joints doesn't inflate the overall false-positive rate.

Simulation does not respect the hardware's physics bounds, so it produces motion the real arm cannot do. The real robot has the opposite problem, like a sensor repeating a stale reading.

What does sim lack? Encoder quantization, sensor freezes, state-action latency, real actuator limits, torque saturation behavior.

What I would filter

1. Hardware velocity limit (sim only): there are 151 frames that peak 5.22 vs 3.0 rad/s which is impossible with the given hardware. Clip them before any sim2real transfer.
2. Gripper scale mismatch: the gripper's tracking error is 29.3x the arm's in sim and 3.7x on real hardware, because the two channels live on different scales. Normalise the gripper separately so it does not dominate the loss.
3. Frozen readings: there are quite a few bit identical channels that assume our arm is stale when it is not. We should interpolate so models don't misunderstand.
4. State/action lag: 46/3/1 episodes sit at a lag of 0/1/2 frames, so realign actions with the correct state timestamps.
5. Effort outliers: 1,359 frames sit at the 99.9th percentile. Label them as contact rather than delete, and drop only what is physically unreasonable.

Egocentric

For egocentric I used two sources: the two wrist cameras on lerobot/aloha_static_coffee and the head cam from n3puiol/epic_kitchens_100_lerobot.

Why two sources for egocentric?

Same reason as teleop. A wrist cam moves because the robot commanded it, so I already know the motion. A human head cam moves because a person turned their head, so I have to guess the motion from pixels. Those are two different problems and I wanted both.

• Laplacian variance: measures how much fine detail a frame holds, which is the cheapest usable proxy for sharpness.
• Phase correlation: estimates how far the whole image shifted between frames, which is how fast the camera itself moved.
• Log-log regression per camera: learns the sharpness that camera normally has at that speed, so each stream gets its own expectation.
• Residual thresholding: flags the frames that are blurrier than their own speed predicts instead of the frames that are simply blurry.
• Duplicate frame detection: catches decoder stalls, where the same frame gets served twice and looks like a perfectly still camera.

What quality issues are specific to egocentric?

1. Blur is caused by the action: sharpness falls with camera speed at a slope of −0.48 on a wrist cam versus −0.09 on a fixed one. The blurriest frames are the reach and the retract, which are the frames a policy most needs.
2. The motion estimate can fail silently: 18 frames came back with a shift of 2507 px on a 224 px image. It does not error, it just returns nonsense, so it needs a sanity bound before anything trusts it.
3. Auto-exposure hunting: 41 exposure shifts on the head cam and 0.68% of frames over 5% blown out. A fixed camera in a fixed room never does this.
4. Clips are too short: median 23 frames at 12 fps, which is shorter than a single ACT chunk, so the episode cannot supply one training sample.
5. Decoder stalls: 9, 6 and 3 duplicate runs across the three real streams. A repeated frame reads as a still camera and quietly poisons the motion estimate.
6. Occlusion is normal, not a defect: a wrist cam is supposed to be full of gripper and object during a grasp. The real defect is a blocked lens.
7. Letterbox padding: 41% of every sim frame is black bars, which made all 25,000 frames read as underexposed until I cropped to the active region.

How would your filtering criteria differ from the joint-state data?

1. Threshold the residual, not the value: on joints a big number is a fault, on video a big blur is usually just speed. An absolute sharpness cut deletes every reach and retract and leaves a dataset of a robot standing still.
2. Calibrate per camera, not once: one model covers all 14 joints, but sharpness floors run from 0.62 to 16.13 across cameras, so a shared threshold condemns one stream and excuses another.
3. Tag and keep instead of repair or drop: a corrupted joint value poisons the label a policy regresses onto, so it has to go. A blurry frame is still a truthful observation with a correct action beside it.
4. Trust the input less: encoder velocity is always valid, camera motion is an estimate that can be wrong. The joint pipeline never has to sanity check its own measurement.
5. Budget for it: joint audit is about 0.005 s per episode, frame scoring is about 2 ms per frame, so vision runs once and gets cached to parquet instead of rerun per experiment.

Task 2. Labeling & Annotation Design

Schema lives in configs/label_schema.yaml. I label at three levels because three different consumers need three different things.

Which boundaries can I derive instead of annotate?

Most of them, which is the whole budget argument. These are pre-annotations, not finished labels. They land in the tool already drawn so an annotator confirms or drags a boundary instead of creating one.

• Gripper aperture midpoint: binarise at the 2nd and 98th percentile and the crossing marks grasp and release. The most reliable of the three, since it reads the commanded mechanism directly.
• Tracking-error step: the gripper's error runs 29.3x the arm's, and a step in it lines up with contact, so it proposes a contact onset. Correlated with contact, not proof of it, so a human confirms the ambiguous ones.
• Douglas-Peucker on the end-effector path: simplify at a 0.01 tolerance and the surviving corners become candidate approach, transport and retract boundaries for an annotator to accept or move.
• What humans still do: align vs place, and marking recover spans. I keep recover out of band because those are the best frames for robustness and the worst frames for plain behaviour cloning.

What tool?

Label Studio, because it syncs a time-series channel with video in one view, so the annotator watches the gripper trace move with the footage instead of guessing. CVAT for pure video work, where track mode and ground-truth QA pay off. Pre-annotations always loaded, so annotators correct rather than create.

How do I measure inter-annotator agreement for motion labels?

Kappa is the wrong primary metric here. Cohen's and Fleiss' assume a fixed set of items every annotator classifies. Free temporal segmentation has no such set, since two people emit different numbers of segments at different boundaries. You can force it by discretising into frames and I do report that, but frame-wise kappa is dominated by the long easy spans and says almost nothing about boundary placement, which is the part that actually disagrees.

What do I label for egocentric?

Existing schemas, not invented ones, so labels stay comparable with public corpora and a pretrained detector can bootstrap them.

1. Object interactions: the 100DOH per-hand format of box, side, contact state and contacted-object box. A pretrained detector proposes, a human corrects.
2. Dense contact masks: EgoHOS for left hand, right hand, first and second order objects. Also from a pretrained model.
3. Hand-eye phases: the Ego4D four keyframes, pre, contact, point-of-no-return, post. PNR is the frame after which the outcome is already decided, which is exactly what a success predictor should key on.
4. Failure moments as instants, not spans: a timestamp plus a 0.5 s window. Failures are near instantaneous, and forcing annotators to invent a start and an end destroys agreement.
5. Gaze proxy: no eye tracker, so either a fixed centre prior, reasonable on a wrist cam rigidly aimed at the end-effector, or the projected end-effector position from forward kinematics. The second is a task-relevance map, not gaze, and I name it that way.

Auto-label first or the budget never closes. HD-EPIC measured 263 annotations per minute of video. Detector bootstrapping, CVAT track mode and VLM hindsight relabelling at 2.2x throughput make that survivable. Auto-labels are training data, never evaluation data.

How do I align egocentric labels with the joint-state labels?

• One clock: hardware capture timestamps converted to a single monotonic episode clock at ingest.
• Measured camera offset: a flashing LED once per rig, since a USB wrist cam routinely sits 1 to 3 frames behind. Cross-correlating gripper aperture against EgoHOS contact-mask area then gives the camera-to-joint offset straight from the data.
• Seconds, never frame indices: a frame index is only valid for one stream at one rate and goes silently wrong the moment anything is resampled. Indices get derived at load time.
• Drift is real: 46/3/1 episodes sit at a lag of 0/1/2 frames on hardware and every sim episode sits at 0, so this has to be checked per episode rather than assumed.

Curation has to protect this. Task 3 never deletes interior frames, so a label timestamp stays valid after curation, and every emitted segment records its source episode and frame range.

Task 3. Curation Pipeline

In: LeRobot v2/v3 or ALOHA HDF5. Out: a LeRobot v3 dataset plus a manifest with the config, the fitted thresholds and a per-episode decision log, so every drop is traceable.

What gets rejected as a whole episode?

• No grasp: zero gripper toggles, so the episode cannot contain a pick.
• Mostly idle: under 50% effective frames.
• Frozen sensor: over 1% held readings, which means the driver was republishing stale state.
• Joint defect rate over 5%: corpus rates are 2.77% and 1.79% with per-episode maxima of 5.2% and 5.7%, so this cut removes the worst one or two episodes instead of reshaping the dataset. Exactly one episode got rejected per corpus.

What gets dropped frame by frame, and what does not?

Dropped: impossible velocity, frozen readings, position spikes, jerk spikes, timestamp gaps. Kept: velocity outliers, joint-limit contact, effort saturation, tracking spikes. That second group is events, not defects, and dropping it leaves a dataset of free-space motion. Treating velocity outliers as defects cut 25 to 42 frames from 12 of 50 sim episodes, all of them the rapid initial reach.

The trap that breaks everything downstream

The obvious implementation is keep = ~bad then reindex. That quietly breaks any policy consuming contiguous windows, which is all of them: ACT, Diffusion Policy, LeRobot's delta_timestamps.

So what happens to a bad run instead?

• Trim if it sits at the head or tail, since no training window straddles the cut.
• Repair if it is an interior run of 5 frames or fewer and the gap is physically plausible: the displacement between the two good endpoints, plus the velocity and acceleration it implies, has to sit inside the actuator spec. A short gap across a slow span interpolates safely; the same length across a fast reach does not, and gets split instead.
• Split if it is longer. Each side becomes its own episode with the timestep intact.
• Drop any segment under 120 frames, which is ACT's chunk of 100 plus margin. Below that the segment yields no training window at all, so raise this whenever the chunk size goes up.

Joint defects and video defects are not treated the same

This is the core design decision. A joint defect breaks the dynamics, since the frames either side are no longer consecutive states, so the episode splits. A video defect ruins one observation while the dynamics stay intact, so the frames stay and get marked in an observation.quality channel. Training then picks drop, down-weight or keep. That choice differs between a VLA and a world model, so curation should not make it irreversibly.

Verified, not assumed

verify_alignment runs on every written segment and raises rather than warns. Seven checks pass on both corpora: loads with stock LeRobotDataset, timestamps uniform within every episode to about 1e-06 s, zero padded steps in 100-step ACT chunks, video frames equal to state rows across all four cameras, no NaN or Inf, quality channel present, every episode clearing the chunk size. Zero padding is the one that matters: it proves no interior frame was silently deleted.

EPIC-KITCHENS is audited but deliberately not curated. Median clip is 23 frames at 12 fps, under both the minimum segment length and an ACT chunk, so nearly every clip would be dropped whole. Pre-segmented human video belongs in a latent-action or world-model path, not an action-chunked BC path.

Task 4. Policy Evaluation

What do I actually measure?

• Success rate with a Wilson 95% confidence interval, never a bare percentage.
• Stage-wise success: reach, grasp, lift, transport, place. Turns "40% success" into "grasps 90%, places 44%", which separates a perception problem from a fine-manipulation problem.
• Efficiency: time to success, path length, retry count.
• Safety: limit hits, effort spikes, collisions, reusing the Task 1 joint QC over the rollout logs.
• Per condition (object, initial pose bin, lighting), never pooled. Pooling hides the one condition that fails.

How many rollouts do I need?

Unaffordable on hardware, so three things buy it back:

• Pair initial conditions, then McNemar on the discordant pairs. Removing scene-difficulty variance roughly halves the required n.
• Wald SPRT early stopping: a policy sitting at 85% against a 60% null resolves in about 20 rollouts instead of 200.
• An automatic success detector turns a 50-rollout eval into a 500-rollout eval at the same human cost.

Success criteria

Written before running, mechanically checkable, time limited. Peg insertion: axis within 5 mm and 5 degrees, held at least 1 s, inside 30 s. Plus three training seeds minimum, the checkpoint fixed in advance since picking the best by eval success and reporting it is a leak, and the scorer blinded to which policy produced the rollout.

It works in sim and fails on the real robot. What now?

Cheapest first. The first two need no robot time and resolve most cases.

1. Feed real frames offline and compare predictions against sim frames at matched state. Points to a perception gap.
2. Replay a demo's actions open loop on both sim and hardware and compare realised state. Points to dynamics, which is fixed with system identification, not more data.
3. Inject the measured 30 to 100 ms latency into the sim loop. If sim success falls to match reality, that was the answer.
4. Per-step distance to the nearest training state in a learned embedding. Covariate shift looks like tracking fine for about 50 steps, then diverging.
5. Check commanded against realised joint positions and camera extrinsics. Calibration is boring and it is the cause more often than anything above.

One candidate is findable from the data alone, before booking robot time: 151 sim frames exceed the ViperX velocity limit at a peak of 5.22 against 3.0 rad/s, versus zero on real hardware. A policy trained on that is being taught to command motion the arm cannot execute.

What does the egocentric stream change?

The wrist camera is the only stream recording what the policy could see at each decision, which is what makes failure attribution possible at all.

1. Empty grasp: joints show the gripper closing and the arm lifting. The wrist cam shows jaws closing on air.
2. Slip after grasp: often no joint signature at all. The object simply leaves frame mid-transport.
3. Wrong target: a clean confident trajectory in joint space, gripper over the wrong object on camera.
4. Visual-servo lock-up: the arm holds still, and the frame turns out to be saturated or smeared with no usable features.
5. Near miss: scored as a success, but the alignment margin is visibly about 1 mm and the next trial fails.

The last one matters most. Success rate scores a 1 mm margin and a 10 mm margin identically, so a policy can degrade with no movement in the headline number. Alignment margin at the point-of-no-return frame is a graded signal with lower variance than a binary outcome, so it should detect the same change in fewer rollouts.

Bonus: I built the success detector, and it does not work

A frozen ImageNet ResNet-18 on wrist frames feeding a logistic probe, trained in seconds on CPU with no manual labels. The supervision is weak, not absent: episode position supplies the pseudo-labels, with the last 10% of an episode as 1, the first 25% as 0, and the middle excluded from training and scored at test.

Three guards caught it, and all three were needed:

• Episode-disjoint split. Adjacent 50 Hz frames are near duplicates, so a frame-level split reads AUROC 0.99 on a detector that learned nothing.
• Shuffled-label control over 50 draws, which sat at chance and ruled out leakage.
• Median fire position. A real completion detector fires near 1.0. Mine fired at 0.53.

So it ships as a pre-filter, not an oracle: score every rollout, human reviews the uncertain band plus a random audit sample, track agreement. Validating it properly needs human-labelled failed rollouts, which by construction do not exist in a demonstration dataset.

Task 5. Model Adaptation

The brief says pick one, VLA or world model. I did both, because they want opposite things from the same pipeline, and that conflict constrains how Task 3 has to be built.

Option A: the VLA

The curated output is already LeRobot v3, so π0 is the lower-friction target; OpenVLA needs a conversion hop into RLDS. Stock pi0_aloha_sim is the closest starting config for an ALOHA-topology arm. On the OpenVLA side, LoRA at rank 32 tunes 1.4% of parameters and is reported to match full fine-tuning, and OpenVLA-OFT's chunked action head reports 26x throughput and 3x lower latency.

What is most likely to silently break the training run?

Action normalisation. π0 rescales q01 to q99, OpenVLA bins that same range into 256 tokens. Both are wrong for absolute joint angles: clipping the outliers means the model can never emit certain joint positions at all.

• Absolute joint angles (OpenArm, ALOHA): use mean and standard deviation, or q001/q999 with a verified round trip.
• Delta end-effector poses: q01/q99 is correct here, which is why the default exists.
• The related trap: a rarely used dimension has a tiny q99 minus q01, so normalisation explodes it. The 14-DoF ALOHA layout has exactly such dimensions. The Task 1 audit reports per-joint ranges, so this surfaces before training rather than after.
• Chunk size couples to curation: the minimum segment length must exceed the action chunk or a curated segment yields zero training windows. Mine is 120 against ACT's 100.

How does egocentric video get preprocessed?

• Centre-crop then resize to 224, never letterbox. Letterboxing recreates the exact padding problem from Task 1, so I assert the active fraction is 1.0 as a guard.
• Per-camera normalisation stats, since wrist and overhead floors differ by an order of magnitude.
• Augmentation differs per view. A wrist cam already sees large natural viewpoint variation, and aggressive cropping destroys the fixed jaw-to-frame relationship. Photometric on the wrist, geometric on the static views.
• Frame quality is an ablation, not a decision made at curation time. The quality channel supports drop, down-weight or keep.

What breaks when a third-person pretrained model meets a wrist camera?

Viewpoint is one of the two most damaging perturbation axes for a VLA. LIBERO-Plus reports performance falling from 95% to under 30% under modest perturbations, viewpoint doing the most damage. A wrist camera is not a modest perturbation, it is a different imaging geometry.

1. Spatial reasoning collapses: the camera frame is the end-effector frame, so "move left" is ambiguous without knowing wrist rotation. The pretrained prior is not merely absent, it is actively wrong.
2. Scale and field of view mismatch: pretraining sees a whole workspace, the wrist sees a few centimetres, so "the red block" fills the entire frame.
3. Constant self-occlusion: the jaws occupy a fixed region of every single frame and no third-person prior covers a large static occluder.
4. Blur becomes a shortcut feature: sharpness tracks camera speed at −0.48 on the wrist versus −0.09 on a static view, so blur is predictive of the action. A model is free to lean on it instead of scene content, which then breaks when frame rate or exposure changes.
5. No global context and no absolute frame: it cannot see whether the target is still on the table, and image coordinates stop mapping to world coordinates.

What I would do: keep both views as separate image tokens, feed history rather than a single frame since one frame is ambiguous about ego-motion, warm up the wrist encoder separately, match the coordinate convention, then ablate wrist-only against both.

Option B: the world model

Supervision is self-supervised, since the next frame is the label. No success labels and no reward engineering, which is exactly why a world model benefits from the data a VLA wants filtered out.

Post-train, do not pretrain. Fifty episodes cannot train a video world model from scratch. Fine-tune a pretrained model like Cosmos on OpenArm episodes, generate rollouts, recover pseudo-actions with an inverse dynamics or latent action model, then train the policy on the result. Ctrl-World reports +44.7% success from that loop.

For unlabelled human video with no actions, the LAPA route applies: learn latent actions between consecutive frames, pretrain to predict them, then fine-tune the mapping onto real 14-DoF actions. That is the consumer for a corpus like EPIC-KITCHENS, which I audit but cannot curate into BC episodes.

Are robot ego and human ego interchangeable inputs?

No. Robot wrist motion has a median of 0.61 px and a 95th percentile near 2.3 px; human head motion sits at 1.72 and 12.84 px. Sharpness tracks motion at −0.48 on the wrist against −0.15 on the head. And the human corpus has no actions at all.

It supplies scene diversity the lab cannot, but its ego-motion statistics are a different distribution, so action conditioning learned there should not be expected to transfer directly. That is the argument for latent actions over direct action regression.

How do I verify the model learned something useful?

Not by reporting FVD and stopping. A model can generate beautiful coherent video and hold no usable dynamics.

What this means for the pipeline

• Failed episodes: a VLA would delete them, a world model needs them, so I partition and never delete.
• Effort spikes and limit contacts: flagged as events of interest rather than filtered as anomalies.
• Blurred frames: tagged through the quality channel so the consumer decides.
• Episode splitting: acceptable to a VLA, harmful to a world model, so I split only on genuine joint discontinuities.