Ego500

Egocentric video is emerging as a core data source for robotics, but there is little work on benchmarking data quality or defining what high-quality annotations look like.

For this data to be useful for policy training, every clip must:

• Pass rigorous quality assurance (e.g. hands remain visible, the task is economically useful, lighting is sufficient, and there are no hardware issues).
• Be temporally accurate, with action boundaries precisely aligned to when each action begins and ends.
• Contain dense, fine-grained action annotations that describe every sub-action without mislabeling actions, objects, or hands.

Existing datasets are neither as diverse, as rigorously QA'd, nor as granularly and accurately annotated as Human Archive's datasets. This is made possible by our custom pipelines and human-in-the-loop review process, where over 100 trained annotators at our QC center review, correct, and refine our labels. We then use this labeled data to continually retrain and improve the underlying annotation models.

Today, we're releasing HA-Ego-500. Not only do our labels set a new state of the art in temporal accuracy and granularity, but HA-Ego-500 is also the first open-source egocentric dataset to provide large-scale, in-the-wild data across a diverse range of real commercial environments.

HA-Ego-500 is a large-scale collection of in-the-wild egocentric video spanning diverse real-world work environments. It contains 500 hours of footage across 60+ work environments (23 with more than four hours of footage) and over 310,000 structured action annotations, including:

• Temporal segment boundaries
• Action descriptions
• Task phases
• Per-hand state
• Visible objects

Hands are engaged in 92% of steps. The dataset spans 390 unique task–environment combinations, with the same task appearing across many different workplaces (e.g. cleaning spans 28 environments, while packaging spans 18). This diversity is essential for training robot policies that generalize across environments rather than overfitting to a single workplace.

Every released hour moves through a seven-stage pipeline, from on-the-ground collection to final reviewer approval.

LEFT-FRONT

LEFT-EYE

Descriptions are additionally normalized by an LLM integrated into our annotation tooling. With a single keystroke, annotators can regenerate a description from the step's structured fields (hands, objects, and task phase), ensuring consistent phrasing across 100+ annotators.

Every annotator completes a thorough qualification protocol before working on production footage. Part of it is an agreement pilot: 100+ trained annotators independently labeled the same 15 minutes of footage, spanning 10 different environments, with no model draft, scored with the same metrics we use to benchmark models. The result: 0.763 inter-annotator agreement. Two of our annotators labeling the same clip independently produce nearly the same annotation, and they agree with each other more than the annotators behind existing action datasets do. Boundaries are ambiguous even for experts; this is the consistency bar every annotator meets before contributing to the release.

Each video is annotated as a sequence of steps:

{
  "step": 1,
  "start_time_seconds": 0.0,
  "end_time_seconds": 3.0,
  "description": "Wipe the door handle with the cloth in the right hand",
  "task_phase": "task",        // "task" | "task_related"
  "left_hand": "empty",        // object held or "empty";
  "right_hand": "cloth",       // each hand annotated independently
  "objects_visible": ["bedsheet", "cloth"]
}

A step from the dataset.

Task decomposition

A flat label such as "prepares drinks at the bar" provides far less supervision for a VLA model than dense, temporally aligned action annotations. This is consistent with both our internal policy-training experiments and recent work in robot learning: increasing annotation density improves both VLA policies and video world-action models, across more than 1 million robot demonstrations and 50,000 EgoVerse egocentric videos, without collecting additional data. Similarly, RT-H (2024) demonstrates that fine-grained language supervision between the task and action levels produces more robust and sample-efficient policies than coarse task labels alone:

The task_phase field separates the task itself from task-related work: locomotion between work areas, fetching tools, and brief detours. This allows a policy to learn not just how to perform a step, but where that step fits within the overall task. No existing public egocentric action dataset explicitly models this distinction.

Two hands, annotated independently

Egocentric human video is the most scalable source of bimanual manipulation data there is. Bimanual coordination (stabilize with one hand, act with the other, hand-offs) is the current frontier for robot learning. We annotate left_hand and right_hand independently, per step: the human-demonstration analog of per-gripper state, with "empty" as a first-class value.

Grounded object vocabulary

Each video includes a curated inventory of every object in the scene, and every objects_visible label is drawn from that inventory. Objects are named consistently throughout the video, descriptions only include information visible in the footage, and nothing is inferred or made up.

We evaluated frontier models zero-shot on held-out clips from the dataset: given a 30 second clip, produce the complete step annotation. Predictions are matched to our released labels by temporal overlap and scored on boundaries, hand state, and task phase.

The models locate the actions: at loose overlap they score close to our annotators. But they do not produce the annotation. On strict boundaries and hand state, the two signals robot learning actually needs, they run at roughly half human accuracy:

520 held-out 30 second windows, one fixed prompt per model. The annotator row comes from the agreement pilot, scored with the same metrics. gpt-5 and kimi-k3 are evaluated from 10 timestamped frames per clip. Gold labels were human-corrected from model drafts, so same-family models benefit from a shared segmentation style.

That gap is what our pipeline closes. Every released label is a model draft verified and corrected by a trained annotator, then approved by an independent reviewer, so the human row is the level the released dataset holds. The model rows are what the footage would carry without the human loop.

Our annotation models will also improve with our new stereo and six-camera hardware, whose 180° DFOV configuration captures the entire scene and both hands simultaneously.

Stay tuned for these updates.