Introduction

Exo-Cook is an exocentric cooking benchmark for joint human action understanding, object interaction analysis, and gaze estimation/anticipation in third-person video. The dataset is built from cooking sequences in Ego-Exo4D and is designed to support unified modeling of what a person is doing, which objects are involved, and where the person is looking in different cooking scenarios.

The key motivation of Exo-Cook is to complement existing gaze–action benchmarks, which are primarily egocentric, by enabling unified gaze–action learning in third-person videos. While Ego-Exo4D provides large-scale third-person recordings, it is not directly usable for this task without extensive preprocessing, including annotation refinement, new label generation, modality alignment, and task-specific restructuring.

Exo-Cook provides short exocentric video clips paired with action annotations, object bounding boxes, and gaze location labels.

Videos

The released clips are stored under:

data/cropped/takes/<take_name>/

Each video follows the naming convention:

<take_name>_<timestamp>_<cam_id>.mp4

For example:

fair_cooking_05_6_9p645_cam01.mp4

The timestamp in the filename is the ending timestamp T of the clip. Each video spans [T − 2.5 s, T].

The released clips are resized from the original 3840 × 2160 Ego-Exo4D videos to 1280 × 720 while preserving the original 16:9 aspect ratio.

Exo-Cook Overview

Exo-Cook provides three main annotation types:

  • Action annotations for cooking interactions.
  • Object bounding-box annotations for interaction-relevant objects.
  • Exocentric gaze annotations, including current frame-level gaze and future gaze targets.

The benchmark split contains:

Partition Number of clips
Training 25,650
Validation 3,200
Test 3,200
Total 32,050

 

The selected clips are grouped into 10 semantic action categories:

Category Number of clips
0 2,562
1 2,741
2 1,193
3 688
4 2,675
5 4,167
6 4,000
7 3,530
8 5,500
9 4,994

 

Action Annotations

Action annotations are stored in:

label/action/.

Main files include:

label/action/
  ├── ExoCook_Action.txt
  ├── train_labels.txt
  ├── val_labels.txt
  └── test_labels.txt

Each action sample is associated with a released video clip and an action category index between 0–9. The (verb, noun) annotation for each video clip is also provided. The corresponding split-specific action labels are stored in:

label/action/train_labels.txt
label/action/val_labels.txt 
label/action/test_labels.txt

 

Object Bounding Boxes

Object bounding-box annotations are stored in:

label/bbox/

Files:

label/bbox/
  ├── object_bbox.csv
  └── object_bbox.txt

Both annotation files provide object bounding boxes for the released 1280 × 720 exocentric videos.

Exocentric Gaze

Exocentric gaze annotations are stored in:

label/gaze/<take_name>/

Each gaze annotation file follows the format of:

<video_name>_gaze.txt.

For example:

Video:
data/cropped/takes/fair_cooking_05_6/
  fair_cooking_05_6_5p064_cam03.mp4

Gaze annotation:
label/gaze/fair_cooking_05_6/
  fair_cooking_05_6_5p064_cam03_gaze.txt

The gaze labels are generated by combining Aria eye-gaze estimates, the Aria device trajectory, and the exocentric GoPro camera calibration provided by the original Ego-Exo4D dataset. Gaze is transformed from the Aria coordinate system into the world coordinate system and then projected into the corresponding exocentric camera.

Each gaze TXT file contains one row per video frame.

Core fields include:

frame_idx
clip_time_sec
take_time_sec

gaze_x
gaze_y
gaze_valid

gaze_x and gaze_y are the projected 2D gaze coordinates in the released 1280 × 720 video.

Exo-Cook additionally provides future gaze targets relative to the ending timestamp T of the input clip:

T + 0.25 s
T + 0.50 s
T + 0.75 s
T + 1.00 s

The corresponding annotation fields are:

future_gaze_x_Tplus0p25
future_gaze_y_Tplus0p25
future_gaze_valid_Tplus0p25
future_timestamp_us_Tplus0p25

future_gaze_x_Tplus0p5
future_gaze_y_Tplus0p5
future_gaze_valid_Tplus0p5
future_timestamp_us_Tplus0p5

future_gaze_x_Tplus0p75
future_gaze_y_Tplus0p75
future_gaze_valid_Tplus0p75
future_timestamp_us_Tplus0p75

future_gaze_x_Tplus1
future_gaze_y_Tplus1
future_gaze_valid_Tplus1
future_timestamp_us_Tplus1

These labels support both current gaze estimation and future gaze anticipation from the same 2.5-second exocentric input clip.

Dataset Splits

The predefined benchmark partitions are stored in:

splits/

Files:

splits/
  ├── train.txt
  ├── val.txt
  └── test.txt

Data Structure

Exo-Cook/
├── data/
│  └── cropped/
│     └── takes/
│        └── <take_name>/
│           └── <take_name>_<timestamp>_<cam_id>.mp4
│
├── demo/
│  ├── visualize_action.py
│  ├── visualize_bbox.py
│  └── visualize_gaze.py
│
├── label/
│  ├── action/
│  │  ├── Action_Full.txt
│  │  ├── ExoCook_Action.txt
│  │  ├── train_labels.txt
│  │  ├── val_labels.txt
│  │  └── test_labels.txt
│  │
│  ├── bbox/
│  │  ├── object_bbox.csv
│  │  └── object_bbox.txt
│  │
│  └── gaze/
│     └── <take_name>/
│        └── <video_name>_gaze.txt
│
└── splits/
    ├── ExoCook_ids.txt
    ├── train.txt
    ├── val.txt
    └── test.txt

Visualization

Example visualization scripts are provided under:

demo/

Action

python demo/visualize_action.py

Generated examples can be stored under:

demo/action_visualizations/.

Bounding Boxes

python demo/visualize_bbox.py

Gaze

python demo/visualize_gaze.py \
    /path/to/selected/video.mp4

The default gaze visualization displays only:

  • the projected head-center proxy;
  • the projected current gaze point.

Recommended Tasks

Exo-Cook can be used for several related tasks:

Action Recognition

Predict the action category associated with the current exocentric clip.

Action Anticipation

Predict future human actions from the observed video history.

Exocentric Gaze Estimation

Estimate the current 2D gaze point from third-person video.

Joint Gaze–Action Modeling

Use gaze, action, and object information jointly to study the relationship between human visual attention and object-centric activity.

Citation

This dataset corresponds to "SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understanding," published at the European Conference on Computer Vision (ECCV) 2026. If you use Exo-Cook in your research, please cite:

@inproceedings{kuang2026sage,
  title     = {SAGE: Synchronized Action-Gaze Recognition and Anticipation
               for Human Behavior Understanding},
  author    = {Kuang, Chenyi and Agarwal, Nakul},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

Exo-Cook is derived from Ego-Exo4D data. Users should also follow the applicable Ego-Exo4D citation requirements.

Pre-print: arXiv:2607.04017

 

Access & License

The dataset is available for non-commercial research purposes. Requesters must be affiliated with a university and use institutional email credentials. Submit requests via the official download form.

Contact

Project webpage: https://kuangcy1998.github.io/SAGE/