Hugging Face's logo

ShubhamRasal
/
cross-person-ego-imu-benchmark

Model card Files
xet
 Community
cross-person-ego-imu-benchmark / PROPOSAL.md
ShubhamRasal's picture
ShubhamRasal
Scale proposal to 500K datapoints from Egocentric-1M dataset β€” updated budgets, timeline, storage, new scaling experiment
cac9a3f verified
preview code
|
raw
history blame contribute delete
25 kB

Cross-Person Generalization in Egocentric Video + IMU Activity Recognition: A Benchmark Study at 500K Scale

Author: Shubham Rasal β€” shubhamrasal.com
Date: April 2026
Status: Proposal Draft

1. Motivation & Problem Statement

Egocentric video and body-worn IMU sensors are the sensing backbone of AR glasses, smartwatches, and workplace wearables. Activity recognition from these modalities has made rapid progress β€” EVI-MAE reaches 87.96 mAP on CMU-MMAC, COMODO achieves 59.1% top-1 on Ego4D with IMU alone, and MoBind achieves state-of-the-art cross-modal retrieval. But all of these results are reported on within-distribution splits where training and test data come from the same population of users.

The real-world failure mode is cross-person generalization. When you train on workers A–H and deploy to worker J, performance degrades β€” often catastrophically. The recent survey on generalizable HAR (Cai et al., 2025) formalizes this as:

U_train ∩ U_test = βˆ…, p(X, y | U_train) β‰  p(X, y | U_test)

This is the setting that matters for any commercial wearable deployment, yet it remains systematically understudied because existing ego+IMU datasets have too few subjects (CMU-MMAC: 12, WEAR: 18, MMEA: unknown) to support meaningful cross-person evaluation at scale.

The Opportunity

We will use 500K clips from the Egocentric-1M dataset β€” drawing from a pool of ~1,000 workers performing industrial/workplace tasks. This is orders of magnitude larger than any existing ego+IMU benchmark and enables studies that are statistically impossible on current datasets.

Additionally, we have early access to a 13K-clip, 100-worker pilot subset (the Egocentric Native Compression dataset) for rapid prototyping and pipeline validation before scaling to 500K.

Property Pilot (Available Now) Full Scale (500K)
Workers 100 ~1,000+ (estimated)
Clips 12,997 500,000
Duration ~97.5 hours ~3,750 hours
Total size 1.22 TiB ~47 TiB
IMU data 6.18 GiB ~238 GiB
5s windows ~65K ~2.5M
IMU rate 200 Hz (acc + gyro) 200 Hz (acc + gyro)
Activity labels working / break / not-on-person working / break / not-on-person
Domain Industrial / workplace Industrial / workplace

From preliminary exploration of the pilot dataset, I've observed consistent differences in motion patterns across workers performing similar tasks β€” exactly the distribution shift that drives generalization failures. With ~1,000 workers and 500K clips, we can study this at a scale that no prior work has approached.

2. Research Questions

  1. How much does cross-person distribution shift degrade ego+IMU activity recognition?
    Quantify the gap between within-person and cross-person accuracy across IMU-only, video-only, and multimodal setups.

  2. Which modality is more robust to person shift β€” video or IMU?
    IMU captures body dynamics (person-specific); video captures visual scene (potentially person-invariant). Which degrades less under cross-person shift?

  3. Does video→IMU distillation improve cross-person robustness?
    COMODO distills video knowledge into IMU encoders for same-distribution settings. Does the distilled representation also transfer better to unseen workers?

  4. Can we predict which workers will be "hard cases" before deployment?
    Use IMU statistics (spectral features, signal energy, temporal patterns) to predict which unseen workers will cause generalization failure β€” enabling targeted data collection or adaptation.

  5. Does self-supervised pre-training on all workers (unlabeled) close the cross-person gap?
    EVI-MAE-style pre-training uses unlabeled data from all workers. Does this shared representation space reduce person-specific biases?

  6. What are the scaling laws for cross-person generalization? (New at 500K scale)
    How does downstream cross-person accuracy change as we increase training data from 13K β†’ 50K β†’ 100K β†’ 250K β†’ 500K clips? Is there a data scaling law for cross-person robustness?

3. Dataset Overview

3.1 Pilot Dataset (Available Now)

Source: gs://build-ai-egocentric-native-compression/

Metric Value
Total workers 100 (selected from 150, 103 complete)
Total clips 12,997
Total duration ~97.5 hours
Total size 1.22 TiB (1,240 GiB video + 6.18 GiB IMU)
Video per clip ~100 MiB MP4, ~27 seconds
IMU per clip ~500 KiB JSONL, ~5,400 samples at 200Hz

Pilot worker statistics (from index.csv):

Statistic Clips per Worker
Min 76
Q1 (25th percentile) 95
Median 129.5
Q3 (75th percentile) 164
Max 203
Mean Β± Std 130.0 Β± 36.3

Activity state distribution (per-worker metadata):

State Mean Fraction Std Min Max
Working 0.843 0.110 0.250 0.957
Taking break 0.123 0.110 0.014 0.721
Not on person 0.034 0.016 0.000 0.058
Worn (working + break) 0.966 0.016 0.942 1.000

3.2 Full Dataset (500K Target)

Metric Estimated Value
Total workers ~1,000+
Total clips 500,000
Total duration 3,750 hours (156 days)
Video data ~46.9 TiB
IMU data ~238 GiB
5s windows ~2,500,000
IMU window tensor (float32) ~56 GiB

3.3 Notable Pilot Workers (Hard Cases)

Worker Working % Break % Off-body % Clips Notes
worker_039 25.0% 72.1% 2.9% 135 Extreme outlier β€” mostly on break
worker_085 37.3% 57.5% 5.2% 149 Majority break time
worker_019 57.0% 41.2% 1.8% 110 Near-equal work/break split
worker_087 62.1% 32.5% 5.3% 164 High break + high off-body

At 500K scale with ~1,000 workers, we expect 10Γ— more behavioral diversity β€” more outliers, richer distribution tails, and harder generalization challenges.

3.4 Curation

  • First and last 2 clips trimmed per worker (removes startup/shutdown artifacts)
  • Audio stripped from all videos
  • IMU format: JSONL with t_us (microseconds), acc [x,y,z] (m/sΒ²), gyro [x,y,z] (rad/s)

4. Experimental Design

4.1 Two-Stage Approach: Pilot β†’ Full Scale

Stage 1 (Weeks 1-4): Pilot on 13K clips

  • Validate all pipelines, debug models, establish baseline numbers
  • Run all experiments on 100-worker pilot for fast iteration (~hours per experiment)
  • Identify which methods are worth scaling

Stage 2 (Weeks 5-14): Scale to 500K clips

  • Run the winning methods at full scale
  • Execute the data scaling experiment (unique contribution at this scale)
  • Full cross-person benchmark at ~1,000-worker scale

4.2 Cross-Person Evaluation Protocol

Primary protocol: 5-Fold Leave-20%-Workers-Out (L20%WO)

Split workers into 5 non-overlapping folds (20% each). For each fold:

  • Train: 60% of workers
  • Validation: 20% of workers
  • Test: 20% of workers
Pilot (100 workers) Full (est. ~1,000 workers)
Train 60 workers (~7,800 clips) 600 workers (300K clips)
Val 20 workers (~2,600 clips) 200 workers (100K clips)
Test 20 workers (~2,600 clips) 200 workers (100K clips)

Report mean Β± std across 5 folds.

Secondary protocols:

  • Leave-1-Worker-Out (L1WO): On a representative subset of 100 workers (from the full 1,000). Train on 99, test on 1 Γ— 100. Measures per-worker difficulty.
  • Worker scaling curve: Train on {100, 200, 400, 600, 800} workers, always test on the same held-out 200. Answers: "how many workers do you need?"
  • Data scaling curve: Train best model at {13K, 50K, 100K, 250K, 500K} clips. Answers: "how much data do you need?"

4.3 Task Definition

3-class activity state classification:

  • Working (84.3% of clips in pilot)
  • Taking break (12.3%)
  • Not on person (3.4%)

Window protocol (following COMODO/Ego4D standard):

  • 5-second non-overlapping windows
  • IMU: 200Hz Γ— 5s = 1,000 timesteps Γ— 6 channels (acc_xyz + gyro_xyz)
  • Video: 10 FPS Γ— 5s = 50 frames, resized to 224Γ—224
  • Per 27s clip: 5 windows β†’ **2.5M total windows at 500K scale**

4.4 Metrics

Metric Why
Macro F1 Primary metric. Accounts for class imbalance (3.4% off-body would be invisible in accuracy)
Accuracy @1 Standard comparison with COMODO/EVI-MAE baselines
Per-worker F1 variance Οƒ across workers β€” directly measures generalization stability
Generalization gap Ξ” between within-person and cross-person F1
Per-class F1 Breakdown across working/break/not-worn

4.5 Baseline Comparisons

Experiment A: IMU-Only Models (5 baselines)

Model Type Params Reference
DeepConvLSTM Supervised CNN+LSTM ~2M OrdΓ³Γ±ez & Roggen, 2016
Attend & Discriminate Supervised attention ~3M Abedin et al., 2021
CrossHAR Cross-dataset transfer ~5M Hong et al., 2024
MOMENT-small Pre-trained time-series foundation ~40M Goswami et al., 2024
Mantis Pre-trained time-series ~15M Liu et al., 2024

All evaluated in both within-person (random split) and cross-person (L20%WO) settings. The gap between the two is the headline result.

Experiment B: Video-Only Models (3 baselines)

Model Type Params Reference
TimeSformer-Base Transformer (K400 pretrained) 121M Bertasius et al., 2021
VideoMAE ViT-B MAE pre-trained Transformer 86M Tong et al., 2022
SlowFast R50 Dual-pathway CNN 34M Feichtenhofer et al., 2019

Experiment C: Cross-Modal Distillation

Method Setup Reference
COMODO (MOMENT student) Frozen TimeSformer β†’ MOMENT IMU encoder Chen et al., 2025
COMODO (Mantis student) Frozen TimeSformer β†’ Mantis IMU encoder Chen et al., 2025
IMU2CLIP Joint video-IMU-text embedding Moon et al., 2023

Key question: Does the distilled IMU model generalize better to unseen workers than the supervised IMU model?

Experiment D: Self-Supervised Pre-training

Method Setup Reference
EVI-MAE Joint video+IMU MAE pre-training β†’ fine-tune Zhang et al., 2024
IMU-only MAE IMU spectrogram MAE β†’ fine-tune Adapted from EVI-MAE

Pre-train on all workers (unlabeled, self-supervised), then fine-tune on train workers with labels, evaluate on held-out test workers. Compare with same architecture trained from scratch.

Experiment E: Hard Case Analysis

For a representative subset of workers, compute:

  • IMU statistics: RMS acceleration, gyro variance, spectral centroid, signal entropy, jerk magnitude, dominant frequency
  • Temporal statistics: autocorrelation decay, stationarity (ADF test), transition rate between high/low activity
  • Model performance: per-worker F1 from L1WO

Then:

  1. Correlate IMU statistics with model difficulty (per-worker F1)
  2. Cluster workers by IMU feature similarity β†’ do clusters predict generalization difficulty?
  3. Predict difficulty: train a regression model (IMU stats β†’ expected F1) β€” can we flag hard workers before deployment?

Experiment F: Data Scaling Laws (New at 500K)

Train the best-performing model from Experiments A-D at increasing data scales:

Scale Clips Est. Workers Est. Hours
13K 12,997 100 97.5h
50K 50,000 ~385 375h
100K 100,000 ~770 750h
250K 250,000 ~1,000 1,875h
500K 500,000 ~1,000 3,750h

Measure: cross-person Macro F1 vs. training data size. Is there a scaling law? Does it plateau?

5. Technical Approach

5.1 IMU Preprocessing Pipeline 

Raw JSONL (200Hz, acc+gyro)
    β†’ Parse, validate timestamps, interpolate gaps
    β†’ 5-second non-overlapping windows (1000 Γ— 6)
    β†’ Two representations:
        (a) Raw time series [1000, 6] for Mantis/MOMENT/DeepConvLSTM
        (b) STFT spectrogram [160, 128, 6] for EVI-MAE/IMU-MAE
    β†’ Per-channel z-normalization (computed on training set only)

At 500K scale: 2.5M windows Γ— 6,000 floats = ~56 GiB processed tensor. Fits comfortably in RAM on a cpu-upgrade instance. Preprocessing takes ~80h CPU but is embarrassingly parallel.

Already built: I have the 200Hz windowing and video sync pipeline from earlier work on the pilot dataset.

5.2 Video Preprocessing Pipeline 

MP4 (27s clips, ~100 MiB each)
    β†’ Decode at 10 FPS β†’ 270 frames per clip
    β†’ 5-second windows β†’ 50 frames per window
    β†’ Resize to 224Γ—224, normalize (ImageNet stats)
    β†’ Pre-compute video encoder embeddings (store to disk)

At 500K scale: Pre-computing video embeddings is critical. We do NOT download all 47 TiB of video. Instead:

  • Stream video clips on-demand during embedding extraction
  • TimeSformer-B produces 768-dim vectors β†’ 2.5M windows Γ— 768 Γ— 4 bytes = ~7.3 GiB per model. Easily stored.
  • Embedding extraction: ~200h on A10G (parallelizable across 4 GPUs β†’ ~50h wall-clock)

5.3 COMODO Distillation Protocol

Following Chen et al. (2025) exactly:

  • Video teacher: TimeSformer-Base, frozen, K400-pretrained. MLP projector (hidden=2048, out=128)
  • IMU student: MOMENT-small or Mantis, trainable. MLP projector (hidden=2048, out=128)
  • Loss: Cross-entropy on similarity distribution with dynamic FIFO instance queue
  • Hyperparameters: 20 epochs, batch 128, lr=3e-4, Ο„_v=0.1, Ο„_x=0.05
  • Hardware: A10G GPU (24 GB VRAM) β€” scales linearly with data, ~12h per student per fold at 500K

5.4 EVI-MAE Protocol

Following Zhang et al. (2024):

  • Video encoder: ViT-Base (patch 16Γ—16), 90% masking ratio
  • IMU encoder: Transformer on STFT spectrogram patches (16Γ—16), 75% masking ratio
  • Fusion: Single-layer Transformer
  • Pre-training loss: MAE reconstruction (Ξ±=1, Ξ²=10) + contrastive (Ξ³=0.01, Ο„=0.05)
  • Pre-train: 100 epochs on all workers (fewer epochs needed β€” each epoch sees 38Γ— more data)
  • Fine-tune: 50 epochs on train split with labels
  • Hardware: A100x4 (4Γ— 80GB) for pre-training (~6 days), A10G for fine-tuning

6. Timeline (16 Weeks)

Stage 1: Pilot Validation (Weeks 1-4)

Week Phase Deliverable Compute
1 Data preparation IMU windowing, video embedding extraction, EDA notebook on pilot CPU: 12h, A10G: 10h
2 IMU baselines (pilot) 5 IMU models Γ— 5 folds on 13K clips. Validate pipeline, establish baselines A10G: 20h
3 Video + COMODO (pilot) Video baselines + COMODO distillation on 13K clips A10G: 30h
4 EVI-MAE (pilot) + triage Self-supervised pre-training on pilot. Select methods to scale A10G: 40h

Stage 2: Full Scale (Weeks 5-14)

Week Phase Deliverable Compute
5-6 500K data ingestion Download 238 GiB IMU. Stream-extract video embeddings for 500K clips (3 models) CPU: 88h, A10G: 600h
7-8 IMU baselines (500K) 5 IMU models Γ— 5 folds at full scale A10G: 250h
9-10 Video + COMODO (500K) Video baselines + COMODO distillation at full scale A10G: 240h
11-12 EVI-MAE (500K) Self-supervised pre-training on all workers (~6 days on A100x4) + fine-tuning A100x4: 150h, A10G: 40h
13 Hard case analysis Per-worker difficulty on 100-worker subset, IMU feature correlation A10G: 100h, CPU: 40h
14 Scaling experiments Data scaling curve (13K→500K), worker scaling curve (100→800) A10G: 375h

Stage 3: Synthesis (Weeks 15-16)

Week Phase Deliverable Compute
15 Analysis Compile results, statistical significance tests, generate figures CPU: 20h
16 Write-up Paper draft (8-page main + appendix) β€”

Total wall-clock: 16 weeks
Active compute window: Weeks 1-14 (14 weeks of experiments)

7. Compute Budget

7.1 Summary

Resource Hours Unit Cost Total
CPU (preprocessing, analysis) 160h $0.60/h $96
A10G GPU (24 GB) 1,705h $2.00/h $3,410
A100x4 GPU (4Γ— 80 GB) 150h $16.00/h $2,400
Subtotal 2,015h $5,906
30% contingency +$1,772
Total ~$7,678

7.2 Phase Breakdown

Phase Resource Hours Cost Notes
1. Pilot validation A10G 100h $200 Weeks 1-4, fast iteration
2. Data ingestion + embedding extraction CPU + A10G 88h + 600h $1,253 Parallelizable across 4 GPUs
3. IMU baselines (500K) A10G 250h $500 5 models Γ— 5 folds
4. Video + COMODO (500K) A10G 240h $480 3 video models + 2 distillation students
5. EVI-MAE pre-training A100x4 150h $2,400 ~6 days continuous
5b. EVI-MAE fine-tuning A10G 40h $80 5 folds
6. Hard case analysis A10G + CPU 100h + 40h $224 L1WO on 100-worker subset
7. Scaling experiments A10G 375h $750 5 scale points Γ— 5 folds
8. Synthesis CPU 20h $12 Figures, stats

7.3 Storage Requirements

Data Size Notes
Raw IMU data (all 500K clips) ~238 GiB Download once, keep in project storage
Processed IMU windows (float32) ~56 GiB 2.5M windows Γ— [1000, 6]
Video embeddings (3 models) ~22 GiB 2.5M windows Γ— 768-dim Γ— 3 encoders
Selective raw video (10% for EVI-MAE) ~4.8 TiB Stream during pre-training; cache hot subset
STFT spectrograms (for EVI-MAE) ~200 GiB 2.5M Γ— [160, 128, 6]
Model checkpoints & logs ~100 GiB ~50 experiments Γ— ~2 GiB each
Total active storage ~5.4 TiB

Key insight: We do NOT need all 47 TiB of video. For 80% of experiments (IMU baselines, COMODO, hard case analysis), only the 238 GiB of IMU data + 22 GiB of pre-computed video embeddings are needed. Only EVI-MAE pre-training requires raw video, and we stream it rather than pre-downloading.

7.4 Comparison: Pilot vs. Full Scale

Pilot (13K) Full (500K) Scale Factor
GPU-hours 273h 2,015h 7.4Γ—
Cost $654 $7,678 11.7Γ—
Timeline 10 weeks 16 weeks 1.6Γ—
Storage 284 GiB 5.4 TiB 19Γ—

The cost scales sub-linearly with data (11.7Γ— cost for 38.5Γ— more data) because: (a) embedding extraction amortizes across all experiments, (b) fewer epochs needed at larger data scale, (c) pilot experiments avoid wasting compute on methods that don't work.

8. Expected Contributions

8.1 Empirical Contributions

  1. First cross-person generalization study on ego+IMU at 1,000-worker, 500K-clip scale. Prior work: CMU-MMAC (12 subjects), WEAR (18), MMEA (unknown), Ego4D (mixed, not controlled). We are 50-80Γ— larger in subject count.

  2. Modality robustness comparison under person shift. No prior work has compared video vs. IMU vs. multimodal degradation under controlled cross-person shift on the same dataset.

  3. Data scaling laws for cross-person generalization. First systematic study of how cross-person accuracy scales with training data (13K β†’ 500K). This has direct practical value: how much data do you need to collect?

  4. Worker difficulty prediction from IMU statistics. If successful, this enables proactive data collection β€” target workers whose motion patterns are underrepresented.

  5. Cross-modal distillation under distribution shift. COMODO was validated on same-distribution splits. We test the open question: does distilled knowledge transfer across people?

8.2 Benchmark Contribution

  • Standardized splits (5-fold L20%WO) and evaluation protocol for 1,000-worker ego+IMU
  • Baseline results for 11+ models across 3 modality configurations at 500K scale
  • Per-worker difficulty scores for hard case analysis
  • Data scaling curves with fitted power laws
  • Open-source evaluation code and pre-computed features

8.3 Practical Contribution

  • Guidelines for "how many workers to collect" (worker scaling curve)
  • Guidelines for "how many clips to collect" (data scaling curve)
  • Method for flagging hard-to-generalize deployment scenarios
  • Evidence for whether IMU-only deployment (privacy-preserving) sacrifices cross-person robustness vs. video

9. Relation to Existing Work

Paper What They Did What We Add
COMODO (Chen et al., 2025) Video→IMU distillation on Ego4D/EgoExo4D/MMEA Test distillation under cross-person shift; ~1,000 workers, 500K clips
EVI-MAE (Zhang et al., 2024) Multimodal MAE on CMU-MMAC (12 subj.) and WEAR (18 subj.) Scale to ~1,000 workers; evaluate cross-person transfer of MAE representations
Generalizable HAR Survey (Cai et al., 2025) Taxonomy of cross-person/device/position HAR Provide the large-scale empirical study their survey explicitly calls for
IMU-Video OOD HAR (Cheshmi et al., 2025) Cross-modal SSL for OOD transfer Study OOD as cross-person (same task, different people) rather than cross-domain
ColloSSL (Jain et al., 2022) Multi-device contrastive SSL for HAR Apply cross-device ideas to cross-person setting with paired video+IMU
UniMTS (Zhang et al., 2024) Text-aligned IMU pre-training for zero-shot HAR Potential extension: text-conditioned cross-person generalization
MIAM (Mehta et al., 2025) Industrial assembly monitoring (RGB+depth+IMU, 8 volunteers) Same domain but ~1,000 workers vs. 8; formal cross-person protocol

10. Risk Mitigation

Risk Likelihood Mitigation
500K dataset access delayed Medium Start with 13K pilot (Weeks 1-4). All pipelines and pilot results are independently valuable
Video streaming bottleneck at 47 TiB Medium Pre-compute embeddings (only 22 GiB). Only EVI-MAE needs raw video; use selective 10% subset
3-class task too easy (>95% accuracy) Medium β€” "working" is 84% majority class Use macro-F1 (penalizes majority bias); explore finer-grained temporal segmentation at scale
EVI-MAE training unstable on A100x4 Low Validated on pilot first (Week 4). Follow published hyperparameters exactly; reduce lr if needed
Compute budget overrun Medium 30% contingency ($1,772). Pilot phase identifies which methods to scale β€” skip underperformers
Insufficient cross-person variance Very Low β€” already confirmed in pilot 1,000 workers guarantees far more diversity than 100

11. About the Researcher

Shubham Rasal β€” shubhamrasal.com

I've already worked with this dataset in its earlier form (the compression challenge). Specifically:

  • Built the IMU (200Hz) windowing and video synchronization pipeline
  • Explored frequency-domain and temporal features from the IMU signals
  • Observed the cross-worker motion pattern differences that motivate this proposal
  • Profiled the full dataset structure (100 workers, 12,997 clips, 1.22 TiB)

This prior work means I can move directly to model training on the pilot without pipeline development overhead, and scale to 500K with confidence in the data pipeline.

12. Key References

  1. Chen, B. et al. "COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric HAR." arXiv:2503.07259 (2025).
  2. Zhang, M. et al. "Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognition." arXiv:2407.06628 (2024).
  3. Cai, Y. et al. "Towards Generalizable Human Activity Recognition: A Survey." arXiv:2508.12213 (2025).
  4. Cheshmi, S. et al. "Improving OOD HAR via IMU-Video Cross-modal Representation Learning." arXiv:2507.13482 (2025).
  5. Moon, S. et al. "IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors." arXiv:2210.14395 (2023).
  6. Zhang, X. et al. "UniMTS: Unified Pre-training for Motion Time Series." arXiv:2410.19818 (2024).
  7. Mehta, N. et al. "A Multimodal Dataset for Enhancing Industrial Task Monitoring." arXiv:2501.05936 (2025).
  8. Jain, Y. et al. "ColloSSL: Collaborative Self-Supervised Learning for HAR." arXiv:2202.00758 (2022).
  9. Grauman, K. et al. "Ego4D: Around the World in 3,000 Hours of Egocentric Video." arXiv:2110.07058 (2022).
  10. Grauman, K. et al. "Ego-Exo4D: Understanding Skilled Human Activity." arXiv:2311.18259 (2023).

End of proposal.