VidParse: Online Parsing of Egocentric Procedures Like a Pro

Abstract

Translating continuous, noisy egocentric video into discrete, temporally ordered action steps is fraught with visual challenges. Heavy ego-motion, transient occlusions, and the high intra-class variability of unscripted human–object interactions cause frame-level online temporal models to struggle, often resulting in severe over-segmentation and structural collapse. We present VidParse, an online, training-free framework that treats activity understanding as a graph-constrained inference problem. Rather than relying on learned temporal filters, we identify semantic transitions using a temporal similarity matrix over manipulation-anchored features extracted from frozen foundation models, prioritising foreground hand–object interactions. A beam search decoder then leverages an induced procedural task graph to enforce valid action transitions and prune impossible trajectories. By anchoring robust visual segments to hard procedural constraints, our approach preserves long-range state transitions and achieves up to a 10× improvement in multi-step parsing accuracy over strong online baselines, without a single gradient update.

Watch it parse

Each clip is a real run on a test video. Three bars run under it on a shared clock: ground truth, the parse VidParse has committed to, and the ProTAS baseline on the same frames. Background is dropped from all three so they line up. Alongside is the beam at the current playhead. Scroll it to see which hypotheses are still alive and what sequence each one has accumulated.

Beam at the playhead —

One column per emitted segment, one box per live hypothesis, each joined to the hypothesis it extends. The highlighted chain is the current best path. The remaining boxes are alternatives the graph still allows. Scroll sideways to see earlier segments.

Method

The pipeline has three parts and none of them are trained. We keep a frozen backbone and add two things to it: a choice of where in the frame to pool its features, and a constraint on which action sequences the decoder is allowed to produce.

(a) Manipulation-anchored features

A frozen DINOv2 ViT-L/14 gives patch tokens, and a frozen hand–object detector gives the interaction box. We average only the tokens that fall inside that box, so the descriptor stays on whatever the hands are holding and ignores the rest of the kitchen.

frame → (1024,)

(b) Training-free boundaries

Within an action the frames look alike, so a temporal similarity matrix shows homogeneous blocks along its diagonal. We slide a Gaussian-tapered checkerboard kernel down it and take peaks in the novelty score as boundaries. Nothing is fit to data, and only the kernel window is buffered.

kernel 2 s · peak height 9 · min gap d = 20

(c) Graph-constrained decoding

Each segment is scored against k-means action prototypes, and a beam search picks the best-scoring sequence of actions. The search may only follow edges of a task graph induced from the training sequences, so a transition the graph lacks is never expanded.

B = 10

The constraint the decoder obeys

The graph is induced offline from the training sequences and never sees test video. Click a step to see which steps have to come before it and which ones it makes available. Those are the edges the beam search is restricted to.

may open the procedure optional step may close it first visit revisit 100%

Results

Frame accuracy excluding background, edit distance, and segmental F1 at IoU {0.1, 0.25, 0.5}, over the 53 EgoPER test videos of the ProTAS split.

Method	Inference	Train-free	Acc	Edit	F1@0.1	F1@0.25	F1@0.5
MS-TCN	offline	✗	87.52	92.34	92.60	91.81	86.12
MS-TCN	online	✗	25.40	44.81	44.09	31.78	15.72
ProTAS	online	✗	76.61	65.50	64.26	62.59	51.31
VidParse	online	✓	80.69	88.69	91.14	88.53	77.61

Feature comparison

Every row uses the same decoder and differs only in the features. Restricting the pooling to the interaction box helps more than either a larger backbone or a video–language model.

Representation	MAF	Level	Acc	Edit	F1@0.5
DINOv2 ViT-L, full frame	✗	frame	69.09	82.68	61.42
EgoVLPv2	✗	clip	53.40	62.75	43.66
EgoVLPv2	✓	clip	60.11	70.61	52.10
EgoVLPv2	✗	frame	78.10	89.35	73.02
EgoVLPv2	✓	frame	80.32	91.46	77.48
DINOv2 ViT-L (ours)	✓	frame	80.69	88.69	77.61

Speed

All times are seconds per video. The hand–object detector and the backbone account for almost all of it, while boundary detection and decoding run on CPU. The total is still well under the ProTAS baseline, whose decoder runs on GPU.

Setting	HOD	Backbone	Parse	Total	FPS	Acc	F1@0.5
ProTAS (GPU decoder)	—	79.5	706.0	785.5	4.5	76.6	51.3
Ours, B = 1	339	36.8	24.8	400.6	8.7	58.5	52.1
Ours, B = 3	339	36.8	28.2	404.0	8.7	76.3	72.4
Ours, B = 5	339	36.8	28.4	404.2	8.6	78.5	74.5
Ours, B = 10	339	36.8	32.8	408.6	8.6	80.7	77.6

Does the parse keep its shape?

Frame accuracy asks only whether individual frames carry the right label, and says little about whether the predicted sequence is a plausible procedure. N-step transition accuracy instead counts how many n-step action transitions in the prediction match the ground truth, swept over progressive completion of each video. The gap grows with n: at five and seven steps our AUC is several times the baseline's.

N-step transition precision-recall on EgoPER

Against a frame-level model

ProTAS predicts a label per frame and flickers near the boundaries. Decoding at the segment level under a graph constraint produces longer, steadier runs. Background frames are removed for legibility.

BibTeX

@inproceedings{gupta2026vidparse,
  title         = {VidParse: Online Parsing of Egocentric Procedures Like a Pro},
  author        = {Gupta, Anubhav and Kambhamettu, Archit and Agarwal, Vatsal
                   and Kumar, Pulkit and Shrivastava, Abhinav},
  booktitle     = {European Conference on Computer Vision (ECCV)},
  year          = {2026},
  eprint        = {2608.27562},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}