CVPR-2022 action segmentation - Honda Research Institute USA

CVPR-2022 action segmentation

Weakly-Supervised Online Action Segmentation in Multi-view Instructional Videos

R. Ghoddoosian N. Agarwal I. Dwivedi C. Choi and B. Dariush

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022

This paper addresses a new problem of weaklysupervised online action segmentation in instructional videos. We present a framework to segment streaming videos online at test time using Dynamic Programming and show its advantages over greedy sliding window approach. We improve our framework by introducing the Online-Offline Discrepancy Loss (OODL) to encourage the segmentation results to have a higher temporal consistency. Furthermore, only during training, we exploit framewise correspondence between multiple views as supervision for training weakly-labeled instructional videos. In particular, we investigate three different multi-view inference techniques to generate more accurate frame-wise pseudo ground-truth with no additional annotation cost. We present results and ablation studies on two benchmark multi-view datasets, Breakfast and IKEA ASM. Experimental results show efficacy of the proposed methods both qualitatively and quantitatively in two domains of cooking and assembly.

Downloadable item