Chapter 9: Motion estimation

• The Middlebury optical flow evaluation website, https://vision.middlebury.edu/flow (Baker, Scharstein et al. 2011) continues to be used for evaluation, since it contains a variety of short real-world sequences.

Most optical flow algorithms are evaluated on the Sintel dataset, http://sintel.is.tue.mpg.de (Butler, Wulff et al. 2012), since it contains both training and test subsets and an active leaderboard, although the videos are stylized computer animations.

• Many algorithms also train and test on the KITTI flow benchmark (Geiger, Lenz, and Urtasun 2012), although it only contains videos acquired from a driving vehicle. The computer-generated sequences in the Visual PERception (VIPER) benchmark (Richter, Hayder, and Koltun 2017) also contain driving sequences. Mayer, Ilg et al. (2018, Table 1) tabulates widely-used datasets for optical flow and depth estimation and shows some sample images in Figure 1.

• A comparison of flow algorithm performance across different datasets (listed in Table C.1) can be found in the Robust Vision Challenge workshop (http://www.robustvision.net).

• For video object segmentation, the Densely Annotated Video Segmentation (DAVIS) dataset Pont-Tuset, Perazzi et al. (2017) contains a set of widely-used evaluation video clips with ground-truth segmentation data. There is also a newer, larger, dataset called YouTube-VOS (Xu, Yang et al. 2018) with its own associated set of challenges and leaderboards.

Datasets for video object tracking (VOT) and multiple object tracking (MOT) can be found at the associated workshops (Kristan, Leonardis et al. 2020; Dendorfer, Ösep et al. 2021). A wider range of objects to track can be found in the Track Any Object (TAO) dataset by Dave, Khurana et al. (2020).