APSIPA Transactions on Signal and Information Processing > Vol 5 > Issue 1

Occlusion-aware temporal frame interpolation in a highly scalable video coding setting

Dominic Rüfenacht, University of New South Wales, Australia, d.ruefenacht@unsw.edu.au , Reji Mathew, University of New South Wales, Australia, David Taubman, University of New South Wales, Australia
 
Suggested Citation
Dominic Rüfenacht, Reji Mathew and David Taubman (2016), "Occlusion-aware temporal frame interpolation in a highly scalable video coding setting", APSIPA Transactions on Signal and Information Processing: Vol. 5: No. 1, e8. http://dx.doi.org/10.1017/ATSIP.2016.8

Publication Date: 01 Apr 2016
© 2016 Dominic Rüfenacht, Reji Mathew and David Taubman
 
Subjects
 
Keywords
Temporal frame interpolationTemporal frame upsamplingGeometrically consistent predictionOcclusion-handlingMotion discontinuity modelingBidirectional hierarchical anchoring of motion fields
 

Share

Open Access

This is published under the terms of the Creative Commons Attribution licence.

Downloaded: 1087 times

In this article:
I. INTRODUCTION 
II. OVERVIEW 
III. BIDIRECTIONAL HIERARCHICAL ANCHORING OF MOTION FIELDS 
IV. HIERARCHICAL WARPING OF MOTION FIELD DISCONTINUITIES 
V. MOTION FIELD OPERATIONS 
VI. MOTION-COMPENSATED TEMPORAL FRAME INTERPOLATION 
VII. ESTIMATION OF PIECEWISE-SMOOTH MOTION FIELDS WITH DISCONTINUITIES 
VIII. EXPERIMENTAL EVALUATION AND DISCUSSION 
IX. CONCLUSIONS AND FUTURE WORK 

Abstract

We recently proposed a bidirectional hierarchical anchoring (BIHA) of motion fields for highly scalable video coding. The BIHA scheme employs piecewise-smooth motion fields, and uses breakpoints to signal motion discontinuities. In this paper, we show how the fundamental building block of the BIHA scheme can be used to perform bidirectional, occlusion-aware temporal frame interpolation (BOA-TFI). From a “parent” motion field between two reference frames, we use information about motion discontinuities to compose motion fields from both reference frames to the target frame; these then get inverted so that they can be used to predict the target frame. During the motion inversion process, we compute a reliable occlusion mask, which is used to guide the bidirectional motion-compensated prediction of the target frame. The scheme can be used in any state-of-the-art codec, but is most beneficial if used in conjunction with a highly scalable video coder which employs piecewise-smooth motion fields with motion discontinuities. We evaluate the proposed BOA-TFI scheme on a large variety of natural and challenging computer-generated sequences, and our results compare favorably to state-of-the-art TFI methods.

DOI:10.1017/ATSIP.2016.8