Generative Adversarial Networks for Stochastic Video Prediction with Action Control

Document Type

Article

Publication Date

1-1-2020

Abstract

The ability of predicting future frames in video sequences, known as video prediction, is an appealing yet challenging task in computer vision. This task requires an in-depth representation of video sequences and a deep understanding of real-word causal rules. Existing approaches for tackling the video prediction problem can be classified into two categories: deterministic and stochastic methods. Deterministic methods lack the ability of generating possible future frames and often yield blurry predictions. On the other hand, although current stochastic approaches can predict possible future frames, their models lack the ability of action control in the sense that they cannot generate the desired future frames conditioned on a specific action. In this paper, we propose new generative adversarial networks (GANs) for stochastic video prediction. Our framework, called VPGAN, employs an adversarial inference model and a cycle-consistency loss function to empower the framework to obtain more accurate predictions. In addition, we incorporate a conformal mapping network structure into VPGAN to enable action control for generating desirable future frames. In this way, VPGAN is able to produce fake videos of an object moving along a specific direction. Experimental results show that the combination of VPGAN with a pre-trained image segmentation model outperforms existing stochastic video prediction methods.

Identifier

85083716880 (Scopus)

Publication Title

IEEE Access

External Full Text Location

https://doi.org/10.1109/ACCESS.2020.2982750

e-ISSN

21693536

First Page

63336

Last Page

63348

Volume

8

Grant

KEP-8-611-38

Fund Ref

King Abdulaziz University

This document is currently not available here.

Share

COinS