Document Type
Dissertation
Date of Award
8-31-2021
Degree Name
Doctor of Philosophy in Computing Sciences - (Ph.D.)
Department
Computer Science
First Advisor
Jason T. L. Wang
Second Advisor
Ioannis Koutis
Third Advisor
Jing Li
Fourth Advisor
Katherine Grace Herbert
Fifth Advisor
Eric Liguori
Abstract
Deep Learning has spanned a variety of applications in computer vision as well as computational astronomy. These two aspects obtained similar data structure, therefore, their solutions can be transferable between each other. This dissertation look into two video-related tasks in computer vision and propose a novel problem in computational astronomy.
Specifically, acquiring an in-depth understanding of videos has been a cornerstone problem in computer vision. This problem has been studied by various researchers from different perspectives, among which video prediction has attracted much attention. Video prediction aims to generate the pixels of future frames given a sequence of context frames. In practice, unlabeled video sequences can be gathered autonomously from a sensor or recording device. A machine capable of predicting future events using these video sequences in an unsupervised manner will gain extensive and deep knowledge about its physical environment and surroundings. However, despite its appealing prospects, accurate video prediction remains an open problem. The major challenge is the inherent uncertainty in the dynamics of the world. A typical example is that the future trajectory of a ball hitting the ground is inherently random.
This dissertation proposes new generative adversarial networks (GANs) for stochastic video prediction. The proposed framework, dubbed Video-Prediction-GAN(VPGAN), employs an adversarial inference model and a cycle-consistency loss function to empower the framework to obtain more accurate predictions. In addition, a conformal mapping network structure is incorporated into VPGAN to enable action control for generating desirable future frames. In this way, VPGAN can produce fake videos of an object moving along a specific direction. Experimental results based on different datasets demonstrate the good performance of VPGAN and its superiority over existing methods. Other contributions of this dissertation include the development of a Three-Dimensional atrous convolutional long short-term memory network for background subtraction used in video processing and an extension of VPGAN for synthesizing vector magnetograms of active regions in different solar cycles. The dissertation concludes by pointing out some directions of future research in applying deep learning to computer vision and astronomy.
Recommended Citation
Hu, Zhihang, "Advances in deep learning with applications to computer vision and astronomy" (2021). Dissertations. 1704.
https://digitalcommons.njit.edu/dissertations/1704
