SG-VAE: Scene Grammar Variational Autoencoder to Generate New Indoor Scenes
Paper in proceedings, 2020

Deep generative models have been used in recent years to learn coherent latent representations in order to synthesize high-quality images. In this work, we propose a neural network to learn a generative model for sampling consistent indoor scene layouts. Our method learns the co-occurrences, and appearance parameters such as shape and pose, for different objects categories through a grammar-based auto-encoder, resulting in a compact and accurate representation for scene layouts. In contrast to existing grammar-based methods with a user-specified grammar, we construct the grammar automatically by extracting a set of production rules on reasoning about object co-occurrences in training data. The extracted grammar is able to represent a scene by an augmented parse tree. The proposed auto-encoder encodes these parse trees to a latent code, and decodes the latent code to a parse tree, thereby ensuring the generated scene is always valid. We experimentally demonstrate that the proposed auto-encoder learns not only to generate valid scenes (i.e. the arrangements and appearances of objects), but it also learns coherent latent representations where nearby latent samples decode to similar scene outputs. The obtained generative model is applicable to several computer vision tasks such as 3D pose and layout estimation from RGB-D data.

Indoor scene synthesis

Scene grammar

VAE

Author

Pulak Purkait

University of Adelaide

Christopher Zach

Chalmers, Electrical Engineering, Signal Processing and Biomedical Engineering, Imaging and Image Analysis

Ian Reid

University of Adelaide

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

03029743 (ISSN) 16113349 (eISSN)

Vol. 12369 LNCS 155-171

16th European Conference on Computer Vision, ECCV 2020
Glasgow, United Kingdom,

Subject Categories

Language Technology (Computational Linguistics)

Bioinformatics (Computational Biology)

Computer Vision and Robotics (Autonomous Systems)

DOI

10.1007/978-3-030-58586-0_10

More information

Latest update

1/4/2021 8