Techniques for Fast and High-Quality 3D Reconstruction of General Scenes
Doctoral thesis, 2022
This thesis is a collection of techniques used for 3D reconstruction; the creation of 3D models from real world objects or scenes. Given the increase in accuracy, robustness and speed of modern methods and algorithms, new and exciting applications of this technology is constantly appearing. Asset creation for games and movies is a successful example, but there are numerous other applications in architecture, medicine, communication and more.
The contribution of each paper in this thesis aims to make the use of 3D reconstruction even moreubiquitous by addressing problems such as performance, memory usage, ease-of-use, robustnessand quality.
Paper I presents a compression technique for volumetric video modeled with voxels. Memory consumption is an important issue when storing volume data, especially if the data is also varying with time.
Paper II describes an end-to-end pipeline for recording and rendering volumetric video. A simple and readily available setup of webcams and a single desktop computer is used to record and render scenes in real-time.
In Paper III, an interactive tool is developed that aims to help in modeling of real-world objects. Structured as a simple quad modeling program, the user can construct 3D models on top of a set of photographs of a chosen object. In the background, or after explicit activation, a multi-view stereo algorithm helps the user to align the geometry correctly to images in world space. This greatly simplifies the problem of modeling real world objects accurately, while levering the input from the user to help with topology and visibility.
Paper IV implements a direct solver for the problem of neural rendering. The reconstruction is formulated as a non-linear least-squares problem which is solved efficiently with the Gauss Newton method and the Preconditioned Conjugate Gradient algorithm. This formulation achieves a significant improvement to reconstruction times compared to previous methods, while also being suitable for distributed computing due to needing three order of magnitudes fewer iterations until convergence.
Paper V handles the shape-radiance ambiguity in neural rendering. Given infinite spatial resolution of view-dependent information, almost any shape can satisfy the incoming radiance to each camera, resulting in errors in the geometry. To address this problem, we propose a solution to separate Lambertian and view-dependent colors
during reconstruction.
The contribution of each paper in this thesis aims to make the use of 3D reconstruction even moreubiquitous by addressing problems such as performance, memory usage, ease-of-use, robustnessand quality.
Paper I presents a compression technique for volumetric video modeled with voxels. Memory consumption is an important issue when storing volume data, especially if the data is also varying with time.
Paper II describes an end-to-end pipeline for recording and rendering volumetric video. A simple and readily available setup of webcams and a single desktop computer is used to record and render scenes in real-time.
In Paper III, an interactive tool is developed that aims to help in modeling of real-world objects. Structured as a simple quad modeling program, the user can construct 3D models on top of a set of photographs of a chosen object. In the background, or after explicit activation, a multi-view stereo algorithm helps the user to align the geometry correctly to images in world space. This greatly simplifies the problem of modeling real world objects accurately, while levering the input from the user to help with topology and visibility.
Paper IV implements a direct solver for the problem of neural rendering. The reconstruction is formulated as a non-linear least-squares problem which is solved efficiently with the Gauss Newton method and the Preconditioned Conjugate Gradient algorithm. This formulation achieves a significant improvement to reconstruction times compared to previous methods, while also being suitable for distributed computing due to needing three order of magnitudes fewer iterations until convergence.
Paper V handles the shape-radiance ambiguity in neural rendering. Given infinite spatial resolution of view-dependent information, almost any shape can satisfy the incoming radiance to each camera, resulting in errors in the geometry. To address this problem, we propose a solution to separate Lambertian and view-dependent colors
during reconstruction.
3D reconstruction
voxels
volumetric video
computer vision
computer graphics
Author
Sverker Rasmuson
Embedded Electronics Systems and Computer Graphics
A low-cost, practical acquisition and rendering pipeline for real-time free-viewpoint video communication
Visual Computer,;Vol. 37(2021)p. 553-565
Journal article
User-guided 3D reconstruction using multi-view stereo
Proceedings - I3D 2020: ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games,;(2020)
Paper in proceeding
Exploiting coherence in time-varying voxel data
Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games,;(2016)p. 15-21
Paper in proceeding
PERF: Performant, Explicit Radiance Fields
Frontiers in Computer Science,;Vol. 4(2022)
Journal article
Our lives are increasingly being lived in digital spaces. Our work life, education, social life, and entertainment
are more often than not performed in front of a screen. A bridge between the real world and the digital virtual world can be achieved
with 3D reconstruction. When 3D reconstructing a person, an object, or a room, 3D models are created that can represent this piece
of the real world in a virtual environment.
The most common way of performing such a 3D reconstruction is through a set of photographs or video streams. High-quality camera
equipment, in the form of smartphones, are nowadays available to almost everyone. This means that the available data for creating 3D
models of the real world is larger and more accessible than ever. In combination with the ever more powerful graphics hardware, and new
interaction devices such as virtual reality goggles, new applications for this technology is popping up every day.
In this thesis a wide variety of scenarios for 3D reconstruction are explored, ranging from faces to toys to outdoor scenes. Both
static scenes of individual objects and dynamic scenes of moving persons are considered. A challenging aspect of 3D reconstruction is
the wide variety of types of objects and materials that can occur in a given scene. The work in this thesis show that many types of
general scenes can be handled with modern methods and techniques, while still offering good performance and high quality of
reconstruction.
are more often than not performed in front of a screen. A bridge between the real world and the digital virtual world can be achieved
with 3D reconstruction. When 3D reconstructing a person, an object, or a room, 3D models are created that can represent this piece
of the real world in a virtual environment.
The most common way of performing such a 3D reconstruction is through a set of photographs or video streams. High-quality camera
equipment, in the form of smartphones, are nowadays available to almost everyone. This means that the available data for creating 3D
models of the real world is larger and more accessible than ever. In combination with the ever more powerful graphics hardware, and new
interaction devices such as virtual reality goggles, new applications for this technology is popping up every day.
In this thesis a wide variety of scenarios for 3D reconstruction are explored, ranging from faces to toys to outdoor scenes. Both
static scenes of individual objects and dynamic scenes of moving persons are considered. A challenging aspect of 3D reconstruction is
the wide variety of types of objects and materials that can occur in a given scene. The work in this thesis show that many types of
general scenes can be handled with modern methods and techniques, while still offering good performance and high quality of
reconstruction.
Subject Categories (SSIF 2011)
Computational Mathematics
Computer Science
Computer Vision and Robotics (Autonomous Systems)
ISBN
978-91-7905-629-2
Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 5095
Publisher
Chalmers