Sustainable Development Goals
Abstract/Objectives
This project aims at breaking the limited application of panoramic images by enriching the editability and interactivity of captured panoramas through innovative deep learning, augmented reality, and smart manipulation technologies. In the end application, our team collaborates with iStaging Co., Ltd to build an automatic system called “Smart 3D Reconstruction from Panoramas”. Our system automatically converts the 2D panoramas captured by the real-estate agents into 3D layouts, which can be further rendered as an interactive 3D content via WebGL. This technology has been used in the commercial production including immersive real-estate live-tour and online virtual exhibition. Our team also closely collaborates with ITRI to enhance the image quality of autonomous car simulation by introducing a novel panoramic lighting and shading technology.
Results/Contributions

1. Develop a novel deep learning architecture that can convert a single indoor panorama into a 3D layout. Our method achieves state-of-the-art performance and has published in the top computer vision conference (CVPR 2019) and journal (IJCV).

2. Build a SaaS for semi-automatic conversion between 2D panorama and 3D layout, and a semi-automatic virtual exhibition design pipeline. Both systems are exploited to speed up the creation of real-estate live touring and virtual exhibition.

3. Assist a global company, iStaging, to import our core technologies, including 2D-to-3D SaaS, virtual exhibition pipeline, and 3D viewer SDK, into their product lines to create new business models under the covid-19 pandemic.

Keywords
Virtual reality360 panoramaindoor 3D reconstructiondeep learningreal-estate live-touringautonomous car simulatorimage-based renderingimage editing
References
1. https://cgv.cs.nthu.edu.tw/projects/LayoutReview

Manhattan Room Layout Reconstruction from a Single 360° image: A Comparative Study of State-of-the-art Methods, IJCV 2021

2. https://cgv.cs.nthu.edu.tw/projects/dulanet

Manhattan Room Layout Reconstruction from a Single 360° image: A Comparative Study of State-of-the-art Methods, IJCV 2021

3. https://cgv.cs.nthu.edu.tw/projects/panoannotator

PanoAnnotator: A Semi-Automatic Tool for Indoor Panorama Layout Annotation, SIGGRAPH Asia 2018, Poster

4. https://cgv.cs.nthu.edu.tw/projects/360SP

Generating 360 Outdoor Panorama Dataset with Reliable Sun Position Estimation, SIGGRAPH Asia 2018, Poster

5. https://arxiv.org/abs/1811.05304

Self-Supervised Learning of Depth and Camera Motion from 360 Videos, ACCV 2018

6. https://arxiv.org/abs/1711.08664

Self-view Grounding Given a Narrated 360&deg Video, AAAI 2018

7. https://arxiv.org/abs/1806.01320

Cube Padding for Unsupervised Saliency Prediction in 360 Videos, CVPR 2018

8. http://cgv.cs.nthu.edu.tw/home

Compute Graphics and Vision LAB

Contact Information
朱宏國教授
hkchu@cs.nthu.edu.tw