Ji Hou (侯骥)

I am a Research Scientist at Meta GenAI , working on foundation models. Previously, I was a Research Scientist at XR Tech in Meta Reality Labs working on 3D Scene Understanding. Prior to joining Meta, I did my Ph.D. at TUM Visual Computing Group headed by Prof. Matthias Niessner, where I work on Computer Vision and 3D Scene Understanding. During my Ph.D., I did an internship at Facebook AI Research (FAIR) with Prof. Saining Xie and Dr. Benjamin Graham on 3D representation and data-efficient learning. Before that, I obtained my master at RWTH Computer Vision Group headed by Prof. Bastian Leibe, where I studied on Computer Vision and Machine Learning.

I am interested in research and applications on Generative Models on Image/Video/3D, as well as 3D Computer Vision, e.g. 3D Reconstruction, VR/AR, Robotics and Autonomous Driving etc.

  Email  /    Google Scholar  /    Github  /    Twitter  /    Linkedin

controlroom3d ControlRoom3D: Room Generation using Semantic Proxy Rooms
Jonas Schult, Sam Tsai, Lukas Höllein, Bichen Wu, Jialiang Wang, Chih-Yao Ma,
Kunpeng Li, Xiaofang Wang, Felix Wimbauer, Zijian He, Peizhao Zhang,
Bastian Leibe, Peter Vajda, Ji Hou
Computer Vision and Pattern Recognition (CVPR), 2024
arxiv / project page / video / bibtex

Given a textual description of the overall room style and a rough 3D room layout based on 3D semantic bounding boxes, our method called ControlRoom3D creates diverse and globally plausible 3D room meshes which align well with the room layout.

CacheMeIfYouCan Cache Me if You Can: Accelerating Diffusion Models through Block Caching
Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He,
Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, Christian Rupprecht,
Daniel Cremers, Peter Vajda, Jialiang Wang
Computer Vision and Pattern Recognition (CVPR), 2024
arxiv / project page / bibtex

Our block caching technique allows us to avoid these unnecessary computations, therefore speeding up inference by a factor of 1.5x-1.8x while maintaining image quality.

nerf_det RIGA: Rotation-Invariant and Globally-Aware Descriptors for Point Cloud Registration
Hao Yu, Ji Hou, Zheng Qin, Mahdi Saleh, Ivan Shugurov, Kai Wang, Benjamin Busam, Slobodan Ilic
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024
paper / arxiv / bibtex

Successful point cloud registration relies on accurate correspondences established upon powerful descriptors. However, existing neural descriptors either leverage a rotation-variant backbone whose performance declines under large rotations, or encode local geometry that is less distinctive. To address this issue, we introduce RIGA to learn descriptors that are Rotation-Invariant by design and Globally-Aware.

nerf_det NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection
Chenfeng Xu, Bichen Wu, Ji Hou *, Sam Tsai, Ruilong Li, Jialiang Wang, Wei Zhan Zijian He, Peter Vajda, Kurt Keutzer, Masayoshi Tomizuka
International Conference on Computer Vision (ICCV), 2023
paper / bibtex / project / code
* Corresponding author.

NeRF-Det is a novel method for 3D detection with posed RGB images as input. Our method makes novel use of NeRF in an end-to-end manner to explicitly estimate 3D geometry, thereby improving 3D detection performance.

RoITr Rotation-Invariant Transformer for Point Cloud Matching
Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam, Slobodan Ilic
Computer Vision and Pattern Recognition (CVPR), 2023
paper / bibtex / code

We introduce RoITr, a Rotation-Invariant Transformer to cope with the pose variations in the point cloud matching task. On the challenging 3DLoMatch benchmark, RoITr surpasses the existing methods by at least 13 and 5 percentage points in terms of the Inlier Ratio and the Registration Recall, respectively

mask3d Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors
Ji Hou, Xiaoliang Dai, Zijian He, Angela Dai, Matthias Nießner
Computer Vision and Pattern Recognition (CVPR), 2023
paper / video / bibtex

We demonstrate the Mask3D is particularly effective in embedding 3D priors into the powerful 2D ViT backbone, enabling improved representation learning for various scene understanding tasks, such as semantic segmentation, instance segmentation and object detection.

pcr_cg PCR-CG: Point Cloud Registration via Deep Explicit Color and Geometry
Yu Zhang, Junle Yu, Xiaolin Huang, Wenhui Zhou, Ji Hou
European Conference on Computer Vision (ECCV), 2022
paper / video / bibtex / poster / code

We introduce PCR-CG: a novel 3D point cloud registration module explicitly embedding the color signals into geometry representation. With our designed 2D-3D projection module, the pixel features in a square region centered at correspondences perceived from images are effectively correlated with point cloud representations.

pri3d Pri3D: Can 3D Priors Help 2D Representation Learning?
Ji Hou, Saining Xie, Benjamin Graham, Angela Dai, Matthias Nießner
International Conference on Computer Vision (ICCV), 2021
paper / video / bibtex / project / code

Recent advances in 3D perception have shown impressive progress in understanding geometric structures of 3D shapes and even scenes. Inspired by these advances in geometric understanding, we aim to imbue image-based perception with representations learned under geometric constraints.

panoptic3d Panoptic 3D Scene Reconstruction From a Single RGB Image
Manuel Dahnert, Ji Hou, Matthias Nießner Angela Dai
Advances in Neural Information Processing System (NeurIPS), 2021
paper / video / bibtex / project / code

Inspired by 2D panoptic segmentation, we propose to unify the tasks of geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation into the task of panoptic 3D scene reconstruction -- from a single RGB image, predicting the complete geometric reconstruction of the scene in the camera frustum of the image, along with semantic and instance segmentations.

3dsis Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts
Ji Hou, Benjamin Graham, Matthias Nießner, Saining Xie
Computer Vision and Pattern Recognition (CVPR), 2021
(Oral Presentation)
paper / video / bibtex / project / code / benchmark

Our study reveals that exhaustive labelling of 3D point clouds might be unnecessary; and remarkably, on ScanNet, even using 0.1% of point labels, we still achieve 89% (instance segmentation) and 96% (semantic segmentation) of the baseline performance that uses full annotations.

3dsis RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction
Yinyu Nie, Ji Hou, Xiaoguang Han, Matthias Nießner
Computer Vision and Pattern Recognition (CVPR), 2021
paper / video / bibtex / code

In this work, we introduce RfD-Net that jointly detects and reconstructs dense object surfaces directly from raw point clouds. Instead of representing scenes with regular grids, our method leverages the sparsity of point cloud data and focuses on predicting shapes that are recognized with high objectness.

revealnet RevealNet: Seeing Behind Objects in RGB-D Scans
Ji Hou, Angela Dai, Matthias Nießner
Computer Vision and Pattern Recognition (CVPR), 2020
paper / video / bibtex / project

This paper introduces the task of semantic instance completion: from an incomplete, RGB-D scan of a scene, we detect the individual object instances comprising the scene and jointly infer their complete object geometry.

3dsis 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans
Ji Hou, Angela Dai, Matthias Nießner
Computer Vision and Pattern Recognition (CVPR), 2019
(Oral Presentation)
paper / video / bibtex / project / code

We introduce 3D-SIS, a novel neural network architecture for 3D semantic instance segmentation in commodity RGB-D scans. The core idea of our method is to jointly learn from both geometric and color signal, thus enabling accurate instance predictions.

Teaching Assistant, Seminar for 3D Machine Learning - Summer 2021
Teaching Assistant, Advanced Deep Learning for Computer Vision - Winter 2019/20 Teaching Assistant, Introdcution to Deep Learning - Summer 2018
Teaching Assistant, Deep Learning for Computer Vision - Winter 2017/18
Review Experiences
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
ACM Transactions on Multimedia Computing Communications and Applications (TOMM)
International Journal of Computer Vision (IJCV)
ISPRS Journal of Photogrammetry and Remote Sensing
IEEE Robotics and Automation Letters (RA-L)
IEEE Transactions on Image Processing (TIP)
Pattern Recognition Letters
Computers & Graphics
Special Interest Group on Computer Graphics and Interactive Techniques (SIGGRAPH)
Conference on Computer Vision and Pattern Recognition (CVPR)
International Conference on Computer Vision (ICCV)
European Conference on Computer Vision (ECCV)
International Conference on Machine Learning (ICML)
Neural Information Processing Systems (NeurIPS)
International Conference on Learning Representations (ICLR)
International Conference on Robotics and Automation (ICRA)
Association for the Advancement of Artificial Intelligence (AAAI)
International Joint Conference on Artificial Intelligence (IJCAI)
The 2nd Computer Vision for Metaverse Workshop at ICCV 2023
GenAI Media Generation Challenge Workshop at CVPR 2024
Efficient Deep Learning for Computer Vision Workshop at CVPR 2024
Visiting Map

Credits: Jon Barron