Xingdou Fu (付 星斗)

[2024] The First High-Speed Active vision Bin Picking in The World

A Low-Cost, High-Speed, and Robust Bin Picking System for Factory Automation Enabled by a Non-stop, Multi-View, and Active Vision Scheme

Xingdou Fu, Lin Miao, Yasuhiro Ohnishi, Yuki Hasegawa, Masaki Suwa
OMRON
Accepted by The 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024)
Project website  Paper(IROS 2024)

Bin picking in factory automation often struggles with sparse, noisy 3D data of metallic objects. Multi-view “sensor on hand” approaches improve robustness and flexibility but are usually slow, as sensing is decoupled from motion tasks. We designed a bin picking system that tightly integrates high-speed, multi-view active vision with robot motion. This parallelization not only accelerates sensing—averaging 0.635 s in takt time—but also plans the next view to maintain workflow continuity. Tested on five object types without human intervention, our system achieves a maximum sensing time of 1.682 s and an average pick rate over 97.75%. To our knowledge, this is the first high-speed active vision bin picking system.

[2018] A hybrid single-shot 3D sensor

A single-shot 3D sensor combining multiple sensing modalities with only a single projection pattern

Xingdou Fu, Masaki Suwa
OMRON
US Patent Granted (US10600193)
Patent document

We designed the first single-shot 3D sensor that combines two complementary modalities: structured-light and active stereo, enabled by a hybrid single-pattern projection. The pattern combines a structured code to enable high-accuracy 3D reconstruction on smooth surfaces, and random noise elements to enable robust stereo matching on non-smooth, complex-shaped, or featureless surfaces. This dual-function design overcomes the limitations of traditional single-modality sensors, delivering precise 3D measurements in a single capture. By unifying these capabilities in a single projection, the sensor achieves fast, reliable 3D reconstruction without multiple captures.
This technology is ideally suited for industrial applications, such as object pose estimation on conveyor belts or robotic bin picking, where high measurement precision, robustness, and computational efficiency are critical. The hybrid single-pattern design represents a practical advancement in 3D sensing, providing an efficient, accurate, and robust solution for complex manufacturing and automation environments.

[2013] Calibration of Projector with Fixed Pattern and Large Distortion
under 3D Reconstruction System for Capturing moving objects of Human body
sponsored by Japan Society for the Promotion of Science (JSPS)

Xingdou Fu Zuofu Wang, , Hiroshi Kawasaki, Ryusuke Sagawa, Ryo Furukawa
The computer vision and graphics lab, Kagoshima University, Japan
Accepted by The Thirteenth IAPR International Conference on Machine Vision Applications (MVA 2013)
Project website  Paper(MVA 2013)

This research contributes to Hyper3DSensing, which develops ultra-fast, high-precision 3D sensing technologies for dynamic shape measurement. Applications include real-time human body capture, industrial inspection, and compact medical devices such as 3D endoscope scanners, where accurate reconstruction under strict optical and hardware constraints is essential. The project focuses on a specialized projector calibration for structured light systems with fixed patterns and wide-angle optics—conditions where conventional methods often fail. Fixed-pattern projectors suit miniaturized devices because they enable compact, lightweight designs. This calibration ensures geometric consistency and depth accuracy, enabling reliable high-speed 3D measurement in compact systems.

[2009-2010] Reconstruction of Display and Eyes from a Single Image

Dirk Schnieders,Xingdou Fu Under supervision of Prof.Kwan-Yee K. Wong
Vision group, The University of Hong Kong (HKU), Hong Kong SAR
Accepted by The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2010)
Project website  Paper(CVPR 2010)

This paper introduces a novel method for reconstructing human eyes and visual display from reflections on the cornea. This problem is difficult because the camera is not directly facing the display, but instead captures the eyes of a person in front of the display. Reconstruction of eyes and display is useful for point-of-gaze estimation, which can be approximated from the 3D positions of the iris and display. It is shown that iris boundaries (limbus) and display reflections in a single intrinsically calibrated image provide enough information for such an estimation. The proposed method assumes a simplified geometric eyeball model with certain anatomical constants which are used to reconstruct the eye. A noise performance analysis shows the sensitivity of the proposed method to imperfect data. Experiments on various subjects show that it is possible to determine the approximate area of gaze on a display from a single image.

[2007-2008] Multi-Camera 3D Facial Motion Capture

Early pioneer work in building 3D face expressioin recognition datasets worldwide

Xingdou Fu, Yuxiao Hu, Hao Tang, Under supervision of Prof.Thomas Huang
IFP Lab, University of Illinois Urbana-Champaign, USA
Project website  Paper

This project aims to build a prototype system to capture the 3D motion of key facial feature points for constructing a feature-based 3D facial motion database, supporting future 3D expression recognition research. The captured data can also be used to drive a 3D face model for realistic facial animation. The system uses five synchronized cameras arranged in a semicircle to form four stereo vision pairs for full-face 3D measurement. Colored markers are placed on predefined facial points. Each stereo pair performs dynamic 3D reconstruction within its region, and all data are unified in a global coordinate system. Markers are assigned unique IDs and tracked across frames using the CamShift algorithm. Epipolar constraints are applied to verify tracking consistency and improve robustness.

[2004-2006] 3D Realistic Portrait Data Acquisition

Early pioneer work in building 3D face recognition datasets worldwide

5-10 years earlier than consumer 3D sensing device Microsoft Kinect and Intel Realsense!

Xingdou Fu, Prof. Pingjiang Wang, Prof. Xiaoqi Tang, Prof. Jihong Chen
Huazhong University of Science and Technology, China
Project website  Paper

This project is funded by the National Natural Science Foundation of China (Project ID: 50175033) and is considered early pioneer work in building 3D face recognition datasets worldwide through 3D portrait measurement. Our 3D facial acquisition system predates consumer scanners like Kinect Fusion and Intel RealSense by 5–10 years. The 3D points were laser-engraved in glass to showcase precision, but the data also supports applications like 3D face recognition.
We developed a grating-projection structured-light 3D measurement system using a customized DSLR Nikon camera and a self-made structured-light projector, which aims to acquire high-accuracy, natural-looking 3D portrait data. The system captures two quick images—one with grating projection for 3D points, and one with normal lighting for grayscale. The grayscale image is then dithered and mapped onto the 3D mesh, rearranging the 3D points to achieve a more realistic and natural effect.

[2001-2004] GIS（Geographic Information System）platform(MapGIS) and information management system

Research and Applications of Distributed Systems for Mineral Resource Planning

Ministry of Land and Resources (MLR)：Land Management System

Qinhuangdao Bureau of Land and Resources：Land Management System

Xingdou Fu, Prof. Wu Xincai
China University of Geosciences, China / Zondy Cyber Group (University spin-off)

During my Master studies, I was actively involved in both the research and development of algorithms for GIS platforms, as well as the development of GIS-based systems. This work was carried out in collaboration with Zondy Cyber, a research-oriented company and university spin-off originating from the China University of Geosciences. The company was founded by my supervisor, Prof. Wu Xincai, and its team largely consists of university professors as well as Master’s and PhD students. It develops MapGIS, one of China’s leading Geographic Information System (GIS) platforms, widely recognized as one of the earliest domestically developed GIS software systems to achieve practical, real-world application.