Artificial-Intelligence-Assisted Multi-Modal Terahertz Sensing and Environment Reconstruction

Journal article, 2026

Multi-modal sensing is an important enabler for future environment-aware wireless systems, since a single sensing modality is generally insufficient to provide accurate metric geometry, material awareness, and semantic interpretability in complex environments. This paper presents a measurement-based multi-modal terahertz (THz) sensing and vision framework for indoor environment reconstruction. A three-dimensional monostatic THz channel sounding system operating at 290–310 GHz is integrated with an omnidirectional fisheye camera to acquire radio-frequency and visual observations from a common sensing viewpoint. From the measured THz data, a signal processing pipeline extracts multipath components and infers geometry-and material-consistent structural primitives through trajectory-tracking-assisted parameter estimation, graph-based structure discovery, planar reconstruction, and reflection-loss analysis. In parallel, artificial intelligence (AI)-based visual perception modules extract object-level semantic masks and depth priors from panoramic images. To associate these heterogeneous representations, an agentic-AI-based task-driven THz-agent module is developed to select appropriate integration tools according to the attributes of the modality-specific outputs. Through angular alignment and consistency analysis, THz-derived metric geometry and material information are associated with vision-derived semantic regions and depth priors, enabling geometry-consistent and semantically interpretable environment reconstruction directly from measurements. Experimental validation in the indoor L-shaped hallway demonstrates that the proposed framework reconstructs dominant structural elements with centimeter-level accuracy while identifying semantic categories and material attributes of representative indoor objects. These results show the potential of THz–vision integration for environment-aware sensing, wireless digital twins, and future ISAC systems.