ICICLE: An Interactive VLM-based System for Information Extraction from Ambiguous Engineering Diagram Legends

Paper i proceeding, 2026

Engineering legend sheets are vital in Engineering, Procurement, and Construction (EPC) projects, visually defining complex assemblies. Automating information extraction from these legends can significantly reduce manual effort, yet their spatially ambiguous layouts pose barriers to digital transformation. To address this, we introduce ICICLE (In-Context Interactive Cue-guided Legend Extractor), a Generative AI system for parsing these documents. ICICLE's core innovation is the In-Context Multimodal Annotation Prompting (ICMAP) method. This technique instructs a Vision Language Model (VLM) by providing a single annotated visual example alongside a textual prompt. This enables the VLM to ground abstract concepts like "detailed assembly"in visual evidence, making the system adaptable without per-query annotation or costly model retraining. Validated on legend sheets from four industrial projects, ICMAP achieved superior extraction accuracy (96-100%) compared to established alternatives and demonstrated robustness across varied visual examples. An ablation study confirms that the tight coupling of visual annotations and textual instructions is critical for success. ICICLE serves as a successful case study for operationalizing Generative AI, offering a blueprint for solving complex information extraction tasks in challenging industrial domains.