Scene Reasoning with Vision-Language Models and 3D Scene Graphs
Autonomous Robots Lab · Norwegian University of Science and Technology (NTNU)
We study how robots can combine vision-language models (VLMs) with structured scene representations to understand and reason about indoor environments. A robot may need to find an object, assess its relationship to other objects, or decide where to explore when the information required for a task is missing. These decisions depend on a representation that keeps track of what has been observed and connects it to the environment.
Incremental hierarchical 3D scene graphs organize a scene into geometry, objects, navigational nodes, and rooms. Visual observations and learned features attached to the graph enable retrieval and reasoning. The three papers below examine how to gather task-relevant observations with VLMs, reason about object relationships, and maintain an open-vocabulary graph as the robot moves. They address different parts of the same problem: connecting a language task to what the robot has observed and what it still needs to find out.
Traditional simultaneous localization and mapping (SLAM) methods reconstruct metric maps from sensors like cameras, LiDARs, radars, or inertial measurement units (IMU), and can be extended to closed- or open-vocabulary metric-semantic maps using vision foundation models. However, these approaches lack a higher level of abstraction and object reasoning. 3D scene graphs (Hydra) address this gap by capturing hierarchical and relational information. Hierarchical models represent indoor scenes at multiple abstraction levels while also encoding geometrical and semantic inter- and intra-layer relationships.
The figure below illustrates this hierarchy. The layers are described in more detail below.
A hierarchical 3D scene graph connects reconstructed geometry to objects, places, frontiers, rooms, and a building.
Metric-semantic mesh
Reconstructed surfaces with position, color, and semantic labels. This provides the geometry from which higher-level entities are extracted.
Objects/Segments
Individual object instances with a 3D position, bounding box, semantic information, and open-vocabulary features for retrieval.
Places
Nodes representing traversable free space and its connectivity. These form a navigation graph.
Frontiers
Boundaries of the observed space, indicating areas that have not yet been explored. They may contain semantic information.
Rooms
Groups of places forming room regions. Room features provide semantic context for search and reasoning.
Building
A top-level node grouping the rooms.
Inter-layer edges connect levels of the hierarchy, such as places to their rooms. Intra-layer edges connect nodes at the same level, such as adjacent rooms or semantically related objects. In the figure, links between room nodes illustrate intra-layer connectivity, while links from the building to rooms and from rooms to places illustrate inter-layer connectivity. ReasoningGraph enriches object edges with visual relationship features. HFLEX-EQA includes semantic frontiers at the boundary of observed space, and TrackGraph maintains a class-agnostic 3D segment layer. The hierarchy is adapted to the information each task needs.
Vision-language Models and Open-vocabulary Features
A VLM processes images and language together. In these works, its input may include selected images, graph information in text or embeddings format, and a task description. This allows it to reason about the environment and enhance decision making. The graph contains spatial and semantic information, while visual observations provide details such as an object's state or placement. Open-vocabulary vision-language models such as CLIP encode images and text into the same feature space. A text query or reference image can then be compared with features stored on graph nodes using cosine similarity.
In these projects, ReasoningGraph uses VLM-based visual features to encode objects and object relationships. HFLEX-EQA uses a VLM to select exploration modes and answer questions, with a low-level planner generating navigation goals. TrackGraph uses DINOv3 for mask propagation and stores CLIP galleries at each segment node which can be used for retrieval.
Reasoning About a Scene
Consider a robot asked to inspect an object in a room or identify objects blocking an exit. It needs to locate the relevant objects, understand their spatial and semantic relations, and determine whether its observations are sufficient for the task. A scene graph connects these questions to a persistent structured representation. Additionally, objects can be retrieved through language and considered in relation to their surrounding objects.
In HFLEX-EQA, this connection is used during exploration. Embodied question answering (EQA) requires a robot to explore a previously unseen environment, gather relevant information, and answer questions about the scene. For example, to determine a microwave's color, the robot may first need to find the room containing it and then find a good view of it. The VLM reasons over the hierarchical scene graph, visual memory, and a topological floorplan to decide what to inspect, which room to explore, or which room type to search for.
Beyond question answering, some tasks may depend on understanding the relationships between observed objects. For example, determining whether an object is blocking an exit requires understanding the spatial relation between the object and the exit. ReasoningGraph tackles this problem by adding VLM-based embeddings of object relationships to a hierarchical scene graph. The scene graph and its object relationships are used by a VLM to assess tasks that require relational reasoning.
Both uses of a scene graph depend on connecting observations to the right objects over time. TrackGraph addresses the efficient mapping and retrieval side of this problem. It tracks mask identities between image frames, fuses observations into persistent 3D segments, and stores CLIP features for text and image queries. This enables a representation that can be constructed and queried online with less expensive segmentation and feature extraction.
Hierarchical scene graphs provide structured persistent memory at the scale of objects, navigational graph, frontiers, and rooms. Relational reasoning uses information stored in the graph, active exploration finds information that is still missing for the task, and online mapping maintains the base structure of the scene graph. This connects the representation of the environment to the information needed for a robot task.
Vision-language models and open-vocabulary features connect language tasks to this memory. ReasoningGraph uses a VLM to encode and assess object relationships; HFLEX-EQA uses VLMs to interpret observations, select exploration strategies, and answer questions. CLIP features support retrieval: they support object and room search in ReasoningGraph, visual-memory selection and semantic exploration in HFLEX-EQA, and text or image retrieval in TrackGraph. Language therefore helps both retrieve relevant scene information and reason about tasks to enhance decision making.
The methods also retain visual information for later use. ReasoningGraph stores features from object-pair observations, HFLEX-EQA selects relevant room views and object inspection images, and TrackGraph maintains multi-view feature galleries for persistent segments. In each case, the graph associates this information with spatial entities, so reasoning and retrieval can draw on observations collected from different viewpoints and at different times.
Embodied question answering requires a robot to explore an unknown environment and collect the information needed to answer a question. HFLEX-EQA combines a VLM-based high-level planner with a scene graph, visual memory, and a topological floorplan. The floorplan provides room connectivity as a structural prior for deciding where to search.
How it works
Hierarchical scene graph representation. RGB-D observations incrementally build a hierarchical scene graph and an open-vocabulary occupancy map. The graph includes objects, rooms, a navigation graph, a metric-semantic mesh, and frontiers (boundary between observed and unexplored space) with open-vocabulary semantic information.
Select an exploration mode. The VLM receives the question, optional answer choices, the scene graph, per-room question-relevant images, action history, and floorplan. It selects one of three modes: go_to_objects, explore_room, or find_room.
Choose a target and navigate. Depending on the selected mode, a low-level semantic planner selects a viewpoint or frontier and generates a path. Room discovery uses floorplan connectivity and frontier semantics to search for relevant room types that have not yet been observed.
Reassess after new observations. The updated graph and visual memory return to the high-level planner. The VLM decides whether to continue exploring or answer from the available information.
The high-level planner selects a strategy; the low-level planner turns it into a navigation target.
Floorplan Guidance and the Hierarchical Planning
The floorplan contains room labels and adjacency, without metric geometry or the robot's position within it. It is a weak structural prior and can be incomplete or inaccurate. Observed room classes are matched to possible floorplan locations to suggest which room types could lead toward the target.
Planning is divided into two levels. The high-level VLM uses the question, scene graph, relevant images, exploration history, and floorplan to decide what information to seek next. It selects an exploration mode and the objects or room types to target. The low-level semantic planner turns that decision into a reachable viewpoint or frontier using the current map and navigation graph. After navigation, new observations update the graph and visual memory, allowing the VLM to revise its decision or answer the question when enough information has been gathered. The three modes below connect these high-level decisions to object inspection, room exploration, and room discovery.
explore_room
Search a known, partly explored room. Objects predicted as relevant to the question are encoded with CLIP and used to rank semantic frontiers within that room.
go_to_objects
Inspect an already observed object. Candidate viewpoints around its bounding box are scored by visible surface points and coverage within the camera image.
find_room
Search for an unobserved room class. Frontier scores combine semantic transition cues, progress through floorplan connectivity, and unexplored area.
Example. To answer “What is the microwave's color?”, the robot first needs to locate the microwave. If it has not observed one, the floorplan can guide a search for the lunch room. The robot then inspects the object and uses the resulting images to answer.
Evaluation: OpenEQA and ExploreEQA benchmarks, with quadruped deployments in real indoor environments.
Robot experiments
HFLEX-EQA deployed on a quadruped robot for two question-answering episodes in indoor environments. Additional public outreach video
ReasoningGraph adds open-vocabulary features and object relationships to a hierarchical 3D scene graph. These relationships are useful for tasks that depend on object relations, such as identifying items blocking an exit or checking the arrangement of pillows and blankets on a bed.
How it works
Build the hierarchy. RGB-D frames and robot poses are processed during exploration. The mapping system reconstructs a semantic mesh, clusters objects, computes a navigation graph, and detects rooms. An open-vocabulary detector and CLIP embeddings provide object embeddings.
Add object relationships. Features derived from a VLM visual encoder represent object relations. The graph stores these as object-level relationships and updates them as new observations are processed.
Interpret the task. A reasoning module combines two large language models (LLMs) and a VLM. Given a task, it identifies relevant objects and formulates subtasks to evaluate the general task. CLIP similarity allows for retrieval of the relevant objects.
Evaluate the subtasks. The VLM uses the graph's object-level relationships to assess the subtasks. This task reasoning is performed after the scene graph has been constructed during exploration.
Graph construction and task reasoning are separate stages. Object CLIP features support retrieval; object relationships allow task evaluation.
Object Features, Relationship Features, and Task Decisions
Object retrieval combines CLIP embeddings from a masked object crop, its bounding-box crop, and the detector's text label. Room-level feature clusters also allow search to focus on relevant parts of the graph.
Relationship features describe a pair of objects observed together. Their joint image crop is passed through the VLM visual encoder, with colored bounding boxes marking the objects of interest. The feature is attached to the edge between their graph nodes and updated from later observations.
At task time, a first LLM identifies relevant objects and interaction subtasks. CLIP similarity retrieves candidate instances. The VLM evaluates each subtask using its prompt and the stored relationship feature, with the box colors identifying the target pair. A second LLM interprets that assessment into a decision about whether the subtask should be carried out.
Example. For a task about blocked exits, the system retrieves relevant objects and assesses their relationship to the exits. Detecting the object is insufficient: its relations to the surrounding objects in the scene matter.
Evaluation: quadruped experiments on object search, exit obstruction, trash disposal, and bedroom preparation.
Robot experiments
Scene graph construction, object search, and object-relational reasoning on a quadruped robot.
TrackGraph
Online Open-Vocabulary 3D Scene Graphs via Image-Space Tracking
TrackGraph builds an online open-vocabulary scene graph while reducing repeated segmentation and vision-language feature extraction. Its core contribution is to track mask identities in the image stream before fusing them into persistent 3D segments.
How it works
Segment and compute features at sparse keyframes. At sparse keyframes, FastSAM produces masks and CLIP provides visual embeddings. Between keyframes, dense DINOv3 features propagate the masks and their identities through incoming frames.
Fuse tracks into 3D. Each temporally tracked 2D mask identity is a source track. Observations from a track are fused into a truncated signed distance field (TSDF) and mesh, then used to build class-agnostic 3D segments within the hierarchical graph.
Associate interruptions and revisits. Geometric and visual association in 3D handle interrupted tracks and observations from revisits.
Retrieve with text or images. CLIP features are averaged per source track and kept as separate views in a segment's feature gallery. A query is embedded into the same CLIP space, allowing object retrieval based on textual or visual queries.
Short-term image-space tracking feeds persistent 3D segments and multi-view feature galleries.
Maintaining Identity Between Frames and Across Revisits
Image-space tracking is central to TrackGraph: dense DINOv3 features propagate segmentation masks and their identities between sparse keyframes, maintaining temporal identity before observations are fused into 3D. At each new keyframe, detected masks are compared with propagated tracks using mask overlap and DINOv3 feature similarity. Matches continue an existing track, unmatched detections create tracks, and tracks that cannot be continued are retired.
During TSDF fusion, voxels retain several source-track hypotheses. Keyframe masks receive full confidence, while propagated masks have lower confidence that decreases with time since correction. This reduces the influence of uncertain propagated boundaries on the 3D representation.
On revisiting an area, a KD-Tree finds nearby historical segments. Geometric overlap and visual similarity determine whether a current segment is merged with an earlier one. Its tracks and CLIP features are transferred to the persistent node, connecting newly observed geometry to the existing map.
Example. After exploring an environment, a text query such as “printer” or a reference image retrieves candidate segments. Their locations in the graph support navigation to the selected object.
Evaluation: segmentation and retrieval on Replica, ScanNet++, and HM3D; onboard quadruped mapping and object search; additional mapping with recorded drone RGB-D data.
Robot experiments
Online graph construction, retrieval with text and image queries, and navigation to retrieved objects.
Conclusions
Scene graphs give language-based reasoning a spatial representation and a memory of the environment. VLMs can use them to reason about object relationships or guide active exploration, while open-vocabulary features allow relevant objects to be retrieved. The three papers study how to build, query, and extend this representation during robot tasks.
Main takeaways:
Structured scene representations. Hierarchical graphs connect reconstructed geometry to objects, places, and rooms, making those entities available for search, reasoning, and navigation.
Object-level relational reasoning. ReasoningGraph stores open-vocabulary object features and VLM-based relationship features, then uses them to retrieve objects and evaluate interaction subtasks.
Exploration guided by the task and floorplan. HFLEX-EQA combines VLM-based decisions with semantic frontiers, object viewpoints, and room connectivity to gather information for embodied question answering.
Online representation and object identity. TrackGraph separates sparse segmentation and CLIP extraction from image-space mask propagation, maintaining persistent 3D segments that can be queried through text and images.
Citations
01 HFLEX-EQABibTeX +
@article{puigjaner2026hflexeqa,
title={Hierarchical Floorplan-Guided Vision-Language Exploration for Embodied Question Answering},
author={Gassol Puigjaner, Albert and Alexis, Kostas},
journal={arXiv preprint arXiv:2609.26360},
year={2026},
url={https://arxiv.org/abs/2609.26360}
}
02 ReasoningGraphBibTeX +
@inproceedings{puigjaner2026reasoninggraph,
title={Relationship-Aware Hierarchical 3D Scene Graph},
author={Gassol Puigjaner, Albert and Zacharia, Angelos and Alexis, Kostas},
booktitle={IEEE International Conference on Robotics and Automation (ICRA)},
year={2026},
pages={1601-1608},
doi={10.1109/ICRA57385.2026.11696439}
}
03 TrackGraphBibTeX +
@article{hellesylt2026trackgraph,
title={{TRACKGRAPH}: Online Open-Vocabulary 3D Scene Graphs via Image-Space Tracking},
author={Hellesylt, Peder Borge and Gassol Puigjaner, Albert and Alexis, Kostas and Stahl, Annette},
journal={arXiv preprint arXiv:2609.31005},
year={2026},
url={https://arxiv.org/abs/2609.31005}
}
