Figure 1. Segmented cell entities represented as nodes with task-relevant relationships describing the interaction between the part and end-effector.
Representing a Manufacturing Work Cell for Autonomous Factories
Manufacturing applications are executed within cells that bring together the robots, industrial parts being processed, fixtures that hold those parts, tools and consumables used by the process, sensors, and surrounding equipment needed to carry out the work. A factory can contain multiple such cells, each responsible for different applications or stages of production. For robots and other agents to operate autonomously within a cell, they need context about the environment in which they are acting: what is present, where it is, which part is being processed, what the agent is currently viewing, manipulating or processing, and what constraints the surrounding cell places on the operation. This context determines whether an action is valid in the current environment. For example, a motion may no longer be collision-free if a fixture has moved, or an operation may require a different tool or consumable for a different part.
This requirement is especially important in high-mix manufacturing, where the same cell may process many parts and variants using different fixtures, tools, and process parameters. The scene can change between production runs and during the processing of a single part. A sanding operation, for example, may progress through several abrasive grits. The robot must recognize when to change consumables, track the tool currently in use, account for any repositioning of the part, and understand how each operation changes the part’s state. The cell cannot be modeled once and treated as static.
Autonomy therefore requires a representation that remains aligned with the physical cell throughout its lifecycle. During commissioning, the system must understand how the cell is configured and identify missing information. During execution, it must track the active part, fixture, tool, consumable, and process stage. During monitoring and sustainment, it must determine whether the physical cell still matches the expected state. Existing automation systems often distribute this information across collision models, calibration records, tool states, scan poses, process regions, and application-specific data structures. When the cell changes, these separate representations must be updated consistently. Any mismatch can lead to invalid planning assumptions, failed processes, or unsafe actions.
This creates a fundamental requirement for autonomy: a coherent representation of the cell in which manufacturing processes take place. A foundation model for scene understanding provides this capability by maintaining a unified, continuously updated view of the cell that connects its entities, geometry, states, relationships, and process context. This representation must support reasoning across multiple scales, from the configuration of an entire factory, to the relationships within an individual cell, to the detailed geometry of a surface or tool interaction. It must move between these levels without losing the relationships that give each element meaning. The objective is not simply to create a more complete map, but to provide autonomous systems with a shared and adaptable understanding of the manufacturing environment: what is present, what is happening, and which actions are valid at any given moment.
A Hierarchical, Task-Aware Scene Representation
A representation with these requirements cannot be only a list of objects and their poses. What can be done with an entity often depends on how it relates to the rest of the manufacturing environment. A part may be mounted on a fixture, a robot may be using a particular tool, a tool may apply only to certain regions of a part, and a region may currently be undergoing a particular operation. These relationships help determine what actions are possible in the cell.
The required level of detail also changes with the task. When reasoning about the layout of a cell, a cabinet or table may need only enough geometry for collision checking. The part being processed may require much more accurate geometry. For the operation itself, that same part may need to be represented as individual surfaces or process-relevant regions. At the factory level, only the relevant state and relationships of individual cells may be needed, rather than the detailed geometry within each one. The representation must support these different levels without representing everything at maximum detail.
A scene graph provides a structure for doing this. Robots, parts, fixtures, tools, sensors, consumables, cells, and process-relevant regions can be represented as nodes. Nodes can contain information such as pose, geometry, material, identity, and process state. Edges represent relationships between these entities, such as a part being mounted-in a fixture, an operation that requires-tool, or a region that is being-processed-by an operation (see Figure 1). A hierarchical graph also allows these entities and relationships to be represented at different scales, from the factory and cell down to an individual part or surface region.
The scene-understanding model creates and maintains this scene graph from observations and available information about the manufacturing environment. It detects, segments, and classifies objects and process-relevant regions, estimates their attributes, constructs relationships between them, and updates those attributes and relationships as the environment changes. It also tracks which parts of the representation are incomplete or uncertain, allowing additional information to be acquired when needed.
What Makes the Scene Understanding Problem Different in Factories
In manufacturing, the information needed to understand a scene depends on the operation being performed. A part can be treated as a single object when reasoning about the layout of the cell, but as a collection of individual surfaces or regions when reasoning about where a process should be applied. The relevant representation therefore depends on how an object is used within the process. The relationships between entities can also depend on manufacturing knowledge that is not available from geometry alone. Determining whether a particular abrasive is applicable to a surface, for example, can depend on the part, its material, the abrasive or tool being used, the current stage of the process, and the desired manufacturing outcome. Some of this knowledge is specific to a factory’s processes, equipment, and operating experience, and may not exist in public datasets or generic object representations. Building a useful representation therefore requires incorporating knowledge learned or established through manufacturing operations, in addition to what can be obtained directly from perception.
The information needed to construct this representation can come from different sources. RGB, depth, laser, and point-cloud observations provide information about the physical environment; force and inspection sensors provide information about interactions and the process; and the robot provides its own joint, tool, and calibration state. These sources must contribute to a consistent representation of the cell. The representation also has to account for change. As an entity moves or its state changes, the attributes and relationships associated with it may change as well. Additional sensing or interaction can also refine what is known about an entity. The representation is therefore built incrementally: it can remain coarse where little information is required, become more detailed where needed, and update as either the environment or the system’s understanding of it changes.
Building and Maintaining the Scene Representation
The scene representation is initialized from the information available about the manufacturing environment and refined as new observations become available or a task requires additional detail. Different parts of the representation can therefore exist at different levels of detail and confidence. Figure 2 summarizes this process, from the multimodal observations used to construct the scene representation to the interfaces through which it is queried, evaluated, and incrementally updated.
Figure 2. Overview of how the scene representation is constructed and maintained. Multimodal observations are processed to build the representation, which captures entity attributes, relationships, and metadata such as fidelity, uncertainty, build state, and update policy. Interfaces expose this state to downstream systems while new observations and task requirements drive incremental updates.
Task-dependent hierarchy
How an entity is represented depends on what the system needs to reason about. At a broad level, the representation may describe the factory as a collection of manufacturing cells and the relationships between them. Within a cell, it can represent robots, parts, fixtures, tools, sensors, and other equipment. For a particular operation, an entity may be decomposed further: a part represented as a single object at the cell level may instead be divided into individual surfaces or process-relevant regions. The hierarchy can also reflect different views of the same environment. A structural view may organize entities for collision reasoning, while an interaction-oriented view may emphasize the relationships between a part, fixture, tool, and the regions involved in an operation.
Fidelity
Fidelity describes how much detail a task requires from an entity. Within a cell, for example, a tool cabinet may only need enough geometric detail for collision checking, while the surface of a part being processed may require a much more accurate reconstruction. Fidelity requirements therefore depend on how the entity is being used.
Build state and uncertainty
The build state describes whether the information required about an entity has been established sufficiently for a task. This depends on whether the required attributes and relationships are available, whether the representation has reached the required fidelity, and whether those properties are known with sufficient confidence. Uncertainty is tracked at the level of individual attributes and relationships. A part’s pose may be well established while its material or its relationship to another entity remains uncertain. If the required fidelity or confidence has not been reached, the missing information can be acquired selectively. During part onboarding, for example, the Autonomous Cell Commissioning workflow can acquire additional views when the available observations do not provide the required coverage or geometric detail.
Updates
How information is updated depends on how the corresponding entity can change. A node’s kinematic category can capture this distinction and determine its expected update mechanism and frequency. Robot and end-effector state can be updated continuously from the robot, while a fixture may only need to be updated when it is moved or re-observed. A part or one of its process-relevant regions may instead change at particular stages of an operation. When an entity changes, the attributes and relationships that depend on it are reconsidered so that the representation remains consistent with the current state of the cell.
Querying and extending the representation
Downstream systems interact with the scene through queries. If the required information is already available at the necessary fidelity and confidence, it can be retrieved directly. If not, the query exposes what is missing and can lead to the required sensing, interaction, or refinement. The requirements of a task therefore determine what must be known, while build state captures whether that information has been established sufficiently to use.
Toward Factory SuperIntelligence
At the factory level, autonomy requires an understanding that extends beyond the state of any individual cell. Different cells perform different operations, expose different capabilities and resources, and depend on processes occurring elsewhere in the factory. A hierarchical scene representation can connect these levels of understanding: factory-level reasoning can operate over the state, capabilities, and relationships of entire cells, while resolving into the equipment, parts, process state, or interactions within a particular cell when greater detail is required.
The long-term vision for Factory SuperIntelligence is not only a collection of autonomous cells, but a factory whose physical state, process state, capabilities, resources, and constraints are continuously represented and available for reasoning across the system. This shared understanding can provide the foundation for commissioning, sensing, planning, process intelligence, and production coordination to operate as connected capabilities rather than isolated forms of autonomy. As production requirements change, the factory can reason about what capabilities are available, where work can be performed, how changes propagate across cells, and how production should adapt. The broader vision is a factory that can understand its own evolving state and use that understanding to coordinate increasingly autonomous production at scale.
Case Study: Scene Understanding in a Sanding Cell
We illustrate the scene representation on a real manufacturing cell. Figure 3 shows the cell context, while Figure 4 shows the same environment after individual objects have been segmented and associated with entities in the scene graph.
Figure 3. Manufacturing cell used in the case study. The cell is visualized using a 3D Gaussian splat only for visual representation.
Figure 4. Segmented entities in the manufacturing cell. Individual objects and regions are segmented from the scene and associated with entities in the scene representation.
Once the entities in the cell are represented, the graph can capture relationships that are relevant to a particular task. Figure 5 shows an example for collision reasoning, where objects such as the table and surrounding equipment are marked as collision-relevant-to the robot. Previously, building this collision representation was largely a manual process: engineers measure objects in the cell, estimate their poses, generate or obtain CAD models when needed, and encode this information in the scene configuration files used by the robot planners. For a typical cell, this requires approximately 3-6 hours of engineering effort and can be iterative as the collision model is tested and tuned. Because the resulting configuration reflects the cell at the time it was created, changes to the physical environment can require the same information to be measured, updated, and tuned again. Capturing collision relevance directly in the scene representation provides a way for downstream planning systems to reason from the current state of the cell instead of relying on a separately maintained collision configuration.
Figure 5. Task-relevant relationships for collision reasoning. Objects such as the table and vacuum equipment are represented as collision-relevant-to the robot, identifying which parts of the scene need to be considered for collision checking.
The scene graph can also represent information that is still incomplete. Figure 6 shows the robot, its end effector, and the sandpaper-changing station. The station itself may already be known: its identity, pose, geometry, and relationship to the robot can all be represented. However, the trajectory required for the end effector to interact with the station may not yet have been established. In that case, the station can contain a property such as tool_change_trajectory: missing, which in turn leaves its build_state incomplete.
Figure 6. Incomplete build state for the sandpaper-changing station. The station is represented in the scene graph, but the required tool-change trajectory has not yet been populated, leaving its build_state incomplete and indicating that additional commissioning is required.
This distinction is important because an incomplete build state does not mean that the object itself is unknown. It means that some information required for the intended task is still missing or no longer valid. In this example, the incomplete build state can trigger the corresponding Cell Commissioning Agent workflow to determine and validate the missing trajectory. Once that information is populated, the scene representation is updated accordingly. Together, these examples show how a visual model of the cell can become an operational resource for autonomy. The scene graph connects entities, task-relevant relationships, and missing information in a shared representation that supports perception, collision checking, and commissioning. As the cell changes, the system can update its understanding, identify missing information, and initiate the actions needed to maintain a valid representation. A foundation model for manufacturing scene understanding can therefore serve as more than a perception system: it can become the evolving model through which an autonomous factory understands its physical environment, identifies what it still needs to learn, and determines what actions are possible next.
