A Unity Digital Twin-based Synthetic Data Pipeline for Manufacturing AI
Digital twins for AI. AI for digital twins.
The technologies a factory floor actually needs.
We build digital twins of real manufacturing lines in Unity, and turn them into environments where AI can be trained and tested. This page is about one of them: an environment for industrial human–robot collaboration, and the dataset it produced.
The problem
AI learns from data — and some fields overflow with it. Autonomous driving has millions of kilometres of logs; vision has the web; language has the entire internet. The data we actually need — a worker approaching a robot, partly occluded, crossing a safety zone — barely exists in public form.
Safety
The moments a person is at risk next to a robot cannot be staged repeatedly just to collect data.
Cost
Stopping a real line, wiring sensors and re-shooting under varied conditions is expensive in time and money.
Labeling
Aligning 3D position, distance, joints and pixel-level regions across sensors at one timestamp is near-impossible by hand.
Rarity
Long-tail events like a near-collision almost never happen — and on a well-run floor, they should happen even less.
And what we really wanted to learn was not whether a person is there. It was proximity itself — how far from the robot, in which direction, in what pose.
The approach
If you can't collect it, generate it — from what you can measure. We anchor a Unity digital twin to real measurements, because Unity lets four things share one runtime:
Physics simulation
Physically plausible interaction between robots, people and objects.
HDRP rendering
Lighting and materials matched to reality to shrink the domain gap.
Sensor simulation
RGB-D, LiDAR and 360° cameras reproduced virtually.
Low-cost labels
Ground truth read from the twin, with minimal human annotation.
Apart, these are four features. Sharing one simulation clock, they become data an AI can train on.
Industrial HRC-Bench
Before generating anything, we built a trustworthy reference dataset — real HRC scenarios designed with manufacturing-domain experts and captured at the KTL Robot Testing & Certification Center.
LiDAR, RGB-D, motion capture, robot state and 360° imagery, captured in sync. What mattered more than volume was structure — every observation and label sharing one exact timeline, so later comparison and generation stay meaningful.
Where it was captured — a certification testbed, not a lab mock-up
Korea Testing Laboratory (KTL) is a government-affiliated testing and certification institute. Its Robot Testing & Certification Center opened in 2024 inside the Gangnam Robot Plus Test Field in Seoul, and performs product certification for collaborative robots and workplace risk assessment and demonstration for collaborative robot systems, against international standards. Collaborative robots are built to work in the same space as people, without the fences that keep industrial robots apart — which is what makes safety certification of both the product and the system a precondition rather than a formality.
That is why the reference data was recorded there rather than in our own lab. What this dataset is about is proximity safety between a person and a robot — certifying exactly that is what the facility exists to do. Recording in a certification testbed is what lets the reference set carry weight beyond our own claim.
The dataset is being prepared for public release
Industrial HRC-Bench will be published through an external repository. If you'd like early access or to collaborate, see the note at the end of this page.
Inside the pipeline
The whole flow is three words: Ground, Calibrate, Generate — fix the twin to real measurements, close the residual gaps, then generate data from that anchor.
Environment — the real line, as-is
The KTL testbed is measured and reconstructed at 1:1 scale, with HDRP matching materials and lighting and a photoreal 3D Gaussian Splatting background, so the domain gap starts small.
Robot — physically plausible, not merely plausible-looking
ArticulationBody reproduces the robot's real joint structure and degrees of freedom, and recorded joint trajectories are replayed frame by frame under inertia and contact.
Labels — from the twin, not from annotators
2D/3D bounding boxes, semantic and instance segmentation, joint keypoints and depth ground truth all come from the same simulation state, cutting both labeling cost and annotation error.
The real-to-sim gap
We built the twin. It still didn't match. Two residual gaps hit proximity and ground truth directly — we found each, then closed it.
Problem — real lenses don't read the datasheet
Same sensor model, same field of view — and the same object still landed on different pixels. In HRC a few pixels of drift is enough to shift the correspondence between a pixel label and the real observation.
Solution — don't model the lens; measure it
A 3DGS view gives paired real/virtual captures; their per-pixel difference becomes a distortion map, applied in Unity as a camera-distortion shader — so the correction happens as the frame is rendered, and corrected observation and ground truth come out of the same runtime.
Problem — occlusion turns human tracking into noise
When the body is occluded, hidden joints get hallucinated — feet slide, poses jitter, and distance and safety-zone labels go wrong with them.
Solution — Foot IK + humanoid IK
Foot IK re-grounds the pose on the measured floor geometry; humanoid IK constrains hidden joints to valid ranges. Both target the non-physical motion that would otherwise corrupt proximity and safety-zone labels.
Scenarios & demo
Two scenarios share the same HRC setting but stress different things.
Palletizing — RB20
A worker repeatedly enters and exits a wide working radius. Full-body pose, distance over large paths, and safety-zone judgement are what matter here.
Surface inspection — UR5e
Person and robot collaborate closely in a tight space. Fine hand and upper-body pose, occlusion by the robot, and short-range proximity are what matter here.
Beyond the benchmark
Seen as a fixed dataset, this would stop at two scenarios. We built it instead as a data generation engine: from a calibrated baseline, a domain-randomization module built on the Unity Perception package varies lighting, materials and camera placement within controlled ranges — realistic variation, not noise.
HRC is one application domain that proves the principle. As robots and humanoids come to share the floor, a simulation and data layer — where AI can learn and be tested before it acts in reality — is what an AI Factory will increasingly need.
For researchers & developers
Working on HRC, proximity perception or sim-to-real? Industrial HRC-Bench is being prepared for public release, and we're open to collaboration. Reach out for early access or to talk — [email protected].
HRC-Bench project team
Collaboration
Enquiries — [email protected] · Jaemin Lee, Team Leader, +82-31-500-2548
Affiliations are as of the presentation date.
Talk
Unity has published the session recording.
Watch the session — A Unity digital-twin synthetic data pipeline for manufacturing AI
Unite Seoul 2026 · Unity Korea
Related video material
Collected separately by Jae-Hoon Hwang, project team.
Research collaboration and enquiries — [email protected]
© 2026 Hong-In Won · Simulator demos require separate access.