Where Frontier Models Break in 3D Engineering & Spatial CUA
Empirical Evaluation on ISO-128 Blueprints & Deterministic B-Rep Solids
An empirical audit of Claude 4.5, GPT-5.2, and Gemini 3 across 20 held-out ISO-128 engineering blueprints from the public spacetime-ml/cad-cua benchmark. Why frontier models suffer an 85.0%–90.0% failure rate in 3D mechanical CAD, and why deterministic ISO-10303-21 B-Rep rewards (verify_step.py) beat VLM judges.
85–90%
Frontier Model Failure Rate
Across Claude 4.5, GPT-5.2 & Gemini 3
0.371–0.463
Mean B-Rep Reward (R)
vs 1.000 Human CAD Ground Truth
5 Core
3D CAD Failure Modes
Topology, projection & 6-DoF CUA
100%
Deterministic B-Rep Reward
ISO-10303-21 .STEP via verify_step.py
Empirical Evaluation on ISO-128 Engineering Blueprints
Evaluated using verify_step.py on 20 held-out mechanical parts paired with 4-view Third-Angle orthographic + isometric blueprints and human ground-truth .STEP solids.
| Model / Agent | Pass@1 (20 Blueprints) | Failure Rate | Mean Reward (R) | BBox IoU | Topology Score | Primary Failure Mode |
|---|---|---|---|---|---|---|
| Human CAD Engineer (Ground Truth) | 100.0% (20/20) | 0.0% | 1.000 | 1.000 | 1.000 | Watertight ISO-10303-21 B-Rep manifold solid |
| Claude 4.5 (Computer Use / CAD Code) | 15.0% (3/20) | 85.0% (17/20) | 0.463 | 0.543 | 0.415 | Missing internal counterbores, fillet self-intersections |
| GPT-5.2 (Vision + CAD Code) | 10.0% (2/20) | 90.0% (18/20) | 0.391 | 0.476 | 0.335 | 3D axis swap, non-manifold shells, bore pitch drift |
| Gemini 3 (Multimodal Spatial + CAD) | 10.0% (2/20) | 90.0% (18/20) | 0.371 | 0.449 | 0.324 | Unclosed sketch loops, missing radial hole arrays |
Where Frontier Models Break Down: ISO-128 Blueprint vs. 3D CAD Ground Truth & B-Rep Diagnostics
Omitted Concentric Counterbores & Extrusion Depth Drift
All three frontier models extrude the outer rectangular block envelope but fail to subtract the concentric inner trunnion counterbores on the Y-Z plane, reducing cylindrical surface count from 42 to <15 and violating ISO 2768-mK bounding tolerances.
- ▸Target Topology: 91 ADVANCED_FACE (19 planar, 42 cylindrical) vs. AI Candidate: 30–38 faces
- ▸Principal Envelope Drift: +18.2% on X-axis, -22.0% on Y-axis (s_bbox drops to 0.52)
- ▸Closed-Shell Volume Error: +44.8% excess solid volume from un-cut internal trunnion bores
Human Mechanical CAD Engineer (Ground Truth)
Captured across 62 verified desktop CAD episodes (SolidWorks, PTC Creo, AutoCAD, FreeCAD, Autodesk Fusion, Autodesk Inventor) with 34,551 lossless PNG frames and 260,203 timestamped telemetry events. Produces watertight ISO-10303-21 B-Rep solids with exact topological genus and bore spectra.
Frontier Foundation Models (Claude 4.5, GPT-5.2, Gemini 3)
Despite strong 2D code and text benchmarks, frontier models collapse when required to synthesize watertight 3D B-Rep geometry from ISO-128 orthographic drawings or drive 6-DoF parametric CAD viewports. Mean composite rewards drop to 0.463 (Claude 4.5), 0.391 (GPT-5.2), and 0.371 (Gemini 3).
The 3D Spatial & B-Rep Capability Gap
Pass rate on the 20-task ISO-128 blueprint benchmark (data/benchmark.jsonl) verified deterministically via verify_step.py.
Deterministic Pass Criterion:
A candidate .STEP model passes if and only if it forms a watertight manifold solid (CLOSED_SHELL >= 1, OPEN_SHELL == 0) and achieves composite geometric reward R >= 0.85.
The 5 Core Failure Modes in 3D CAD & Spatial CUA
Forensic analysis of why frontier models break when translating 2D ISO-128 engineering blueprints into 3D parametric B-Rep solids.
Non-Manifold Shell & Open-Loop Topology Collapse
Failure to construct watertight ISO-10303-21 CLOSED_SHELL solids
When translating 2D engineering sketches into 3D extrusions and revolves, frontier models frequently leave microscopic gaps between sketch entities, self-intersecting spline profiles, or zero-thickness edges. The CAD kernel rejects the feature rebuild or exports degenerate OPEN_SHELL surfaces that fail Euler-Poincaré manifold invariants.
Orthographic Projection & 3D Axis Swap
Misinterpreting Third-Angle ISO-128 Front, Top, and Right views
Frontier multimodal models routinely conflate Third-Angle orthographic projection planes—extruding Front-view features along the Z-axis while mapping Right-view depth callouts onto the X-axis. Even when individual feature dimensions are read accurately via OCR, the resulting 3D solid fails principal-axis bounding box alignment.
Internal Counterbore & Hole Array Omission
Ignoring hidden-line callouts and bolt-circle pitch patterns
Models focus disproportionately on outer silhouette envelopes while omitting internal coaxial bores, stepped counterbores (⌴), countersinks (∨), and circular bolt-hole patterns specified in section views. Naive 2D rendered silhouettes look plausible, but B-Rep cylindrical surface spectrum verification immediately exposes the missing internal faces.
6-DoF Viewport Starvation & Sub-Pixel Snap Misses
Inability to orbit, pan, and select occluded B-Rep edges in GUI CUA
In native desktop CAD CUA (SolidWorks, PTC Creo, Fusion, Inventor), selecting an occluded back-face or tangent edge requires coordinated 6-DoF middle-mouse viewport orbiting and sub-pixel cursor hover verification before clicking. Frontier CUA policies click blindly from a static isometric camera, missing sketch coincidence snaps.
Parametric Rebuild Fragility
Destructive feature trees that shatter under dimension edits
Rather than anchoring sketches to canonical datum planes and geometric constraints (concentric, tangent, symmetric), models hardcode absolute Cartesian coordinates. When downstream fillets, chamfers, or shell thicknesses are applied—or when a single driving dimension is updated—the parametric feature tree collapses with rebuild errors.
Why Deterministic B-Rep Reward (verify_step.py) Beats VLM Judges
2D rendered screenshots and VLM-as-a-judge pipelines hallucinate high scores on non-manifold shells and missing internal bores. Exact ISO-10303-21 B-Rep parsing provides uncheatable RLVR signal.
Principal-Axis Bounding Box IoU
Extracts all CARTESIAN_POINT coordinates to verify sorted principal-axis dimensions (L, W, H), aspect ratios, and volumetric bounding-box intersection-over-union.
Closed-Shell Volume & Surface Conservation
Computes closed-shell volume and total boundary surface area conservation against human ground-truth .STEP deliverables, penalizing unhollowed bosses or missing cutouts.
B-Rep Topological Graph & Euler-Poincaré
Audits MANIFOLD_SOLID_BREP, CLOSED_SHELL, ADVANCED_FACE, EDGE_CURVE, and VERTEX_POINT counts plus Euler characteristic χ = V - E + F.
Analytical Surface & Bore Radius Spectrum
Matches analytical surface distributions (PLANE, CYLINDRICAL_SURFACE, CONICAL_SURFACE, TOROIDAL_SURFACE) and sorted cylinder/circle radius spectra.
@techreport{spacetimeml2026cadcua,
title={Where Frontier Models Break in 3D Engineering & Spatial CUA: Empirical Evaluation on ISO-128 Blueprints},
author={SpacetimeML Research Team},
institution={SpacetimeML},
year={2026},
url={https://huggingface.co/datasets/spacetime-ml/cad-cua}
}Scale 3D CAD CUA & Geometric RLVR Post-Training
Integrate spacetime-ml/cad-cua and deterministic B-Rep reward verification into your frontier model training clusters.