Methods
Everything below is also in the repository's research/ directory, where the hypotheses were committed before the main benchmark ran.
Representation
Metres, site frame: the origin is the front-left lot corner on the street; x runs along the street, y into the lot. Rooms are axis-aligned rectangles on storeys. Doors and windows are derived from geometry, never searched. The tree-based methods search guillotine slicing trees: rooms tile their containers exactly, and common cores, stairs and courtyards stack by construction. CP-SAT places free rectangles on a 0.5 m grid instead; LLM proposals are arbitrary rectangles. One checker scores them all.
Briefs
72 synthetic briefs from a seeded generator (brief set 1.1.0, hash 31a635ad375385d2): six typologies — duplex, triplex, fourplex, rowhouse, courtyard housing and a principal dwelling with an accessory unit — each at three difficulty bands set by the ratio of programme area to buildable volume.
Benchmark constraints
Seventeen rules. They are benchmark constraints, not building code, and no jurisdiction is implied.
| ID | Name | Rule |
|---|---|---|
| C01 | geometry valid | Every room rectangle is finite with strictly positive extents. |
| C02 | no overlap | No two rooms on the same storey overlap. |
| C03 | within buildable | Every room lies inside the lot minus its setbacks. |
| C04 | min room area | Each room meets the benchmark minimum area for its type. |
| C05 | min room dimension | Each room's shorter side meets the benchmark minimum clear dimension. |
| C06 | max room aspect | Each room's aspect ratio is within the benchmark maximum for its type. |
| C07 | min unit area | Each dwelling meets the brief's minimum floor area. |
| C08 | unit connectivity | Every room of a dwelling is reachable from that dwelling's entry. |
| C09 | street access | Every dwelling's entry is reachable from the street. |
| C10 | habitable daylight | Every habitable room has an exterior wall able to take a window. |
| C11 | required adjacency | Brief-required room adjacencies are present as door-capable contacts. |
| C12 | forbidden door | No door connects a brief-forbidden pair of room types. |
| C13 | open space | Open space net of parking meets the brief's target fraction of the lot. |
| C14 | courtyard provision | A required courtyard void is delivered at usable size. |
| C15 | programme complete | Every room the brief requires is present, once, on its required storey. |
| C16 | unit contiguity | A dwelling's rooms on a storey form a single contiguous block. |
| C17 | vertical support | Every upper-storey room sits over built floor area on the storey below (a stacking proxy, not a structural check). |
Objectives
Six proxies, kept as a vector. Scores are oriented higher-is-better in [0, 1]; raw values are stored so any score can be recomputed under another band.
| Objective | Raw quantity | Band | Direction |
|---|---|---|---|
| density | GFA / (buildable area x storeys) | 0.35 -> 0, 1.00 -> 1 | higher is better |
| daylight | geometric daylight availability per habitable room | as computed, 0-1 | higher is better |
| privacy | bedroom depth, street distance, party walls, overlook | as computed, 0-1 | higher is better |
| circulation | circulation share, walking distance, reach | as computed, 0-1 | higher is better |
| carbon | kgCO2e / m2 GFA, A1-A3, modelled elements only | 620 -> 0, 260 -> 1 | lower raw is better |
| cost | unitless cost index / m2 GFA (no currency) | 3.30 -> 0, 1.75 -> 1 | lower raw is better |
Methods compared
- HeuristicA — a deterministic planner: envelope sized for open space, an enumerated family of partis, a service band along each dwelling's access side, then coordinate descent on proportions.
- Random searchB — independent random genomes; the floor any optimiser has to clear.
- Simulated annealingC — simulated annealing on a weighted sum of the brief's priorities minus a constraint penalty, four restarts.
- NSGA-IID — NSGA-II with constrained domination: feasible beats infeasible, then Pareto rank and crowding.
- CP-SATE — OR-Tools CP-SAT: feasibility first, then one warm-started solve per priority regime, single-threaded, deterministic time.
- Random sketch + repairF′ and Heuristic-seeded NSGA-II — a proposal (random sketch, or the heuristic's first parti) repaired, then used to seed NSGA-II on the same budget. They are the controls for the LLM condition.
- LLM + repairF — Claude Opus 5 proposes a schematic plan in a strict JSON schema; the same repair and seeded NSGA-II follow. Not run in this release: not run - no credentials.
Protocol
72 briefs × 3 seeds; 1,500 evaluations per run through one metered evaluator; CP-SAT gets 6.4 s, the median single-process time NSGA-II needs for that budget. Hypervolume is estimated by Monte Carlo on one fixed sample cloud, so differences between methods are paired. Brief-level statistics: seed-averaged, bootstrapped 10,000 times, sign-flip permutation tests with Holm correction.
Amendments made before the main run
- Brief count 60 → 72, so tiers are exact multiples of the 18-brief typology × band cycle.
- Wall-clock caps made safety-only for evaluation-budgeted methods; CP-SAT's time set from a smoke-tier measurement.
- Open-ground depth measured per exterior wall segment rather than from each wall's midpoint.
- Density band ceiling moved to its natural bound (1.0) after it saturated on the smoke tier; the carbon band was not changed.
Reproduce
make setup # venv + editable install make test # pytest + property tests make benchmark # STANDARD tier, resumable from checkpoints make repair # repair ablation make analyze # statistics and sensitivity make figures # publication figures make site # export data, build this site