Replication report · UK · experimental · deltas only

DEFINE-UK: running and gating an unlicensed ecological SFC model

The authors’ unmodified R code runs at a pinned commit. The baseline macro block matches Table 4 within every gate, two by only 0.01pp (the 2025 growth gate was set at that miss); emissions diverge and no numeric scenario results are published, so the adapter serves deltas only.

Climate policy meets the UK financial system

DEFINE-UK is the ecological stock-flow consistent (E-SFC) model of the UK built by George and Dafermos (SOAS), building on the DEFINE model of Dafermos, Nikolaidi and Galanis (define-model.org/define-uk).

What the model coversScenarios · versions · the replication standard

It simulates the UK macrofinancial system and its environmental impacts under climate-policy scenarios — a fossil-fuel ban, power-sector subsidies, housing regulation, green public investment, and a mixed package. No other model in the suite answers the question it is here for: who bears a climate-policy scenario. Versions 1.0 and 1.1 were released in April 2026. The specification is the DEFINE-UK Model Manual, Version 1.1 and the accompanying paper “Evaluating climate policy mixes in the UK: an E-SFC approach”.

The suite's standard is that a model earns its place by replication: run the published code or reimplement the published equations, then compare against the published results at recorded tolerances. This report applies that standard, and records where the publications themselves impose a ceiling.

Every gate, its verdict, and the test that holds it

Each gate below is enforced by a test in the adapter repository against the cached pinned run (upstream commit 846081a), so drift fails loudly rather than silently. The supporting narrative for each — tolerances, component checks, arithmetic, search history — is one disclosure away.

The macro block matches Table 4 within every gate (the 2025 growth gate was set at that miss); 2025 growth (−0.30pp, gate ±0.31) and 2040 unemployment (−0.19pp, gate ±0.2) clear by only 0.01pp.

DEFINE-UK replication gates: verdict, date, and enforcing test
GateVerdictDateEnforced by
Manual Table 4, macro block (growth, unemployment, population 16+, labour force), S1 baseline PASS — but 2025 growth misses by −0.30pp (gate ±0.31) and 2040 unemployment by −0.19pp (gate ±0.2): both clear by only 0.01pp, which is not the same as matching 4 Aug 2026 tests/test_replication_baseline.py
Manual Table 4, total emissions, S1 baseline DIVERGENCE — pinned at the observed ratios 4 Aug 2026 tests/test_replication_baseline.py
Scenario design — the policy switches each of the 8 checked scenarios toggles PINNED EXACTLY 5 Aug 2026 tests/test_scenario_design.py
FMM 2023 open-vintage anchors — green public investment GDP peak; baseline 2030 emissions PINNED, not a replication — peak +0.92% inside our 0.7–1.3% band; the “1% of GDP” recorded from FMM 2023 is the size of the investment; no published GDP response is confirmed. Baseline 2030 emissions 342.8 vs “just under 350” 5 Aug 2026 tests/test_scenario_design.py
Published numeric scenario results for v1.1 CEILING REACHED — none are published 5 Aug 2026 validation/published_targets.json (search record)
Baseline vs external observations (ONS, DESNZ, OBR) COMPUTED — gaps are material 5 Aug 2026 tests/test_validation.py
Green public investment cumulative multiplier COMPUTED — 2.25 nominal per £ of power-sector spending, from the 2027 policy start to 2040; ours, not upstream’s; back-loaded, so below published short-horizon multipliers at their horizons; includes a parameter mismatch 17 Nov 2026 tests/test_validation.py
Clean-room milestone 1 — §2.2 matrices on the §5 initial values PASS 4 Aug 2026 tests/test_accounting.py
Clean-room milestones 2 to 4 — macro and production, sectors, ecosystem, against the upstream oracle NOT PASSED — every printed equation is registered and the whole model solves. GDP is within 0.41% with the other blocks held at the oracle, outside the 0.2% printing tolerance, and up to 44% below it by 2040 solved whole (the reimplementation section) 17 Nov 2026 tests/test_full_model.py, tests/test_oracle.py
Macro block: the tolerances, and what each component hit

Tolerances: population and labour force ±0.01m; growth ±0.31pp; unemployment ±0.2pp. Against Table 4's macro block at 2025/2030/2040 on the S1 baseline, population and labour force are exact to 0.01m; growth matches to 0.06pp at 2030 and 2040 and misses by −0.30pp at 2025; unemployment is within 0.2pp, though 2040 misses by −0.19pp (4.53% vs 4.72%); and the 2025–40 mean and standard deviation of growth and unemployment match the table. The run is calendar-anchored on the population path (1987Q1–2040Q4, 2025 = t153).

The growth gate is not independent of the deviation it admits. ±0.31pp was set after the run, and the only year that uses any of it misses by −0.30pp — the gate clears by 0.01pp. It is doing no work as a test of agreement in 2025; what it does is stop that year drifting further, and the 2030/2040 matches to 0.06pp are the part of this gate that is genuinely informative. The 2025 gap is not an annualisation convention: no single growth convention reproduces all three years, and 2025 sits in the post-2022 start-up transient (validation/reference_outputs.json, re-checked 2026-11-16). It is an unresolved miss, not a demonstrated agreement.

Emissions: 393 vs 407 in 2025, 343 vs 382 in 2030, 249 vs 324 by 2040

The pinned code runs below the published Table 4 emissions, and the gap widens with the horizon: 393 vs 407 MtCO2e/yr in 2025 (−3.5%), 343 vs 382 in 2030 (−10.2%), 249 vs 324 by 2040 (−23.3%). The percentages come from the pinned run/table ratios themselves (0.965, 0.898, 0.767 in validation/reference_outputs.json), not from the rounded levels printed here.

Two checks rule out an extraction error on our side: the components EMIS_NELEC + EMIS_ELEC sum exactly to EMIS, and the baseline is identical across every scenario folder. A re-check of all 15 Table 4 rows (2026-11-16) finds the power and energy block diverging while the exogenous inputs and macro block match: consistent with, but not proof of, the table coming from a different model vintage than commit 846081a. The divergence has been queried with the upstream authors, and the tests gate at the observed ratios, so any further drift fails loudly.

Scenario design: the exact switch sets, the two anchors, and the search that closed the target

tests/test_scenario_design.py pins that each of the 8 checked scenarios toggles exactly the policy switches its published description claims: FF_BAN plus capacity/investment ban timings for the ban scenarios; POWER_SUB alone for the subsidy; GVT_INVEST + GREEN_BONDS + GREEN_POWER exactly for green public investment; and the published 40% housing subsidy rate, HOUSING_SUB_RATE = 0.4.

Two coarse numeric anchors are taken from the open FMM 2023 conference version of the paper, held at wide tolerances because it is a pre-1.0 vintage. The cached run’s green public investment GDP peak is +0.92%, band 0.7–1.3% (includes the αPP mismatch); but the “1% of GDP” recorded from FMM 2023 is the size of its investment, no published GDP response has been confirmed, and its shock is larger and starts earlier. The current-policies baseline emits “just under 350 MtCO2e” in 2030 — the cached run gives 342.8, band 330–352. Both are pinned, not replicated.

Read those two as regression pins, not as agreement tests. Both bands were chosen by us to fit a verbal quantity, not published as tolerances, and neither can discriminate much: ±30% around “around +1%”, and an emissions band whose upper edge (352) is above the 350 the phrase “just under” is supposed to bound, so a run at 351 would pass a gate named for a claim it violates. What they establish is that the cached run has not moved since the anchors were recorded.

Nothing finer exists. An exhaustive search, catalogued in validation/published_targets.json, established that no machine-readable numeric scenario results are published for v1.1: the manual's results stop at the baseline (Table 4) plus scenario design parameters, and the George/Dafermos SSRN paper (abstract 6541398) is paywalled with no open mirror. Its scenario set — carbon pricing and subsidies — predates v1.1's regulation policies in any case.

Calibration: the baseline against ONS and DESNZ outturns

Since 2026-08-05 this comparison is an artifact, not a paragraph. define_uk.validation.baseline_calibration() recomputes every row from the cached pinned run against vintage-exact external observations; the result is committed as validation/baseline_vs_external.csv; and tests/test_validation.py fails on any drift between the committed table and a recomputation.

The headline rows: 2025 real GDP growth of 4.66% (annual-mean convention; Table 4 prints 4.96%) against an ONS outturn of 1.31% y/y; 2025 unemployment of 4.39% against an ONS annual mean of 4.88%; population 16+ 55.66m vs ≈56m, consistent; labour force 35.97m, slightly high; and emissions starting above the DESNZ actuals. The full table is on the validation page.

Multiplier: our arithmetic, and how it sits against published ones
  • 2.25 is our own computation: nominal cumulative ΔGDP over cumulative ΔSPEND_GVTPS, the government power-sector spending the policy switch sets, from the 2027Q1 policy start to 2040 (209.9 less 4.8 of pre-policy change, over 91.2).
  • It is back-loaded. The profile by horizon, the full-run 2.30 and the 1.78 ratio per pound of all non-wage spending are on the validation page, their one home.
  • Published comparators are short-horizon: Batini et al. (2021, IMF Working Paper WP/21/87) estimate about 1.1–1.5 for renewable-energy investment over a few years, and the OBR’s capital-spending impact multiplier is about 1.0.
  • At those horizons this model’s ratio sits below them; it exceeds them only cumulated to 2040, so it is not like-for-like.
  • An earlier IMF paper the repositories hold, Batini et al. (2014), measures a different quantity (first-year multipliers of 0–1 for advanced economies).
  • It includes the parameter mismatch: 2.25 if that effect stops in 2027, as low as about 2.08 if it persists.
  • The caveats travel with the number wherever a delta is served (GPI_MULTIPLIER_CAVEATS, held by a test).
  • The upstream repository’s own tables/Multiplier_Summary.csv is documented unusable and never quoted: its multiplier columns are identical across all eight scenarios.

Unlicensed upstream, pinned commit, and a run that proves nothing on its own

The upstream repository is public but carries no licence, so its code is never vendored or hosted.

The 2026-08-01 run record

R 4.3.0 on macOS, upstream unmodified, an rstudioapi shim the only accommodation. The 151 files are the figures and tables for all four scenario blocks plus tables/Multiplier_Summary.csv.

What the run establishesExecution is not validation

The adapter (github.com/PolicyEngine/define-uk-model) fetches it at pinned commit 846081a at runtime and executes it as published, in R, unmodified. That also fixes the run surface: DEFINE-UK is local-only, never hosted, and the eventual hosting path is the clean-room reimplementation in the reimplementation section.

Execution ≠ validation. On 2026-08-01 the full notebook rendered end to end through define_uk.runner.run, producing 151 output files. That demonstrated only that the pinned code executes, not that its outputs mean anything. The adapter's VALIDATION.md holds the gate: nothing built on the repository may present model output as meaningful except by naming the target it rests on — the same bar applied to BoE SVAR and US HANK. The gates section records what that gate has since established.

That file's own status line was wrong for a stretch, and the correction belongs in this record because this report cites it as the record. It read “NOT VALIDATED. No published DEFINE-UK result has been reproduced” for some time after target 1a passed, which put it in contradiction with every page that cited it. It now reads PARTIALLY VALIDATED and breaks the claim down target by target: 1a pass, 1b divergence pinned, targets 2 and 3 closed at the achievable ceiling rather than passed, reimplementation milestone 1 pass and milestone 2 outstanding. Nothing there licenses presenting scenario output as a validated result.

Deltas, never levels — closed at the achievable ceiling

Baseline levels sit far from official outturns: 2025 growth of 4.66% against an ONS 1.31%, unemployment below the outturn, emissions above the DESNZ actuals they should start from.

What the adapter servesDeltas only · the gate stack · the upgrade path

The manual itself says the baseline “should not be seen as a prediction”; we take that literally. Near-term levels are not competitive with the OBR emulator or the BoE SVAR and are never presented as forecasts.

So the adapter serves annualised scenario-minus-baseline delta paths only — deltas, never levels — framed with mandatory caveats including the demand-led closure, via pe-macro define-scenario (local-only, experimental). The gate stack behind them is the pinned upstream run as oracle, the exact scenario-design pins, and the two FMM 2023 anchors. The upgrade path is equally explicit: the authors publish scenario tables, or the SSRN paper becomes accessible, and the published-figure gate reopens.

Reimplementing the published equations, milestone by milestone

Because the upstream code cannot be hosted, the hosting path is a from-scratch Python implementation of the published equations — the same play as the OBR emulator.

The clean-room protocolManual as specification · upstream as output oracle only

The protocol is clean-room: the specification sources are the manual, the paper, and Dafermos, Nikolaidi & Galanis (2017); the upstream repository is an output oracle only, its source never read to write equations; and every equation carries a manual_ref naming the manual section and equation number it implements.

Three of the pinned manual defects, worked through

Table 6 LENDM_ROW omits the DIVN_ROW term of Eq. (383), so the MFI/RoW transaction columns miss LEND by ∓DIVN_ROW (= 5.44); the manual itself tabulates overall LENDM = 5.44 where it “should equal 0”. Pinned exactly in tests/test_accounting.py.

Table 6's capital block is deflated at the wrong price. Eq. (27), GCF_R = GCF/P_P, gives 111.88 against a tabulated 112.30 — 0.37%, roughly ten times the rounding noise — and Eq. (25) only reproduces the tabulated GDP_R using 112.30. This is one finding, not five: GCF, GCF_NFC, GCF_HH, GCF_GVT, K_P and K_NFC all show nominal/real ratios of 1.0309–1.0312 where the equations prescribe P_P = 1.035, and the same 0.4% reappears in §3.3.2's Eqs. (104), (105), (132) and (133). One systematic property of Table 6's capital rows — most likely a national-accounts investment deflator where the equations want the production deflator — counted once. Pinned in the tests rather than absorbed into a tolerance.

Eq. (31) uses an intercept α₀λ that Table 5 never tabulates — the α₀ block ends at α₀WS. It is defaulted to 0.0, recorded in macro.MANUAL_GAPS, and pinned by a test that fails if a value is later published.

The milestonesMilestone 1 passed · milestones 2 to 4 not passed

Milestone 1: PASS (2026-08-04). The §2.2 transactions and balance-sheet matrices and the full §5 parameter and initial-value tables (Tables 5–6) are transcribed, and every Table 1/2 row and column identity holds on the §5 initial values within the manual's own 4-significant-figure printing precision. tests/test_accounting.py collects 62 tests, of which 44 are the Table 1/2 row and column identities themselves (20 transactions rows, 6 transactions columns, 7 balance-sheet rows, 11 balance-sheet columns); the remainder are the residual-instrument and net-worth definitions and the two that pin the one manual inconsistency below.

Milestones 2 to 4: NOT PASSED (status 2026-11-17). Every printed equation of the manual (§3.1–§3.4.7) is registered and the whole model solves quarter by quarter. Milestone 2 (macro and production, §3.2–3.3) is gated on the baseline GDP path: with the other blocks held at the upstream run it is within 0.41%, outside the 0.2% printing tolerance. Solved as a whole model (milestones 3 and 4, sectors and ecosystem), baseline GDP falls up to 44% below the upstream run by 2040 and the green-investment multiplier is 1.52 against the upstream’s full-run 2.30 (tests/test_oracle.py, tests/test_full_model.py).

Reimplementation is already earning its keep as a reading of the manual that the R source cannot give: thirteen manual defects are pinned in tests as machine-readable gap records rather than papered over or absorbed into a tolerance, with the wider set written up in the reimplementation notes. An audit of those notes also re-characterised four findings that were reading as the manual contradicting itself — Eqs. (51), (59), (94) and (111). Table 5 describes the parameters concerned as historical means or means of past implied values, and so never claimed to reproduce the manual's own initial period; those are first-period jumps a run started from Table 6 will take in its first quarter, not defects. The same audit withdrew one claim outright, on Eq. (51), where the corroborating inference had ignored a lag.

The model design belongs to its authors; the implementation is ours, AGPL-3.0, and independent — not the authors' code, and not endorsed by them.

Cite this work

Ahmadi, Vahid (2026). “DEFINE-UK: running and gating an unlicensed ecological SFC model.” PolicyEngine Macro replication report. https://policyengine-macro.vercel.app/reports/define-uk-replication

@techreport{policyengine2026defineuk,
  title       = {DEFINE-UK: running and gating an unlicensed ecological SFC model},
  author      = {Ahmadi, Vahid},
  year        = {2026},
  institution = {PolicyEngine},
  url         = {https://policyengine-macro.vercel.app/reports/define-uk-replication},
  note        = {PolicyEngine Macro replication report}
}

No DOI is assigned. For an exact version stamp, cite the git commit of the artifact you used from github.com/PolicyEngine/define-uk-model. Full validation record: VALIDATION.md and REIMPLEMENTATION.md in that repository, mirrored on the validation page.