Replication report · UK · experimental · deltas only
DEFINE-UK: running and gating an unlicensed ecological SFC model
The authors’ unmodified R code runs at a pinned commit. The baseline macro block matches Table 4 within every gate, two by only 0.01pp (the 2025 growth gate was set at that miss); emissions diverge and no numeric scenario results are published, so the adapter serves deltas only.
Climate policy meets the UK financial system
DEFINE-UK is the ecological stock-flow consistent (E-SFC) model of the UK built by George and Dafermos (SOAS), building on the DEFINE model of Dafermos, Nikolaidi and Galanis (define-model.org/define-uk).
What the model coversScenarios · versions · the replication standard
It simulates the UK macrofinancial system and its environmental impacts under climate-policy scenarios — a fossil-fuel ban, power-sector subsidies, housing regulation, green public investment, and a mixed package. No other model in the suite answers the question it is here for: who bears a climate-policy scenario. Versions 1.0 and 1.1 were released in April 2026. The specification is the DEFINE-UK Model Manual, Version 1.1 and the accompanying paper “Evaluating climate policy mixes in the UK: an E-SFC approach”.
The suite's standard is that a model earns its place by replication: run the published code or reimplement the published equations, then compare against the published results at recorded tolerances. This report applies that standard, and records where the publications themselves impose a ceiling.
Every gate, its verdict, and the test that holds it
Each gate below is enforced by a test in the adapter repository
against the cached pinned run (upstream commit 846081a),
so drift fails loudly rather than silently. The supporting narrative
for each — tolerances, component checks, arithmetic, search history —
is one disclosure away.
The macro block matches Table 4 within every gate (the 2025 growth gate was set at that miss); 2025 growth (−0.30pp, gate ±0.31) and 2040 unemployment (−0.19pp, gate ±0.2) clear by only 0.01pp.
| Gate | Verdict | Date | Enforced by |
|---|---|---|---|
| Manual Table 4, macro block (growth, unemployment, population 16+, labour force), S1 baseline | PASS — but 2025 growth misses by −0.30pp (gate ±0.31) and 2040 unemployment by −0.19pp (gate ±0.2): both clear by only 0.01pp, which is not the same as matching | 4 Aug 2026 | tests/test_replication_baseline.py |
| Manual Table 4, total emissions, S1 baseline | DIVERGENCE — pinned at the observed ratios | 4 Aug 2026 | tests/test_replication_baseline.py |
| Scenario design — the policy switches each of the 8 checked scenarios toggles | PINNED EXACTLY | 5 Aug 2026 | tests/test_scenario_design.py |
| FMM 2023 open-vintage anchors — green public investment GDP peak; baseline 2030 emissions | PINNED, not a replication — peak +0.92% inside our 0.7–1.3% band; the “1% of GDP” recorded from FMM 2023 is the size of the investment; no published GDP response is confirmed. Baseline 2030 emissions 342.8 vs “just under 350” | 5 Aug 2026 | tests/test_scenario_design.py |
| Published numeric scenario results for v1.1 | CEILING REACHED — none are published | 5 Aug 2026 | validation/published_targets.json (search record) |
| Baseline vs external observations (ONS, DESNZ, OBR) | COMPUTED — gaps are material | 5 Aug 2026 | tests/test_validation.py |
| Green public investment cumulative multiplier | COMPUTED — 2.25 nominal per £ of power-sector spending, from the 2027 policy start to 2040; ours, not upstream’s; back-loaded, so below published short-horizon multipliers at their horizons; includes a parameter mismatch | 17 Nov 2026 | tests/test_validation.py |
| Clean-room milestone 1 — §2.2 matrices on the §5 initial values | PASS | 4 Aug 2026 | tests/test_accounting.py |
| Clean-room milestones 2 to 4 — macro and production, sectors, ecosystem, against the upstream oracle | NOT PASSED — every printed equation is registered and the whole model solves. GDP is within 0.41% with the other blocks held at the oracle, outside the 0.2% printing tolerance, and up to 44% below it by 2040 solved whole (the reimplementation section) | 17 Nov 2026 | tests/test_full_model.py, tests/test_oracle.py |
Macro block: the tolerances, and what each component hit
Tolerances: population and labour force ±0.01m; growth ±0.31pp; unemployment ±0.2pp. Against Table 4's macro block at 2025/2030/2040 on the S1 baseline, population and labour force are exact to 0.01m; growth matches to 0.06pp at 2030 and 2040 and misses by −0.30pp at 2025; unemployment is within 0.2pp, though 2040 misses by −0.19pp (4.53% vs 4.72%); and the 2025–40 mean and standard deviation of growth and unemployment match the table. The run is calendar-anchored on the population path (1987Q1–2040Q4, 2025 = t153).
The growth gate is not independent of the deviation it
admits. ±0.31pp was set after the run, and the only
year that uses any of it misses by −0.30pp — the gate
clears by 0.01pp. It is doing no work as a test of agreement in 2025;
what it does is stop that year drifting further, and the 2030/2040
matches to 0.06pp are the part of this gate that is genuinely
informative. The 2025 gap is not an annualisation convention: no single growth
convention reproduces all three years, and 2025 sits in the
post-2022 start-up transient
(validation/reference_outputs.json, re-checked
2026-11-16). It is an unresolved miss, not a demonstrated
agreement.
Emissions: 393 vs 407 in 2025, 343 vs 382 in 2030, 249 vs 324 by 2040
The pinned code runs below the published Table 4 emissions, and
the gap widens with the horizon: 393 vs 407
MtCO2e/yr in 2025 (−3.5%), 343 vs 382 in 2030
(−10.2%), 249 vs 324 by 2040 (−23.3%).
The percentages come from the pinned run/table ratios themselves
(0.965, 0.898, 0.767 in
validation/reference_outputs.json), not from the rounded
levels printed here.
Two checks rule out an extraction error on our side: the components
EMIS_NELEC + EMIS_ELEC sum exactly to EMIS, and the
baseline is identical across every scenario folder. A re-check of all 15 Table 4 rows
(2026-11-16) finds the power and energy block diverging while the
exogenous inputs and macro block match: consistent with, but not
proof of, the table coming from a different model vintage than
commit 846081a. The divergence has been queried with the
upstream authors, and the tests gate at the observed ratios, so any
further drift fails loudly.
Scenario design: the exact switch sets, the two anchors, and the search that closed the target
tests/test_scenario_design.py pins that each of the 8
checked scenarios toggles exactly the policy switches its published
description claims: FF_BAN plus capacity/investment ban timings for
the ban scenarios; POWER_SUB alone for the subsidy;
GVT_INVEST + GREEN_BONDS + GREEN_POWER exactly for green
public investment; and the published 40% housing subsidy rate,
HOUSING_SUB_RATE = 0.4.
Two coarse numeric anchors are taken from the open FMM 2023 conference version of the paper, held at wide tolerances because it is a pre-1.0 vintage. The cached run’s green public investment GDP peak is +0.92%, band 0.7–1.3% (includes the αPP mismatch); but the “1% of GDP” recorded from FMM 2023 is the size of its investment, no published GDP response has been confirmed, and its shock is larger and starts earlier. The current-policies baseline emits “just under 350 MtCO2e” in 2030 — the cached run gives 342.8, band 330–352. Both are pinned, not replicated.
Read those two as regression pins, not as agreement tests. Both bands were chosen by us to fit a verbal quantity, not published as tolerances, and neither can discriminate much: ±30% around “around +1%”, and an emissions band whose upper edge (352) is above the 350 the phrase “just under” is supposed to bound, so a run at 351 would pass a gate named for a claim it violates. What they establish is that the cached run has not moved since the anchors were recorded.
Nothing finer exists. An exhaustive search, catalogued in
validation/published_targets.json, established that no
machine-readable numeric scenario results are published for v1.1:
the manual's results stop at the baseline (Table 4) plus
scenario design parameters, and the George/Dafermos SSRN
paper (abstract 6541398) is paywalled with no open mirror. Its
scenario set — carbon pricing and subsidies — predates v1.1's
regulation policies in any case.
Calibration: the baseline against ONS and DESNZ outturns
Since 2026-08-05 this comparison is an artifact, not a paragraph.
define_uk.validation.baseline_calibration() recomputes
every row from the cached pinned run against vintage-exact external
observations; the result is committed as
validation/baseline_vs_external.csv; and
tests/test_validation.py fails on any drift between the
committed table and a recomputation.
The headline rows: 2025 real GDP growth of 4.66% (annual-mean convention; Table 4 prints 4.96%) against an ONS outturn of 1.31% y/y; 2025 unemployment of 4.39% against an ONS annual mean of 4.88%; population 16+ 55.66m vs ≈56m, consistent; labour force 35.97m, slightly high; and emissions starting above the DESNZ actuals. The full table is on the validation page.
Multiplier: our arithmetic, and how it sits against published ones
- 2.25 is our own computation: nominal cumulative ΔGDP over cumulative ΔSPEND_GVTPS, the government power-sector spending the policy switch sets, from the 2027Q1 policy start to 2040 (209.9 less 4.8 of pre-policy change, over 91.2).
- It is back-loaded. The profile by horizon, the full-run 2.30 and the 1.78 ratio per pound of all non-wage spending are on the validation page, their one home.
- Published comparators are short-horizon: Batini et al. (2021, IMF Working Paper WP/21/87) estimate about 1.1–1.5 for renewable-energy investment over a few years, and the OBR’s capital-spending impact multiplier is about 1.0.
- At those horizons this model’s ratio sits below them; it exceeds them only cumulated to 2040, so it is not like-for-like.
- An earlier IMF paper the repositories hold, Batini et al. (2014), measures a different quantity (first-year multipliers of 0–1 for advanced economies).
- It includes the parameter mismatch: 2.25 if that effect stops in 2027, as low as about 2.08 if it persists.
- The caveats travel with the number wherever a delta is served (
GPI_MULTIPLIER_CAVEATS, held by a test). - The upstream repository’s own
tables/Multiplier_Summary.csvis documented unusable and never quoted: its multiplier columns are identical across all eight scenarios.
Unlicensed upstream, pinned commit, and a run that proves nothing on its own
The upstream repository is public but carries no licence, so its code is never vendored or hosted.
The 2026-08-01 run record
R 4.3.0 on macOS, upstream unmodified, an rstudioapi shim the
only accommodation. The 151 files are the figures and tables for all
four scenario blocks plus
tables/Multiplier_Summary.csv.
What the run establishesExecution is not validation
The adapter
(github.com/PolicyEngine/define-uk-model)
fetches it at pinned commit 846081a at runtime and
executes it as published, in R, unmodified. That also fixes the run
surface: DEFINE-UK is local-only, never hosted, and
the eventual hosting path is the clean-room reimplementation in
the reimplementation section.
Execution ≠ validation. On 2026-08-01 the full
notebook rendered end to end through
define_uk.runner.run, producing 151 output files. That
demonstrated only that the pinned code executes, not that its outputs
mean anything. The adapter's VALIDATION.md holds the gate:
nothing built on the repository may present model output as meaningful
except by naming the target it rests on — the same bar applied to
BoE SVAR and US HANK.
The gates section records what that gate has since
established.
That file's own status line was wrong for a stretch, and the correction belongs in this record because this report cites it as the record. It read “NOT VALIDATED. No published DEFINE-UK result has been reproduced” for some time after target 1a passed, which put it in contradiction with every page that cited it. It now reads PARTIALLY VALIDATED and breaks the claim down target by target: 1a pass, 1b divergence pinned, targets 2 and 3 closed at the achievable ceiling rather than passed, reimplementation milestone 1 pass and milestone 2 outstanding. Nothing there licenses presenting scenario output as a validated result.
Deltas, never levels — closed at the achievable ceiling
Baseline levels sit far from official outturns: 2025 growth of 4.66% against an ONS 1.31%, unemployment below the outturn, emissions above the DESNZ actuals they should start from.
What the adapter servesDeltas only · the gate stack · the upgrade path
The manual itself says the baseline “should not be seen as a prediction”; we take that literally. Near-term levels are not competitive with the OBR emulator or the BoE SVAR and are never presented as forecasts.
So the adapter serves annualised scenario-minus-baseline delta paths
only — deltas, never levels — framed with mandatory caveats
including the demand-led closure, via pe-macro
define-scenario (local-only, experimental). The gate stack
behind them is the pinned upstream run as oracle, the exact
scenario-design pins, and the two FMM 2023 anchors. The upgrade path
is equally explicit: the authors publish scenario tables, or the SSRN
paper becomes accessible, and the published-figure gate reopens.
Reimplementing the published equations, milestone by milestone
Because the upstream code cannot be hosted, the hosting path is a from-scratch Python implementation of the published equations — the same play as the OBR emulator.
The clean-room protocolManual as specification · upstream as output oracle only
The protocol is
clean-room: the specification sources are the manual, the paper, and
Dafermos, Nikolaidi & Galanis (2017); the upstream repository is
an output oracle only, its source never read to write
equations; and every equation carries a manual_ref naming
the manual section and equation number it implements.
Three of the pinned manual defects, worked through
Table 6 LENDM_ROW omits the DIVN_ROW term of
Eq. (383), so the MFI/RoW transaction columns miss LEND by
∓DIVN_ROW (= 5.44); the manual itself tabulates overall
LENDM = 5.44 where it “should equal 0”. Pinned
exactly in tests/test_accounting.py.
Table 6's capital block is deflated at the wrong
price. Eq. (27), GCF_R = GCF/P_P,
gives 111.88 against a tabulated 112.30 — 0.37%, roughly ten times
the rounding noise — and Eq. (25) only reproduces the tabulated
GDP_R using 112.30. This is one finding, not
five: GCF, GCF_NFC, GCF_HH, GCF_GVT, K_P and K_NFC all show
nominal/real ratios of 1.0309–1.0312 where the equations
prescribe P_P = 1.035, and the same 0.4% reappears in
§3.3.2's Eqs. (104), (105), (132) and (133). One systematic
property of Table 6's capital rows — most likely a
national-accounts investment deflator where the equations want the
production deflator — counted once. Pinned in the tests rather than
absorbed into a tolerance.
Eq. (31) uses an intercept α₀λ that
Table 5 never tabulates — the α₀ block ends at α₀WS. It is
defaulted to 0.0, recorded in macro.MANUAL_GAPS, and
pinned by a test that fails if a value is later published.
The milestonesMilestone 1 passed · milestones 2 to 4 not passed
Milestone 1: PASS (2026-08-04). The §2.2 transactions
and balance-sheet matrices and the full §5 parameter and initial-value
tables (Tables 5–6) are transcribed, and every Table 1/2 row and
column identity holds on the §5 initial values within the manual's own
4-significant-figure printing precision. tests/test_accounting.py
collects 62 tests, of which 44 are
the Table 1/2 row and column identities themselves (20 transactions
rows, 6 transactions columns, 7 balance-sheet rows, 11 balance-sheet
columns); the remainder are the residual-instrument and net-worth
definitions and the two that pin the one manual inconsistency below.
Milestones 2 to 4: NOT PASSED (status 2026-11-17).
Every printed equation of the manual (§3.1–§3.4.7) is registered and the whole model solves quarter by quarter. Milestone 2 (macro and production, §3.2–3.3) is gated on the baseline GDP path: with the other blocks held at the upstream run it is within 0.41%, outside the 0.2% printing tolerance. Solved as a whole model (milestones 3 and 4, sectors and ecosystem), baseline GDP falls up to 44% below the upstream run by 2040 and the green-investment multiplier is 1.52 against the upstream’s full-run 2.30 (tests/test_oracle.py, tests/test_full_model.py).
Reimplementation is already earning its keep as a reading of the manual that the R source cannot give: thirteen manual defects are pinned in tests as machine-readable gap records rather than papered over or absorbed into a tolerance, with the wider set written up in the reimplementation notes. An audit of those notes also re-characterised four findings that were reading as the manual contradicting itself — Eqs. (51), (59), (94) and (111). Table 5 describes the parameters concerned as historical means or means of past implied values, and so never claimed to reproduce the manual's own initial period; those are first-period jumps a run started from Table 6 will take in its first quarter, not defects. The same audit withdrew one claim outright, on Eq. (51), where the corroborating inference had ignored a lag.
The model design belongs to its authors; the implementation is ours, AGPL-3.0, and independent — not the authors' code, and not endorsed by them.
Cite this work
Ahmadi, Vahid (2026). “DEFINE-UK: running and gating an unlicensed ecological SFC model.” PolicyEngine Macro replication report. https://policyengine-macro.vercel.app/reports/define-uk-replication
@techreport{policyengine2026defineuk,
title = {DEFINE-UK: running and gating an unlicensed ecological SFC model},
author = {Ahmadi, Vahid},
year = {2026},
institution = {PolicyEngine},
url = {https://policyengine-macro.vercel.app/reports/define-uk-replication},
note = {PolicyEngine Macro replication report}
}No DOI is assigned. For an exact version stamp, cite the git commit of the artifact you used from github.com/PolicyEngine/define-uk-model. Full validation record: VALIDATION.md and REIMPLEMENTATION.md in that repository, mirrored on the validation page.