path: root/working/context-engineering-rightsizing/metrics.org

#+TITLE: Context-Engineering Rightsizing — Metrics and Stop Conditions
#+AUTHOR: Craig Jennings
#+DATE: 2026-07-27

* Who is holding the instrument

Most of what follows would be measured by me, about changes to my own
instructions, in a direction I have an obvious interest in. Self-reported
compliance is unreliable, and it is unreliable in a predictable direction: I
will under-report misses I didn't notice, because not noticing is the failure.

That is the argument for weighting mechanical detectors over my judgment
wherever both exist. =lint-org= does not have a stake. Neither does a token
count, a =git diff=, or you seeing a window on the wrong workspace. Where a
claim can only be assessed by my self-report, it is marked below as judgment
rather than measurement, and it should be discounted accordingly.

* Part 1 — Which claims are testable

Each post makes separable claims. Some can be tested here cheaply, some need an
eval harness we don't have, and inventing a metric for the second group would be
worse than admitting it.

** Testable now, with existing instrumentation

*** T1 — Report-everything beats pre-filtering (Opus 5 guide)

*Claim.* A review told to be conservative reports less; reporting everything and
filtering separately surfaces more real issues.

*Test.* A true A/B, runnable today with no waiting. Take three past diffs with
known outcomes. Run =/review-code= under current rules and under the
report-all-then-filter version. Compare findings that survive verification.

*Metric.* Real findings per pass, and false-positive rate. The claim holds if
report-all finds more real issues without the false-positive rate rising
enough to drown them.

*Why this one first.* It is the only claim in the three posts we can settle in
an afternoon instead of a week.

*** T2 — Progressive disclosure preserves behavior (context-engineering post)

*Claim.* Moving guidance out of always-loaded context into on-demand loading
doesn't degrade adherence.

*ANSWERED 2026-07-27 for the deterministic half.* =/context= in a live work
session lists 17 generic rules under Memory files, with the three path-scoped
ones absent. Path-scoping is a glob match rather than a model judgment, so it
either fires or doesn't, and it fires. That half needs no trial and no miss-rate
metric.

*Still open for the semantic half.* Rules whose condition can't be a glob
("when a commit is in play") route to skills, where triggering *is* a model
judgment. Everything in Part 2 applies there and only there.

*A caution the answer doesn't cover.* Path-scoping fires when Claude *reads* a
matching file. A session that writes an org file without reading one first
never triggers =todo-format.md=. Edit requires a prior read, so edits to
existing files are safe; creating a new org file from scratch is the gap. Worth
watching rather than blocking on.

*** T3a — The token baseline was wrong by 45% [SETTLED]

Word counts converted at a guessed ~1.3 tokens per word. The live number is
2.28. =commits.md= is 12,800 tokens, not the ~7,000 estimated, and
=claude-rules/= was ~57,800 tokens per session before today rather than the
~33,000 implied.

The lesson is narrower than "measure better." I had a real measurement
available the whole time — =/context= reports per-file token counts — and used
an estimate instead because the estimate was easier to compute from inside the
repo. Reach for the instrument that reports the actual quantity.

*** T3 — Deliverables run long without explicit calibration (Opus 5 guide)

*Claim.* Files written to disk are longer than the task needs.

*Test.* Measure what already exists. Word counts of the last twenty session
archives, the specs in =docs/specs/=, and =todo.org= task bodies. Then add a
length-calibration instruction and measure the next ten.

*Metric.* Median words per artifact, before and after. Paired with a judgment
call on whether anything useful was lost, which is the part I can't measure.

*Baseline worth taking now,* since it costs one command and the before-number
disappears the moment we change anything.

*** T4 — Lower effort holds quality [WITHDRAWN 2026-07-27]

Dropped with P4. The goal is output quality first, and this claim trades
quality for cost, so testing it would answer a question we've decided not to
act on either way.

*** T4 (original text, retained for the record)

*Claim.* =low= and =medium= produce strong quality at a fraction of the tokens.

*Test.* Sentry fires hourly and does the same passes each time. Run a week at
default, a week at =medium=, on the same repo state where possible.

*Metric.* Tokens per fire, and findings per fire. The claim holds if findings
per fire holds within noise while token cost drops materially.

*Confound to respect.* Sentry's input changes night to night, so findings-per-
fire is noisy. Two weeks is probably the floor for a readable signal, and the
result will still be suggestive rather than conclusive.

*** T5 — Duplicate and conflicting instructions cost something (both posts)

*Claim.* Overlapping guidance across surfaces makes the model work harder to
reconcile.

*Test.* Partially measurable. The duplication itself is countable — Phase 5's
three axes. Whether removing it improves anything is not measurable without
evals.

*Metric.* Count of rules stated on more than one surface, driven to zero. That
measures the cleanup, not the benefit. Honest framing: we're removing a known
cost, not demonstrating a gain.

** Not testable here — judgment calls, and they should be labelled as such

*** J1 — Examples constrain the exploration space

The post asserts this and I have no way to test it. Removing the examples from
=commits.md= and observing "things seem fine" is not evidence. This one gets
adopted on the post's authority or not at all, and I'd lean toward *not*
stripping examples that encode your taste, since the same post says skills
should encode exactly that.

*** J2 — Removing verification instructions loses no quality

The claim underneath C1. Testing it properly means running the same tasks with
and without =verification.md= and comparing error escape rates, which needs a
task suite we don't have. What we *can* measure is the cost side — tool calls
and tokens spent on verification per task — but not what it prevents.

Asymmetric risk: the cost of over-verification is tokens, and the cost of
under-verification is a false completion claim reaching you. Those are not
equally bad, which argues for keeping the honesty core regardless of what a
test would show.

*** J3 — Unknowns-discovery reduces rework

Long-horizon and confounded by everything else. Adopt on judgment.

*** J4 — Rich references beat prose descriptions

=ui-prototyping.md= already assumes it and it has worked. That's one project's
experience, not a measurement, but it's the evidence we have.

* Part 2 — Pilot go/no-go

** The denominator problem

The obvious metric is a miss rate, and the obvious trap is that a rule nobody
exercised shows zero misses and reads as a pass. Every result below is
therefore reported as *misses per exposure*, and a rule with too few exposures
returns no verdict rather than a passing one.

*Exposure* means a session did work in the rule's domain: edited an org table,
touched a spec's lifecycle, displayed a keymap, captured a window, ran a UI
spec.

*Minimum exposures before a rule's result counts: 3.* Below that, extend the
window or swap the rule for one that gets exercised more.

** Primary metric

Misses per exposure, per rule, where a miss is the guidance not being applied
when it should have been.

| Result per rule            | Verdict                                |
|----------------------------+----------------------------------------|
| 0 misses in 3+ exposures   | Pass                                   |
|----------------------------+----------------------------------------|
| 1 miss, detector caught it | Pass with note; record the miss        |
|----------------------------+----------------------------------------|
| 2 or more misses           | Turn back that rule                    |
|----------------------------+----------------------------------------|
| Miss reached a commit      | Turn back now, don't wait for the week |
|----------------------------+----------------------------------------|
| Miss reached a project     | Turn back now, and see abandon triggers|
|----------------------------+----------------------------------------|
| Under 3 exposures          | No verdict; extend or swap the rule    |
|----------------------------+----------------------------------------|

** Secondary metrics

- *Token delta.* Measured directly. Expected around 3,000 words. This is the
  only guaranteed benefit, so if the primary metric fails, we know exactly what
  we were buying and can decide it wasn't worth it.
- *Correction cost.* When a miss happens, how long to fix. A misformatted table
  is seconds. This is what distinguishes an acceptable miss rate from an
  unacceptable one, and it is why the threshold tightens in Phase 4.
- *False-trigger rate.* A skill loading when it isn't needed spends tokens for
  nothing. Worth watching, not worth blocking on.

** Turn back versus abandon

Two different actions, and conflating them would over-react to a single bad
rule.

*Turn back a rule* — revert that one file to always-loaded. Triggered by 2+
misses on that rule, or any miss reaching a commit. The rest of the pilot
continues. Cost: one revert.

*Abandon the rollout* — stop at Phase 0 and keep the current architecture.
Triggered by any of:

- Three or more of the six rules turn back.
- A miss reaches another project through the sync.
- The pattern of misses shows the skill index isn't the fix, meaning D2 was the
  wrong lever and there's no obvious next one.

*Proceed to Phase 3* requires: at least four of six rules pass, no miss reached
a commit, and the token drop landed near expectation.

** Threshold scales with blast radius

The pilot tolerates one caught miss per rule because the worst case is a badly
formatted table. Phase 4 moves =commits.md=, where the worst case is an
unattributed commit or a leaked path in a public artifact.

*Phase 4 threshold is zero.* One miss on an attribution, scope, or grading rule
turns that rule back immediately, with no pass-with-note tier. Stated now so
it isn't negotiated later under the pressure of wanting the phase to succeed.

* Part 3 — Baselines to capture before anything changes

Cheap now, impossible to reconstruct later:

1. Always-loaded word count per surface. Captured: 32,123 total.
2. Median word count of the last twenty session archives, the specs, and open
   task bodies. For T3.
3. Sentry tokens per fire and findings per fire, from the existing metrics
   file. For T4.
4. Findings per pass from the last several =/review-code= runs. For T1.

Item 1 is done. The rest are one session's work and should happen before
Phase 0 changes the review skill, since that change contaminates item 4.

* Instrument reliability — the day's second lesson

The plan weights mechanical detectors over self-report because they have no
stake. Two failed on 2026-07-27, both reporting success while doing damage:
=wrap-org-table.el= reflowed a table into a worse shape and =lint-org= then
certified it clean, and the wrap-teardown hook consumed a two-hour-old sentinel
and killed a live work session that had done nothing wrong.

Neither invalidates the preference for mechanical detectors, which is still
right. Both narrow the claim: a detector has no stake, but it can be
confidently wrong, and a green check from a guard that never looked at the
thing is indistinguishable from a green check that did. When a detector clears
a moved rule, the useful question is whether it actually evaluated it, not just
whether it reported clean.

* What this can and cannot tell us

It can tell us whether on-demand loading fires reliably here, whether
report-everything finds more real bugs, whether artifacts shrink usefully, and
what lower effort costs in findings.

It cannot tell us whether examples constrain, whether removing verification
instructions is safe, or whether unknowns-discovery reduces rework. Those stay
judgment calls made on the posts' authority and your read, and they should be
labelled that way in whatever we write down afterward — so that a future session
doesn't mistake an adopted opinion for a tested result.