1 files changed, 225 insertions, 0 deletions
diff --git a/working/context-engineering-rightsizing/metrics.org b/working/context-engineering-rightsizing/metrics.org
new file mode 100644
index 0000000..9cced4b
--- /dev/null
+++ b/working/context-engineering-rightsizing/metrics.org
@@ -0,0 +1,225 @@
+#+TITLE: Context-Engineering Rightsizing — Metrics and Stop Conditions
+#+AUTHOR: Craig Jennings
+#+DATE: 2026-07-27
+
+* Who is holding the instrument
+
+Most of what follows would be measured by me, about changes to my own
+instructions, in a direction I have an obvious interest in. Self-reported
+compliance is unreliable, and it is unreliable in a predictable direction: I
+will under-report misses I didn't notice, because not noticing is the failure.
+
+That is the argument for weighting mechanical detectors over my judgment
+wherever both exist. =lint-org= does not have a stake. Neither does a token
+count, a =git diff=, or you seeing a window on the wrong workspace. Where a
+claim can only be assessed by my self-report, it is marked below as judgment
+rather than measurement, and it should be discounted accordingly.
+
+* Part 1 — Which claims are testable
+
+Each post makes separable claims. Some can be tested here cheaply, some need an
+eval harness we don't have, and inventing a metric for the second group would be
+worse than admitting it.
+
+** Testable now, with existing instrumentation
+
+*** T1 — Report-everything beats pre-filtering (Opus 5 guide)
+
+*Claim.* A review told to be conservative reports less; reporting everything and
+filtering separately surfaces more real issues.
+
+*Test.* A true A/B, runnable today with no waiting. Take three past diffs with
+known outcomes. Run =/review-code= under current rules and under the
+report-all-then-filter version. Compare findings that survive verification.
+
+*Metric.* Real findings per pass, and false-positive rate. The claim holds if
+report-all finds more real issues without the false-positive rate rising
+enough to drown them.
+
+*Why this one first.* It is the only claim in the three posts we can settle in
+an afternoon instead of a week.
+
+*** T2 — Progressive disclosure preserves behavior (context-engineering post)
+
+*Claim.* Moving guidance out of always-loaded context into on-demand skills
+doesn't degrade adherence.
+
+*Test.* The Phase 1 pilot. Detailed in Part 2.
+
+*Metric.* Miss rate per exposure. Mechanical, since four of the six pilot rules
+have a checker.
+
+*** T3 — Deliverables run long without explicit calibration (Opus 5 guide)
+
+*Claim.* Files written to disk are longer than the task needs.
+
+*Test.* Measure what already exists. Word counts of the last twenty session
+archives, the specs in =docs/specs/=, and =todo.org= task bodies. Then add a
+length-calibration instruction and measure the next ten.
+
+*Metric.* Median words per artifact, before and after. Paired with a judgment
+call on whether anything useful was lost, which is the part I can't measure.
+
+*Baseline worth taking now,* since it costs one command and the before-number
+disappears the moment we change anything.
+
+*** T4 — Lower effort holds quality on mechanical passes (Opus 5 guide)
+
+*Claim.* =low= and =medium= produce strong quality at a fraction of the tokens.
+
+*Test.* Sentry fires hourly and does the same passes each time. Run a week at
+default, a week at =medium=, on the same repo state where possible.
+
+*Metric.* Tokens per fire, and findings per fire. The claim holds if findings
+per fire holds within noise while token cost drops materially.
+
+*Confound to respect.* Sentry's input changes night to night, so findings-per-
+fire is noisy. Two weeks is probably the floor for a readable signal, and the
+result will still be suggestive rather than conclusive.
+
+*** T5 — Duplicate and conflicting instructions cost something (both posts)
+
+*Claim.* Overlapping guidance across surfaces makes the model work harder to
+reconcile.
+
+*Test.* Partially measurable. The duplication itself is countable — Phase 5's
+three axes. Whether removing it improves anything is not measurable without
+evals.
+
+*Metric.* Count of rules stated on more than one surface, driven to zero. That
+measures the cleanup, not the benefit. Honest framing: we're removing a known
+cost, not demonstrating a gain.
+
+** Not testable here — judgment calls, and they should be labelled as such
+
+*** J1 — Examples constrain the exploration space
+
+The post asserts this and I have no way to test it. Removing the examples from
+=commits.md= and observing "things seem fine" is not evidence. This one gets
+adopted on the post's authority or not at all, and I'd lean toward *not*
+stripping examples that encode your taste, since the same post says skills
+should encode exactly that.
+
+*** J2 — Removing verification instructions loses no quality
+
+The claim underneath C1. Testing it properly means running the same tasks with
+and without =verification.md= and comparing error escape rates, which needs a
+task suite we don't have. What we *can* measure is the cost side — tool calls
+and tokens spent on verification per task — but not what it prevents.
+
+Asymmetric risk: the cost of over-verification is tokens, and the cost of
+under-verification is a false completion claim reaching you. Those are not
+equally bad, which argues for keeping the honesty core regardless of what a
+test would show.
+
+*** J3 — Unknowns-discovery reduces rework
+
+Long-horizon and confounded by everything else. Adopt on judgment.
+
+*** J4 — Rich references beat prose descriptions
+
+=ui-prototyping.md= already assumes it and it has worked. That's one project's
+experience, not a measurement, but it's the evidence we have.
+
+* Part 2 — Pilot go/no-go
+
+** The denominator problem
+
+The obvious metric is a miss rate, and the obvious trap is that a rule nobody
+exercised shows zero misses and reads as a pass. Every result below is
+therefore reported as *misses per exposure*, and a rule with too few exposures
+returns no verdict rather than a passing one.
+
+*Exposure* means a session did work in the rule's domain: edited an org table,
+touched a spec's lifecycle, displayed a keymap, captured a window, ran a UI
+spec.
+
+*Minimum exposures before a rule's result counts: 3.* Below that, extend the
+window or swap the rule for one that gets exercised more.
+
+** Primary metric
+
+Misses per exposure, per rule, where a miss is the guidance not being applied
+when it should have been.
+
+| Result per rule            | Verdict                                |
+|----------------------------+----------------------------------------|
+| 0 misses in 3+ exposures   | Pass                                   |
+|----------------------------+----------------------------------------|
+| 1 miss, detector caught it | Pass with note; record the miss        |
+|----------------------------+----------------------------------------|
+| 2 or more misses           | Turn back that rule                    |
+|----------------------------+----------------------------------------|
+| Miss reached a commit      | Turn back now, don't wait for the week |
+|----------------------------+----------------------------------------|
+| Miss reached a project     | Turn back now, and see abandon triggers|
+|----------------------------+----------------------------------------|
+| Under 3 exposures          | No verdict; extend or swap the rule    |
+|----------------------------+----------------------------------------|
+
+** Secondary metrics
+
+- *Token delta.* Measured directly. Expected around 3,000 words. This is the
+  only guaranteed benefit, so if the primary metric fails, we know exactly what
+  we were buying and can decide it wasn't worth it.
+- *Correction cost.* When a miss happens, how long to fix. A misformatted table
+  is seconds. This is what distinguishes an acceptable miss rate from an
+  unacceptable one, and it is why the threshold tightens in Phase 4.
+- *False-trigger rate.* A skill loading when it isn't needed spends tokens for
+  nothing. Worth watching, not worth blocking on.
+
+** Turn back versus abandon
+
+Two different actions, and conflating them would over-react to a single bad
+rule.
+
+*Turn back a rule* — revert that one file to always-loaded. Triggered by 2+
+misses on that rule, or any miss reaching a commit. The rest of the pilot
+continues. Cost: one revert.
+
+*Abandon the rollout* — stop at Phase 0 and keep the current architecture.
+Triggered by any of:
+
+- Three or more of the six rules turn back.
+- A miss reaches another project through the sync.
+- The pattern of misses shows the skill index isn't the fix, meaning D2 was the
+  wrong lever and there's no obvious next one.
+
+*Proceed to Phase 3* requires: at least four of six rules pass, no miss reached
+a commit, and the token drop landed near expectation.
+
+** Threshold scales with blast radius
+
+The pilot tolerates one caught miss per rule because the worst case is a badly
+formatted table. Phase 4 moves =commits.md=, where the worst case is an
+unattributed commit or a leaked path in a public artifact.
+
+*Phase 4 threshold is zero.* One miss on an attribution, scope, or grading rule
+turns that rule back immediately, with no pass-with-note tier. Stated now so
+it isn't negotiated later under the pressure of wanting the phase to succeed.
+
+* Part 3 — Baselines to capture before anything changes
+
+Cheap now, impossible to reconstruct later:
+
+1. Always-loaded word count per surface. Captured: 32,123 total.
+2. Median word count of the last twenty session archives, the specs, and open
+   task bodies. For T3.
+3. Sentry tokens per fire and findings per fire, from the existing metrics
+   file. For T4.
+4. Findings per pass from the last several =/review-code= runs. For T1.
+
+Item 1 is done. The rest are one session's work and should happen before
+Phase 0 changes the review skill, since that change contaminates item 4.
+
+* What this can and cannot tell us
+
+It can tell us whether on-demand loading fires reliably here, whether
+report-everything finds more real bugs, whether artifacts shrink usefully, and
+what lower effort costs in findings.
+
+It cannot tell us whether examples constrain, whether removing verification
+instructions is safe, or whether unknowns-discovery reduces rework. Those stay
+judgment calls made on the posts' authority and your read, and they should be
+labelled that way in whatever we write down afterward — so that a future session
+doesn't mistake an adopted opinion for a tested result.