#+TITLE: Context-Engineering Rightsizing — Metrics and Stop Conditions #+AUTHOR: Craig Jennings #+DATE: 2026-07-27 * Who is holding the instrument Most of what follows would be measured by me, about changes to my own instructions, in a direction I have an obvious interest in. Self-reported compliance is unreliable, and it is unreliable in a predictable direction: I will under-report misses I didn't notice, because not noticing is the failure. That is the argument for weighting mechanical detectors over my judgment wherever both exist. =lint-org= does not have a stake. Neither does a token count, a =git diff=, or you seeing a window on the wrong workspace. Where a claim can only be assessed by my self-report, it is marked below as judgment rather than measurement, and it should be discounted accordingly. * Part 1 — Which claims are testable Each post makes separable claims. Some can be tested here cheaply, some need an eval harness we don't have, and inventing a metric for the second group would be worse than admitting it. ** Testable now, with existing instrumentation *** T1 — Report-everything beats pre-filtering (Opus 5 guide) *Claim.* A review told to be conservative reports less; reporting everything and filtering separately surfaces more real issues. *Test.* A true A/B, runnable today with no waiting. Take three past diffs with known outcomes. Run =/review-code= under current rules and under the report-all-then-filter version. Compare findings that survive verification. *Metric.* Real findings per pass, and false-positive rate. The claim holds if report-all finds more real issues without the false-positive rate rising enough to drown them. *Why this one first.* It is the only claim in the three posts we can settle in an afternoon instead of a week. *** T2 — Progressive disclosure preserves behavior (context-engineering post) *Claim.* Moving guidance out of always-loaded context into on-demand skills doesn't degrade adherence. *Test.* The Phase 1 pilot. Detailed in Part 2. *Metric.* Miss rate per exposure. Mechanical, since four of the six pilot rules have a checker. *** T3 — Deliverables run long without explicit calibration (Opus 5 guide) *Claim.* Files written to disk are longer than the task needs. *Test.* Measure what already exists. Word counts of the last twenty session archives, the specs in =docs/specs/=, and =todo.org= task bodies. Then add a length-calibration instruction and measure the next ten. *Metric.* Median words per artifact, before and after. Paired with a judgment call on whether anything useful was lost, which is the part I can't measure. *Baseline worth taking now,* since it costs one command and the before-number disappears the moment we change anything. *** T4 — Lower effort holds quality on mechanical passes (Opus 5 guide) *Claim.* =low= and =medium= produce strong quality at a fraction of the tokens. *Test.* Sentry fires hourly and does the same passes each time. Run a week at default, a week at =medium=, on the same repo state where possible. *Metric.* Tokens per fire, and findings per fire. The claim holds if findings per fire holds within noise while token cost drops materially. *Confound to respect.* Sentry's input changes night to night, so findings-per- fire is noisy. Two weeks is probably the floor for a readable signal, and the result will still be suggestive rather than conclusive. *** T5 — Duplicate and conflicting instructions cost something (both posts) *Claim.* Overlapping guidance across surfaces makes the model work harder to reconcile. *Test.* Partially measurable. The duplication itself is countable — Phase 5's three axes. Whether removing it improves anything is not measurable without evals. *Metric.* Count of rules stated on more than one surface, driven to zero. That measures the cleanup, not the benefit. Honest framing: we're removing a known cost, not demonstrating a gain. ** Not testable here — judgment calls, and they should be labelled as such *** J1 — Examples constrain the exploration space The post asserts this and I have no way to test it. Removing the examples from =commits.md= and observing "things seem fine" is not evidence. This one gets adopted on the post's authority or not at all, and I'd lean toward *not* stripping examples that encode your taste, since the same post says skills should encode exactly that. *** J2 — Removing verification instructions loses no quality The claim underneath C1. Testing it properly means running the same tasks with and without =verification.md= and comparing error escape rates, which needs a task suite we don't have. What we *can* measure is the cost side — tool calls and tokens spent on verification per task — but not what it prevents. Asymmetric risk: the cost of over-verification is tokens, and the cost of under-verification is a false completion claim reaching you. Those are not equally bad, which argues for keeping the honesty core regardless of what a test would show. *** J3 — Unknowns-discovery reduces rework Long-horizon and confounded by everything else. Adopt on judgment. *** J4 — Rich references beat prose descriptions =ui-prototyping.md= already assumes it and it has worked. That's one project's experience, not a measurement, but it's the evidence we have. * Part 2 — Pilot go/no-go ** The denominator problem The obvious metric is a miss rate, and the obvious trap is that a rule nobody exercised shows zero misses and reads as a pass. Every result below is therefore reported as *misses per exposure*, and a rule with too few exposures returns no verdict rather than a passing one. *Exposure* means a session did work in the rule's domain: edited an org table, touched a spec's lifecycle, displayed a keymap, captured a window, ran a UI spec. *Minimum exposures before a rule's result counts: 3.* Below that, extend the window or swap the rule for one that gets exercised more. ** Primary metric Misses per exposure, per rule, where a miss is the guidance not being applied when it should have been. | Result per rule | Verdict | |----------------------------+----------------------------------------| | 0 misses in 3+ exposures | Pass | |----------------------------+----------------------------------------| | 1 miss, detector caught it | Pass with note; record the miss | |----------------------------+----------------------------------------| | 2 or more misses | Turn back that rule | |----------------------------+----------------------------------------| | Miss reached a commit | Turn back now, don't wait for the week | |----------------------------+----------------------------------------| | Miss reached a project | Turn back now, and see abandon triggers| |----------------------------+----------------------------------------| | Under 3 exposures | No verdict; extend or swap the rule | |----------------------------+----------------------------------------| ** Secondary metrics - *Token delta.* Measured directly. Expected around 3,000 words. This is the only guaranteed benefit, so if the primary metric fails, we know exactly what we were buying and can decide it wasn't worth it. - *Correction cost.* When a miss happens, how long to fix. A misformatted table is seconds. This is what distinguishes an acceptable miss rate from an unacceptable one, and it is why the threshold tightens in Phase 4. - *False-trigger rate.* A skill loading when it isn't needed spends tokens for nothing. Worth watching, not worth blocking on. ** Turn back versus abandon Two different actions, and conflating them would over-react to a single bad rule. *Turn back a rule* — revert that one file to always-loaded. Triggered by 2+ misses on that rule, or any miss reaching a commit. The rest of the pilot continues. Cost: one revert. *Abandon the rollout* — stop at Phase 0 and keep the current architecture. Triggered by any of: - Three or more of the six rules turn back. - A miss reaches another project through the sync. - The pattern of misses shows the skill index isn't the fix, meaning D2 was the wrong lever and there's no obvious next one. *Proceed to Phase 3* requires: at least four of six rules pass, no miss reached a commit, and the token drop landed near expectation. ** Threshold scales with blast radius The pilot tolerates one caught miss per rule because the worst case is a badly formatted table. Phase 4 moves =commits.md=, where the worst case is an unattributed commit or a leaked path in a public artifact. *Phase 4 threshold is zero.* One miss on an attribution, scope, or grading rule turns that rule back immediately, with no pass-with-note tier. Stated now so it isn't negotiated later under the pressure of wanting the phase to succeed. * Part 3 — Baselines to capture before anything changes Cheap now, impossible to reconstruct later: 1. Always-loaded word count per surface. Captured: 32,123 total. 2. Median word count of the last twenty session archives, the specs, and open task bodies. For T3. 3. Sentry tokens per fire and findings per fire, from the existing metrics file. For T4. 4. Findings per pass from the last several =/review-code= runs. For T1. Item 1 is done. The rest are one session's work and should happen before Phase 0 changes the review skill, since that change contaminates item 4. * What this can and cannot tell us It can tell us whether on-demand loading fires reliably here, whether report-everything finds more real bugs, whether artifacts shrink usefully, and what lower effort costs in findings. It cannot tell us whether examples constrain, whether removing verification instructions is safe, or whether unknowns-discovery reduces rework. Those stay judgment calls made on the posts' authority and your read, and they should be labelled that way in whatever we write down afterward — so that a future session doesn't mistake an adopted opinion for a tested result.