diff options
Diffstat (limited to 'working/context-engineering-rightsizing')
| -rw-r--r-- | working/context-engineering-rightsizing/metrics.org | 269 | ||||
| -rw-r--r-- | working/context-engineering-rightsizing/proposals.org | 350 | ||||
| -rw-r--r-- | working/context-engineering-rightsizing/rollout.org | 371 |
3 files changed, 990 insertions, 0 deletions
diff --git a/working/context-engineering-rightsizing/metrics.org b/working/context-engineering-rightsizing/metrics.org new file mode 100644 index 0000000..e7ff189 --- /dev/null +++ b/working/context-engineering-rightsizing/metrics.org @@ -0,0 +1,269 @@ +#+TITLE: Context-Engineering Rightsizing — Metrics and Stop Conditions +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-27 + +* Who is holding the instrument + +Most of what follows would be measured by me, about changes to my own +instructions, in a direction I have an obvious interest in. Self-reported +compliance is unreliable, and it is unreliable in a predictable direction: I +will under-report misses I didn't notice, because not noticing is the failure. + +That is the argument for weighting mechanical detectors over my judgment +wherever both exist. =lint-org= does not have a stake. Neither does a token +count, a =git diff=, or you seeing a window on the wrong workspace. Where a +claim can only be assessed by my self-report, it is marked below as judgment +rather than measurement, and it should be discounted accordingly. + +* Part 1 — Which claims are testable + +Each post makes separable claims. Some can be tested here cheaply, some need an +eval harness we don't have, and inventing a metric for the second group would be +worse than admitting it. + +** Testable now, with existing instrumentation + +*** T1 — Report-everything beats pre-filtering (Opus 5 guide) + +*Claim.* A review told to be conservative reports less; reporting everything and +filtering separately surfaces more real issues. + +*Test.* A true A/B, runnable today with no waiting. Take three past diffs with +known outcomes. Run =/review-code= under current rules and under the +report-all-then-filter version. Compare findings that survive verification. + +*Metric.* Real findings per pass, and false-positive rate. The claim holds if +report-all finds more real issues without the false-positive rate rising +enough to drown them. + +*Why this one first.* It is the only claim in the three posts we can settle in +an afternoon instead of a week. + +*** T2 — Progressive disclosure preserves behavior (context-engineering post) + +*Claim.* Moving guidance out of always-loaded context into on-demand loading +doesn't degrade adherence. + +*ANSWERED 2026-07-27 for the deterministic half.* =/context= in a live work +session lists 17 generic rules under Memory files, with the three path-scoped +ones absent. Path-scoping is a glob match rather than a model judgment, so it +either fires or doesn't, and it fires. That half needs no trial and no miss-rate +metric. + +*Still open for the semantic half.* Rules whose condition can't be a glob +("when a commit is in play") route to skills, where triggering *is* a model +judgment. Everything in Part 2 applies there and only there. + +*A caution the answer doesn't cover.* Path-scoping fires when Claude *reads* a +matching file. A session that writes an org file without reading one first +never triggers =todo-format.md=. Edit requires a prior read, so edits to +existing files are safe; creating a new org file from scratch is the gap. Worth +watching rather than blocking on. + +*** T3a — The token baseline was wrong by 45% [SETTLED] + +Word counts converted at a guessed ~1.3 tokens per word. The live number is +2.28. =commits.md= is 12,800 tokens, not the ~7,000 estimated, and +=claude-rules/= was ~57,800 tokens per session before today rather than the +~33,000 implied. + +The lesson is narrower than "measure better." I had a real measurement +available the whole time — =/context= reports per-file token counts — and used +an estimate instead because the estimate was easier to compute from inside the +repo. Reach for the instrument that reports the actual quantity. + +*** T3 — Deliverables run long without explicit calibration (Opus 5 guide) + +*Claim.* Files written to disk are longer than the task needs. + +*Test.* Measure what already exists. Word counts of the last twenty session +archives, the specs in =docs/specs/=, and =todo.org= task bodies. Then add a +length-calibration instruction and measure the next ten. + +*Metric.* Median words per artifact, before and after. Paired with a judgment +call on whether anything useful was lost, which is the part I can't measure. + +*Baseline worth taking now,* since it costs one command and the before-number +disappears the moment we change anything. + +*** T4 — Lower effort holds quality [WITHDRAWN 2026-07-27] + +Dropped with P4. The goal is output quality first, and this claim trades +quality for cost, so testing it would answer a question we've decided not to +act on either way. + +*** T4 (original text, retained for the record) + +*Claim.* =low= and =medium= produce strong quality at a fraction of the tokens. + +*Test.* Sentry fires hourly and does the same passes each time. Run a week at +default, a week at =medium=, on the same repo state where possible. + +*Metric.* Tokens per fire, and findings per fire. The claim holds if findings +per fire holds within noise while token cost drops materially. + +*Confound to respect.* Sentry's input changes night to night, so findings-per- +fire is noisy. Two weeks is probably the floor for a readable signal, and the +result will still be suggestive rather than conclusive. + +*** T5 — Duplicate and conflicting instructions cost something (both posts) + +*Claim.* Overlapping guidance across surfaces makes the model work harder to +reconcile. + +*Test.* Partially measurable. The duplication itself is countable — Phase 5's +three axes. Whether removing it improves anything is not measurable without +evals. + +*Metric.* Count of rules stated on more than one surface, driven to zero. That +measures the cleanup, not the benefit. Honest framing: we're removing a known +cost, not demonstrating a gain. + +** Not testable here — judgment calls, and they should be labelled as such + +*** J1 — Examples constrain the exploration space + +The post asserts this and I have no way to test it. Removing the examples from +=commits.md= and observing "things seem fine" is not evidence. This one gets +adopted on the post's authority or not at all, and I'd lean toward *not* +stripping examples that encode your taste, since the same post says skills +should encode exactly that. + +*** J2 — Removing verification instructions loses no quality + +The claim underneath C1. Testing it properly means running the same tasks with +and without =verification.md= and comparing error escape rates, which needs a +task suite we don't have. What we *can* measure is the cost side — tool calls +and tokens spent on verification per task — but not what it prevents. + +Asymmetric risk: the cost of over-verification is tokens, and the cost of +under-verification is a false completion claim reaching you. Those are not +equally bad, which argues for keeping the honesty core regardless of what a +test would show. + +*** J3 — Unknowns-discovery reduces rework + +Long-horizon and confounded by everything else. Adopt on judgment. + +*** J4 — Rich references beat prose descriptions + +=ui-prototyping.md= already assumes it and it has worked. That's one project's +experience, not a measurement, but it's the evidence we have. + +* Part 2 — Pilot go/no-go + +** The denominator problem + +The obvious metric is a miss rate, and the obvious trap is that a rule nobody +exercised shows zero misses and reads as a pass. Every result below is +therefore reported as *misses per exposure*, and a rule with too few exposures +returns no verdict rather than a passing one. + +*Exposure* means a session did work in the rule's domain: edited an org table, +touched a spec's lifecycle, displayed a keymap, captured a window, ran a UI +spec. + +*Minimum exposures before a rule's result counts: 3.* Below that, extend the +window or swap the rule for one that gets exercised more. + +** Primary metric + +Misses per exposure, per rule, where a miss is the guidance not being applied +when it should have been. + +| Result per rule | Verdict | +|----------------------------+----------------------------------------| +| 0 misses in 3+ exposures | Pass | +|----------------------------+----------------------------------------| +| 1 miss, detector caught it | Pass with note; record the miss | +|----------------------------+----------------------------------------| +| 2 or more misses | Turn back that rule | +|----------------------------+----------------------------------------| +| Miss reached a commit | Turn back now, don't wait for the week | +|----------------------------+----------------------------------------| +| Miss reached a project | Turn back now, and see abandon triggers| +|----------------------------+----------------------------------------| +| Under 3 exposures | No verdict; extend or swap the rule | +|----------------------------+----------------------------------------| + +** Secondary metrics + +- *Token delta.* Measured directly. Expected around 3,000 words. This is the + only guaranteed benefit, so if the primary metric fails, we know exactly what + we were buying and can decide it wasn't worth it. +- *Correction cost.* When a miss happens, how long to fix. A misformatted table + is seconds. This is what distinguishes an acceptable miss rate from an + unacceptable one, and it is why the threshold tightens in Phase 4. +- *False-trigger rate.* A skill loading when it isn't needed spends tokens for + nothing. Worth watching, not worth blocking on. + +** Turn back versus abandon + +Two different actions, and conflating them would over-react to a single bad +rule. + +*Turn back a rule* — revert that one file to always-loaded. Triggered by 2+ +misses on that rule, or any miss reaching a commit. The rest of the pilot +continues. Cost: one revert. + +*Abandon the rollout* — stop at Phase 0 and keep the current architecture. +Triggered by any of: + +- Three or more of the six rules turn back. +- A miss reaches another project through the sync. +- The pattern of misses shows the skill index isn't the fix, meaning D2 was the + wrong lever and there's no obvious next one. + +*Proceed to Phase 3* requires: at least four of six rules pass, no miss reached +a commit, and the token drop landed near expectation. + +** Threshold scales with blast radius + +The pilot tolerates one caught miss per rule because the worst case is a badly +formatted table. Phase 4 moves =commits.md=, where the worst case is an +unattributed commit or a leaked path in a public artifact. + +*Phase 4 threshold is zero.* One miss on an attribution, scope, or grading rule +turns that rule back immediately, with no pass-with-note tier. Stated now so +it isn't negotiated later under the pressure of wanting the phase to succeed. + +* Part 3 — Baselines to capture before anything changes + +Cheap now, impossible to reconstruct later: + +1. Always-loaded word count per surface. Captured: 32,123 total. +2. Median word count of the last twenty session archives, the specs, and open + task bodies. For T3. +3. Sentry tokens per fire and findings per fire, from the existing metrics + file. For T4. +4. Findings per pass from the last several =/review-code= runs. For T1. + +Item 1 is done. The rest are one session's work and should happen before +Phase 0 changes the review skill, since that change contaminates item 4. + +* Instrument reliability — the day's second lesson + +The plan weights mechanical detectors over self-report because they have no +stake. Two failed on 2026-07-27, both reporting success while doing damage: +=wrap-org-table.el= reflowed a table into a worse shape and =lint-org= then +certified it clean, and the wrap-teardown hook consumed a two-hour-old sentinel +and killed a live work session that had done nothing wrong. + +Neither invalidates the preference for mechanical detectors, which is still +right. Both narrow the claim: a detector has no stake, but it can be +confidently wrong, and a green check from a guard that never looked at the +thing is indistinguishable from a green check that did. When a detector clears +a moved rule, the useful question is whether it actually evaluated it, not just +whether it reported clean. + +* What this can and cannot tell us + +It can tell us whether on-demand loading fires reliably here, whether +report-everything finds more real bugs, whether artifacts shrink usefully, and +what lower effort costs in findings. + +It cannot tell us whether examples constrain, whether removing verification +instructions is safe, or whether unknowns-discovery reduces rework. Those stay +judgment calls made on the posts' authority and your read, and they should be +labelled that way in whatever we write down afterward — so that a future session +doesn't mistake an adopted opinion for a tested result. diff --git a/working/context-engineering-rightsizing/proposals.org b/working/context-engineering-rightsizing/proposals.org new file mode 100644 index 0000000..1300a7b --- /dev/null +++ b/working/context-engineering-rightsizing/proposals.org @@ -0,0 +1,350 @@ +#+TITLE: Context-Engineering Rightsizing — Proposals +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-27 + +* Status + +Proposals only. Nothing here is applied. Source: three Anthropic posts Craig +supplied 2026-07-27 — the Claude 5 context-engineering post (2026-07-24), the +Opus 5 prompting guide, and the Fable 5 field guide (2026-07-06). + +* Goal + +Output quality and results first. Token reduction is a real goal and worth +having, but it is the second one, and where the two conflict quality wins. +Craig's framing, 2026-07-27: the concern is agents having the freedom to +produce at their highest capacity, unbound by guardrails that constrain them or +work against them. + +That reordering matters more than it sounds. Anthropic's 80% figure was a +*finding* — they cut and quality held — not a target. Aimed at directly it +optimizes the thing we don't care about, and P4 below is the proposal that +falls to it. + +* The measurement (corrected 2026-07-27 from live =/context=) + +The original figures in this document were word counts converted at a guessed +ratio, and they understated by about 45%. A =/context= run in a live work +session gave the real numbers: + +| Surface | Tokens | +|------------------------------------+--------| +| =claude-rules/= before this session | 57,800 | +|------------------------------------+--------| +| =claude-rules/= now (17 files) | 44,410 | +|------------------------------------+--------| +| Path-scoped out (3 files) | 13,390 | +|------------------------------------+--------| + +The real ratio is 2.28 tokens per word, not the ~1.3 assumed. =commits.md= +alone is 12,800 tokens, not the ~7,000 estimated. Every earlier figure here +should be read as low. + +** Two loading paths, not one + +The original framing called 32,123 words "always-loaded" and was wrong to +lump them. They arrive by different mechanisms: + +- *Memory files* — =claude-rules/= (via the =~/.claude/rules/= symlinks), + =CLAUDE.md=, project =.claude/rules/=, and auto-memory's =MEMORY.md=. Loaded + at session start by the harness. This is the 44,410 above, plus CLAUDE.md and + project rules. +- *Read during startup* — =protocols.org= and the workflow files. These never + appear under Memory files in =/context=; the startup workflow reads them, so + they land in Messages. Still a real per-session cost, but a different lever: + they shrink by editing the workflow, not by scoping a rule. + +Conflating the two made =protocols.org= look like it competed with +=commits.md= for the same fix. It doesn't. + +** What the harness says on its own + +=/context= ends with its own suggestion: prune =commits.md= (12.8k), +=testing.md= (6.3k), and =MEMORY.md= (5.5k). That is the Phase 4 target list, +arrived at independently. Worth treating as corroboration rather than +coincidence. + +* Proposals + +Ranked by value. Each carries my confidence and what I think the real risk is. + +** P1 — Progressive disclosure for =claude-rules/= [high value, highest risk] + +*Change.* Split the twenty rule files into two tiers. + +- *Always-loaded core* (target under 3,000 words): the genuine invariants that + must fire without being summoned. The no-AI-attribution rule, the + cross-project boundary stop, the no-popup-menus and no-reverse-video output + constraints, the =date=-before-timestamps rule, and a short index naming + which skill covers what. +- *On-demand tier*: everything procedural, converted to skills whose + descriptions trigger them. =commits.md= becomes a publish skill that loads + when a commit or PR is in play. =todo-format.md= loads when an org todo file + is touched. =testing.md=, =working-files.md=, =docs-lifecycle.md=, + =org-tables.md=, =keybinding-display.md= likewise. + +*Why.* This is the post's central move, and the token math is the argument. + +*The real risk, stated plainly.* A rule that isn't loaded can't fire. Skill +triggering is probabilistic in a way that always-resident text isn't. The +failure mode is silent: a session commits without the voice pass because the +publish skill didn't trigger, and nothing announces the miss. That's the same +silent-failure shape as the two probe defects fixed this morning. + +*Mitigation.* Anything whose violation is expensive and hard to reverse stays +in the always-loaded core, whatever its length. The split is by *blast radius*, +not by word count. And the migration goes one file at a time with a live trial, +not as a single cutover. + +*Confidence.* High that the direction is right. Medium on where exactly each +line falls — that's a judgment call per rule, and worth walking together. + +** P2 — Stop pre-filtering review findings [high value, low risk] + +*Change.* =review-code/SKILL.md:251= says "Drop Low-confidence issues before +the final report." Line 434 repeats it. Replace with: report every finding +carrying an explicit confidence label, then filter in a named second pass. + +*Why.* The Opus 5 guide is specific about this: "If your review prompt says +'only report high-severity issues' or 'be conservative,' the model may follow +that instruction literally and report less; ask it to report everything and +filter in a separate pass instead." The guide also reports that on this model +the extra findings are mostly real rather than false positives, which is the +premise the drop-rule was written against. + +*Confidence.* High. This is the most directly-actionable finding in the three +posts, and it names the exact pattern the skill implements. + +** P3 — Add the unknowns-discovery practices [medium value, low risk] + +The field guide describes eight practices. The sweep found these absent: +=unknown unknowns= framing (0 files), =implementation notes= (0), =quiz= (0). +Present but thin: =blind spot= (1 file), =interview= (2), =pitch= (1). +=brainstorm= appears in 5 and =references= is well covered by +=ui-prototyping.md=, which already runs ahead of the post. + +*Change.* Add a blind-spot-pass practice and an interview practice, and an +implementation-notes convention for long builds (a scratch file logging +deviations from plan, which then feeds the retrospective). Add the quiz pattern +to the review or wrap surface. + +*Placement matters.* These go in the on-demand tier from P1, never the +always-loaded core — otherwise this proposal fights the one above it. + +*Confidence.* Medium-high on the practices being useful. Lower on the quiz, +which may not fit how you actually work. + +** P4 — Effort calibration [DROPPED 2026-07-27 — trades quality for cost] + +*Change.* Sentry, work-the-backlog, and the no-approvals speedrun run many +passes at default effort. The Opus 5 guide says to use =low= and =medium= +liberally as the primary cost control wherever quality holds, stepping up only +for demanding work. + +*Why.* Sentry fires hourly. Effort is the lever with the largest cost +multiple, and most sentry passes are mechanical sweeps. + +*Dropped.* Lowering effort buys tokens by spending quality, which is exactly +backwards under the goal above. Revisit only if a specific pass proves +genuinely mechanical and a cost problem shows up on its own. + +** P5 — Positive framing over prohibition [PROMOTED — now the top quality lever] + +*Change.* =commits.md= carries 41 prohibition markers (NEVER / DO NOT / +MANDATORY / CRITICAL) across 5,561 words. =todo-format.md= 21, +=testing.md= 19. Rewrite the ones that aren't hard invariants as positive +descriptions of the wanted behavior. + +*Why.* The Opus 5 guide: "Positive examples of the communication style you want +tend to be more effective than instructions about what not to do." + +*Keep as prohibitions:* the AI-attribution rule and anything else where the +worst case is genuinely unacceptable. Those earn their emphasis. + +*Promoted 2026-07-27.* Ranked low when the score was token savings. Under a +quality-first goal this is the proposal that most directly targets guardrails +working against good output, which is the stated concern. It still rides along +with the Phase 4 edits rather than running as its own campaign, because every +file it touches is a file those phases open anyway. + +** P6 — Deduplicate the two always-loaded surfaces [medium value, low risk] + +=protocols.org= restates rules that also live in =claude-rules/=: the +cross-project boundary, the working-files convention, the AI-attribution ban, +inbox cadence. Both are loaded every session, so each duplicated rule is paid +for twice, and the two copies can drift apart. + +*Change.* One home per rule. =protocols.org= keeps the pointer, the rule file +keeps the content — or the reverse, but not both. + +*Confidence.* High on the duplication being real, medium on which surface +should own each rule. + +* Conflicts — your call, not mine to inherit + +Two places where a post contradicts something this system arrived at +deliberately. I'm flagging rather than adopting. + +** C1 — =verification.md= versus the over-verification warning + +The Opus 5 guide says: "If your prompt contains explicit verification +instructions ('include a final verification step for any non-trivial task', +'use a subagent to verify'), remove them: instructions like these cause +over-verification on Claude Opus 5, and removing them reduces wasted tokens +with no loss in quality." + +=verification.md= is 1,486 always-loaded words of exactly that shape. But the +two aren't the same thing, and the distinction decides the answer: + +- Its *honesty core* — don't claim tests pass without running them, "unable to + verify" is a required outcome, replace beliefs with evidence — is about + truthful reporting, not about adding verification steps. The guide doesn't + argue against it. +- Its *process injection* — green baseline before starting, full suite as its + own step before every commit — is the shape the guide names. + +*My read:* keep the honesty core, shorten it, and let the process injection +move into the publish skill where it fires only when publishing. *My +confidence: medium*, and this is the one I'd most want you to overrule if it +feels wrong. It's also the rule closest to your standing "never guess, always +check" direction, so the guide's advice and your stated preference genuinely +pull against each other here. + +** C2 — =subagents.md= versus the delegation warning + +The guide says "do not use subagents to verify or double-check your own work." +=subagents.md= has a review-gate cadence and a rule to dispatch a *fix* +subagent rather than repairing in the orchestrator's context. + +*My read:* the fix-subagent rule is about context pollution, not verification, +so it survives. The review-gate cadence is closer to the flagged pattern and +wants a look. Most of =subagents.md= already matches the guide's advice — it +argues against spawning for small work and against letting the agent pick its +own scope, which is what the guide asks for. *Confidence: medium-high.* + +* One thing the posts would change about this document + +Both the context-engineering post and the field guide argue that a rich +reference beats a prose description — an HTML artifact, a test suite, source +code. This proposal document is prose in org, which is your reading format and +the right call for a decision doc. Worth noting the tension rather than +silently ignoring it: for the *next* artifact in this line of work, an HTML +comparison of the before and after rule tree would likely beat another org +file. + +* Scope proposal for the consistency sweep + +The audit behind this document was targeted, not exhaustive — I checked the +claims the posts made and measured the surface. A real inconsistency sweep over +20 rule files, 47 workflows, and the skills is its own pass. + +Proposed scope, in order: + +1. *Contradictions between always-loaded surfaces* — the P6 duplication set, + read side by side for drift rather than just counted. +2. *Stale facts* — assertions about tools, paths, and behavior that were true + when written. The spot check found the =agent-page= to =agent-text= rename + correctly handled, so this may be in better shape than expected. +3. *Instructions that contradict each other across files* — the failure the + context-engineering post opens with. This is the expensive one and the most + valuable. + +Sizing: item 1 is an afternoon, item 2 is mechanical, item 3 is the real work. + +* From your side of the desk + +Everything above treats these files as the agent's context, to be rightsized. +That was the smaller question. Read as *your prompts* — the map you hand every +project — the finding is different, and it's the one worth acting on. + +** The bottleneck this system was built for has moved + +Counting the workflows by what they're for: 41 are execution and hygiene +(publish, task grading, inbox routing, session archiving, calendar, email, +sync), 6 are discovery and design (the spec trio, retrospective, code-quality, +readability-audit). Roughly seven to one. + +That ratio was correct when the risk was the model doing things wrong. The +field guide's claim is that the risk moved: "Claude Fable is the first model +where I find the quality of the work is bottlenecked by my ability to clarify +its unknowns." If that's true here too, the system is heavily invested in the +half of the problem that got easier and thin on the half that didn't. + +Not an argument to delete the execution machinery. It's load-bearing, and +hygiene that runs itself is exactly what you want automated. The argument is +that *the growth has all been on one side*, and the next increment of quality +probably comes from the other one. + +** The instructions don't practice what they demand + +Three concrete cases, all checkable: + +- =commits.md= says "Brief. Terse is preferred. A one-sentence body beats a + paragraph saying the same thing." It is 5,561 words, the longest file in the + set. +- =interaction.md= bans bold and code spans in chat output because they render + as reverse video. The rule files carry 591 bold markers. +- =testing.md= mandates TDD as "non-negotiable" and follows with a table of + eight rationalizations to refuse. That's the repeat-yourself-and-overconstrain + shape the context-engineering post retired, applied to a rule the model no + longer needs argued into. + +This matters beyond tidiness. An instruction whose own form contradicts its +content is a mixed signal of exactly the kind the post opens with — the +model spends effort reconciling "be terse" against a source that isn't. + +** Over-specification has a cost you're paying, not me + +The field guide: "If you are too specific, Claude will follow your instructions +even when a pivot may be more appropriate. If you are too vague, Claude will +often make choices and assumptions based on industry best practices that may +not be a fit." + +The workflows are phase-numbered and gate-heavy. For repeatable operations — +wrap-up, publish, inbox — that precision is right and should stay. For the +creative surfaces it's a straitjacket: a spec workflow that prescribes its own +phases forecloses the pivot that a brainstorm was supposed to surface. The +system doesn't currently distinguish those two classes, and it should. Same +file format, opposite optimal specificity. + +** Which gates are guardrails and which are preference + +The approval gates (publish, inbox shared-asset, spec) were written when the +worst case was an agent shipping something wrong. Some of them you'd want +regardless, because you want to be in the loop on what goes out under your +name. Those are preference and should never be cut. + +Others are pure guardrail, and the context-engineering post's argument applies: +"these constraints were once needed to avoid worst case scenarios, we have +since found we can delete many of them." + +I can't tell the two apart from the files — they read identically. Only you +know which gates you'd keep if the agent were perfectly reliable. That +separation is the highest-leverage thing you could tell me, and it's not +something I should guess at. + +** One map, many territories + +Every project loads the same 32,000 words. The finances project loads the TDD +mandate and the full publish machinery. =.emacs.d= loads the org-table +standard. Some of that is right (the universal rules genuinely are universal) +and some is the map/territory mismatch the field guide names. + +The language-bundle mechanism already solves this for =languages/=. Nothing +equivalent exists for the rules layer. + +** What's already right + +Worth saying, because the list above is all deficit: + +- =ui-prototyping.md= is the field guide's brainstorm-and-prototype practice, + written down before the post and in more detail than the post gives it. +- The spec spine (decisions plus implementation phases) is the guide's + implementation-plan pattern. +- =session-context.org= is the implementation-notes pattern, and it already + captures deviations and dead ends. +- =retrospective.org= exists. + +So the practices are partly here. What's missing is that they're framed as +*process compliance* rather than as *unknowns discovery*, and none of them fire +at the moment the guide says they pay off — before scope is set. diff --git a/working/context-engineering-rightsizing/rollout.org b/working/context-engineering-rightsizing/rollout.org new file mode 100644 index 0000000..d641ce7 --- /dev/null +++ b/working/context-engineering-rightsizing/rollout.org @@ -0,0 +1,371 @@ +#+TITLE: Context-Engineering Rightsizing — Rollout Schedule +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-27 + +* Why this is phased rather than done in one pass + +The change is a bet that on-demand loading fires as reliably as always-resident +text. That bet is cheap to test and expensive to assume. Every phase below +either produces evidence or spends evidence already earned. Nothing +load-bearing moves before the mechanism has been watched working. + +The second reason is blast radius. Everything here rides the template sync into +every project on its next startup, so a bad phase is not contained to this +repo. The early phases are chosen so a failure is visible and harmless. + +* A finding that changes the plan + +The harness system prompt already carries most of what the Opus 5 guide +recommends adding. Its task-scope block, its correction-narration block, and +its subagent-delegation cap are present nearly verbatim. The +context-engineering post's replacement comment guidance ("write code that reads +like the surrounding code") is present as the post's own new wording. + +Two consequences: + +1. *Do not "apply the posts" by adding their suggested prompt blocks.* They are + already live. Adding them to =claude-rules/= would create exactly the + duplicate-and-conflict problem the first post opens with, while making the + token count worse. +2. *There is a third deduplication axis.* The proposals named =protocols.org= + against =claude-rules/=. There is also =claude-rules/= against the harness + system prompt, and that one is invisible from inside the repo. Any rule that + restates harness guidance is pure cost. + +This is why the posts' value here is subtractive, not additive. + +* Status — 2026-07-27, end of first working session + +Phase 0 and the deterministic half of Phase 1 are done and verified in a live +session. The pilot's central question is answered, which changes what remains. + +** Shipped + +- =paths:= frontmatter on the three rules that already declared a file-type + scope in prose (=todo-format=, =org-tables=, =emacs=), plus a =lint.sh= + checker that catches the prose/frontmatter mismatch, plus a heading check + taught to skip frontmatter. Commit 0adcb1a. +- Generic rules no longer ship per project. =install-lang= stopped copying them + and =sync-language-bundle= sweeps what earlier installs left, guarded on the + global rule existing. Swept 20 files each from work and =.emacs.d=. Commit + 7ea1d7b. +- The live session anchor is gitignored, so rulesets stops reporting + sync-blocked for the whole of every session. Same commit. + +** Verified in a live work session + +=/context= lists 17 generic rules under Memory files. =todo-format.md=, +=org-tables.md=, and =emacs.md= are absent, and only =python-testing.md= and +=publishing.md= come from the project's own rules directory. + +So *path-scoping works at user level* and *the de-duplication holds*. Both were +open questions this morning. + +** What that changes + +Path-scoping is a glob match, not a model judgment. It is deterministic, so the +silent-miss risk the whole pilot was designed around does not apply to it. That +splits the remaining work in two: + +1. *Path-scopable* — any rule whose scope is a file type or directory. Ships + immediately, no trial, no detectors. =docs-lifecycle.md= is the obvious next + one (=docs/**=), and parts of =working-files.md= may qualify. +2. *Semantic* — rules whose condition can't be written as a glob ("any spec + with a non-trivial UI", "when a commit is in play"). These still need the + skills route, and they are the only place the pilot's detectors and stop + conditions apply. + +=commits.md= is the case that matters: 12,800 tokens, the single largest item, +and almost all of it is publish machinery that only applies when a commit is in +play. That is a task scope rather than a path scope, so it is the skills route +and the real test of the risky tier. + +** Correction carried from the live numbers + +Earlier phases in this document quote word counts converted at a guessed ratio +and understate by about 45%. The real ratio is 2.28 tokens per word. Read the +targets below as token figures needing that correction, and see proposals.org +for the corrected table. + +** A pattern worth designing around + +Two mechanical guards failed in the same day, both mine, both reporting success +while doing damage: =wrap-org-table.el= reflowed a table into a worse shape and +=lint-org= then certified it clean, and the wrap-teardown hook consumed a +two-hour-old sentinel and killed a live work session. + +This plan leans on mechanical detectors precisely because they have no stake in +the outcome. Both incidents say that is necessary but not sufficient — a +detector can be confidently wrong. Whatever Phase 3 decides, the verification +step should include "did the guard's own claim get checked," not just "did the +guard report clean." + +* Phase 0 — Free wins [DONE except /doctor] + +*Scope.* Three items that interact with nothing. + +1. Fix the review-finding pre-filter (P2). =review-code/SKILL.md= lines 251 and + 434 tell the reviewer to drop low-confidence findings before reporting. + Replace with report-everything-labelled, filter in a named second pass. +2. Run =/doctor=. The context-engineering post says Anthropic shipped these + practices as a command that rightsizes skills and =CLAUDE.md=. Its output is + free evidence, and it may disagree with this plan, which is worth knowing + before executing it. +3. Record the harness-overlap finding above where it will be seen at the moment + it matters — a note in the rules index, not buried in this document. + +*Reasoning.* None of these depend on the pilot's outcome, and item 2 could +change the plan. + +*Decision needed:* none. + +*Success criteria.* Review skill reports with confidence labels and a separate +filter step. =/doctor= output read and reconciled against this schedule. + +*Rollback.* Single revert; nothing downstream depends on it. + +* Phase 1 — The pilot migration [deterministic half DONE; semantic half pending] + +*Scope.* Six rule files move from always-loaded to on-demand skills. Roughly +3,000 words, about 12% of the rules surface. + +| File | Words | How a silent miss would be caught | +|-----------------------+-------+-------------------------------------------| +| =org-tables.md= | 464 | =lint-org= checker =org-table-standard= | +|-----------------------+-------+-------------------------------------------| +| =docs-lifecycle.md= | 582 | spec status-board grep; =lint-org= checkers | +|-----------------------+-------+-------------------------------------------| +| =ui-prototyping.md= | 696 | =spec-review= verifies the process ran | +|-----------------------+-------+-------------------------------------------| +| =keybinding-display.md= | 505 | you see the wrong format immediately | +|-----------------------+-------+-------------------------------------------| +| =desktop-capture.md= | 458 | a window lands on your active workspace | +|-----------------------+-------+-------------------------------------------| +| =patterns.md= | 291 | already only a pointer; nothing to miss | +|-----------------------+-------+-------------------------------------------| + +*Reasoning — the selection rule matters more than the list.* These were not +picked for being small or cheap. They were picked because *a failure to fire is +detectable*. Four have a mechanical checker or workflow gate that catches the +miss; two produce an error you see within seconds. That is what makes the pilot +an experiment rather than a hope. + +=daily-drivers.md= and =emacs.md= were considered and held back. Both are +low-risk in content, but a miss on either surfaces slowly — as drift on the +other machine, or as a stale daemon — so neither would tell us anything within +the trial window. + +*Decisions needed.* + +- *D1 — Confirm the pilot set.* Six files as listed, or trim further. My + recommendation is the six: fewer than that and the trial may not exercise the + mechanism enough to learn from. +- *D2 — Does the always-loaded core carry a skill index?* A one-line-per-skill + list naming what exists and when it applies. It costs perhaps 200 words and + should materially improve trigger reliability, since the model can see that a + rule exists even when its content isn't loaded. My recommendation is yes, and + the pilot is the right place to test whether the index is what does the work. + +*Success criteria.* Always-loaded surface drops to about 29,000 words. All six +skills exist with trigger descriptions. Suite green, sync clean, every project +picks up the change on next startup without drift. + +*Rollback.* One revert restores the files to =claude-rules/=. The skills can +stay in place harmlessly. + +* Phase 2 — Live trial (one week of real sessions, no work required) + +*Scope.* Use the system normally. Do not compensate for the pilot by mentioning +the moved rules — that would invalidate the result. + +*Reasoning.* This is the phase that buys everything after it. The question is +narrow and answerable: when work touches one of the six domains, does the skill +fire without prompting? + +*What gets recorded.* Each session that touches a pilot domain notes one line +in the session log: which domain, whether the skill fired, and whether the +detector caught anything. At the end of the week that's a short table rather +than an impression. + +*Decision needed:* none during the trial. + +*Success criteria.* Defined in advance so the verdict isn't argued after the +fact: + +- *Pass* — no detector fires on a moved rule, or any miss is caught by its + detector and corrected in the same session. +- *Fail* — a miss reaches a commit, or the same rule misses twice. +- *Ambiguous* — no session touched the domain. That is not a pass; extend the + window or move a rule that gets exercised more. + +*Rollback.* Revert on a Fail, and the plan stops at Phase 0. + +* Phase 3 — Go/no-go and the gate separation (one session) + +*Scope.* Read the trial table, decide whether the mechanism is trusted, and +separate the approval gates. + +*Reasoning.* The gate separation is the highest-leverage input in the whole +plan, and it sits here rather than earlier for one reason: if Phase 2 fails, +the question is moot, because nothing more moves either way. + +*Decision needed.* + +- *D3 — Which gates are preference and which are guardrail?* Every approval gate + in the system reads identically in the files. Some you would keep even if the + agent were perfectly reliable, because you want to see what goes out under + your name. Others exist because the worst case used to be worse. The list to + walk: the publish approval gate, the inbox shared-asset approval, the + spec-review flip, the wrap certification, and the no-approvals mode's carve + outs. + + Preference gates are untouchable and stay always-loaded regardless of length. + Guardrail gates are candidates for relaxation on the posts' argument. I can + prepare the list with my read of each, but the answers are yours. + +*Success criteria.* Every gate labelled. The label determines what Phase 4 may +move. + +* Phase 4 — The load-bearing files (two or three sessions) + +*Scope.* =commits.md= (5,561), =todo-format.md= (4,494), =testing.md= (2,824), +=working-files.md= (950), =subagents.md= (1,041). About 15,000 words, the bulk +of the remaining surface. + +The split within each file is by blast radius, not by length. =commits.md= is +the worked example: the AI-attribution ban and the content-scope rule stay +always-loaded and get shorter, while the publish flow, the message format, and +the voice mechanics become the publish skill that loads when a commit is in +play. + +*Reasoning.* This is where the token math actually pays. It runs last because +it is where a silent miss is expensive: an unattributed commit, a leaked path, +an ungraded task. + +*Decision needed.* + +- *D4 — Resolve the =verification.md= conflict (C1).* The Opus 5 guide says + explicit verification instructions cause over-verification and should be + removed. Your standing direction is never guess, always check. My read is + that these are compatible because they address different things: the honesty + core (don't claim a green suite you didn't run) stays, and the process + injection (green baseline before starting, suite as its own step) moves into + the publish skill. But it is your rule and your call, and this decision blocks + =commits.md= moving because the two files reference each other. + +*Success criteria.* Always-loaded surface under about 8,000 words. Two full +weeks of sessions with no attribution, scope, or grading miss. + +*Rollback.* Per-file, since each moves independently. + +* Phase 5 — Deduplication (one session) + +*Scope.* Three axes, in increasing order of payoff: + +1. =protocols.org= against =claude-rules/= — the cross-project boundary, + working-files, AI-attribution, and inbox cadence are each stated twice. +2. =claude-rules/= against the harness system prompt — the finding at the top + of this document. Invisible from inside the repo and therefore never audited. +3. Within =claude-rules/= — rules that restate each other. + +*Decision needed.* + +- *D5 — Which surface owns each duplicated rule.* Generally the more specific + one should own the content and the more general should carry a pointer, but + there are cases where the reverse is right. + +*Success criteria.* Each rule stated once. A stated rule for where new rules go, +so the duplication doesn't regrow. + +* Phase 6 — Terseness and positive framing (rides along with Phases 4 and 5) + +*Scope.* Rewrite prohibitions that aren't hard invariants as positive +descriptions. Cut the files that don't practice what they demand: =commits.md= +arguing terseness at 5,561 words, =testing.md= arguing an eight-row table +against rationalizations the model no longer needs talked out of, 591 bold +markers in files that ban bold in output. + +*Reasoning.* Not a separate campaign. Every file opened in Phases 4 and 5 gets +this pass while it's open, because doing it separately means editing everything +twice. + +*Decision needed:* none. This is style, and the voice skill already owns the +standard. + +* Phase 7 — Discovery practices (after the surface is down) + +*Scope.* The field guide's missing practices: a blind-spot pass, an interview +pattern, an implementation-notes convention for long builds, and possibly the +quiz. + +*Reasoning.* Deliberately last, for two reasons. It adds surface, which fights +every phase before it, so it should land only once there is room. And the +seven-to-one execution-to-discovery ratio is the finding most likely to change +how the system actually feels to use, which makes it worth doing carefully +rather than early. + +*Decision needed.* + +- *D6 — Which practices you actually want.* I have low confidence on the quiz + fitting how you work, and medium-high on the rest. + +* Phase 8 — Effort calibration [DROPPED 2026-07-27] + +*Scope.* Set effort levels for the unattended loops: sentry's hourly fires, +work-the-backlog, the no-approvals speedrun. + +*Dropped.* Buys tokens by spending quality, which inverts the stated goal. +D7 is withdrawn with it. + +*Decision needed.* + +- *D7 — Accepted quality floor for unattended passes.* A sentry sweep that runs + cheaper but misses one finding per night may be a good trade or a bad one. + That's a preference, not a measurement. + +* Ongoing — The consistency sweep + +Runs alongside, not as a phase. Each file opened in Phases 4 through 6 gets read +for contradictions and stale facts while it's open, and findings go to a running +list rather than being fixed opportunistically. The expensive item — instructions +that contradict each other across files — is what the first post opens with and +what this whole exercise is downstream of. + +* Decisions, collected + +| ID | Decision | Needed by | +|----+----------------------------------------------+-----------| +| D1 | Confirm the six-file pilot set | Phase 1 | +|----+----------------------------------------------+-----------| +| D2 | Skill index in the always-loaded core? | Phase 1 | +|----+----------------------------------------------+-----------| +| D3 | Which gates are preference vs guardrail | Phase 3 | +|----+----------------------------------------------+-----------| +| D4 | Resolve the verification.md conflict | Phase 4 | +|----+----------------------------------------------+-----------| +| D5 | Which surface owns each duplicated rule | Phase 5 | +|----+----------------------------------------------+-----------| +| D6 | Which discovery practices you want | Phase 7 | +|----+----------------------------------------------+-----------| +| D7 | Quality floor for unattended passes | Phase 8 | +|----+----------------------------------------------+-----------| + +Only D1 and D2 are needed to start. + +* The number this is aiming at + +| Stage | Always-loaded words | +|-------------------+---------------------| +| Today | 32,123 | +|-------------------+---------------------| +| After Phase 1 | 29,100 | +|-------------------+---------------------| +| After Phase 4 | under 8,000 | +|-------------------+---------------------| +| After Phase 5 | under 6,000 | +|-------------------+---------------------| + +Roughly an 80% reduction, which lands near what Anthropic reported. That +symmetry is a coincidence worth distrusting rather than aiming for: the target +is whatever survives the blast-radius test, and if that turns out to be 12,000 +words then 12,000 words is the right answer. |