diff options
| author | Craig Jennings <c@cjennings.net> | 2026-07-27 13:46:32 -0500 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2026-07-27 13:46:32 -0500 |
| commit | d74d98d03e7321b6c69dae9ef427da66b0f6fbc6 (patch) | |
| tree | 4d61b604085afc019b40903a9e8db9072b46147c /working | |
| parent | 2c664cb5651fbf03bc52d7848add0c571708adcd (diff) | |
| download | rulesets-d74d98d03e7321b6c69dae9ef427da66b0f6fbc6.tar.gz rulesets-d74d98d03e7321b6c69dae9ef427da66b0f6fbc6.zip | |
docs: correct the rightsizing docs against live measurements
I had the goal backwards in the lead. Craig's framing is output quality first, with token reduction real but second, and where they conflict quality wins. Anthropic's 80% was a finding rather than a target, and aiming at it directly optimizes the thing we don't care about. P4 falls to that and is dropped: lowering effort buys tokens by spending quality. P5 rises for the same reason, since positive framing over prohibition is what actually targets guardrails working against output.
My token figures were low by about 45%. I converted word counts at a guessed ratio when /context reports real per-file numbers, so I used an estimate because it was easier to compute from inside the repo. The live ratio is 2.28 tokens per word. commits.md is 12,800 tokens rather than the 7,000 I estimated, and claude-rules was near 57,800 per session before today.
I also conflated two loading paths. Memory files come in through the harness at session start. protocols.org and the workflows are read by startup and land in Messages, so they never appear under Memory files at all. They shrink by editing the workflow, not by scoping a rule, and treating them as one surface made protocols.org look like it competed with commits.md for the same fix.
Path-scoping is confirmed working at user level, and the de-duplication holds. That splits the remaining work: anything scopable to a file type or directory ships with no trial, because a glob match is deterministic. Only the semantic rules need the skills route and the stop conditions, and commits.md is the one that matters there at 12,800 tokens of publish machinery that only applies when a commit is in play.
I recorded one caution the confirmation doesn't cover. Path-scoping fires when a matching file is read, so creating a new org file from scratch never triggers todo-format.md. Edits are safe because Edit requires a prior read.
I also wrote up the day's two guard failures, both mine. wrap-org-table reflowed a table into a worse shape and lint-org certified it clean, and the teardown hook consumed a stale sentinel and killed a live work session. The plan leans on mechanical detectors because they have no stake, which is still right, but a detector can be confidently wrong and a green check from a guard that never looked is indistinguishable from one that did.
Diffstat (limited to 'working')
| -rw-r--r-- | working/context-engineering-rightsizing/metrics.org | 54 | ||||
| -rw-r--r-- | working/context-engineering-rightsizing/proposals.org | 99 | ||||
| -rw-r--r-- | working/context-engineering-rightsizing/rollout.org | 77 |
3 files changed, 185 insertions, 45 deletions
diff --git a/working/context-engineering-rightsizing/metrics.org b/working/context-engineering-rightsizing/metrics.org index 9cced4b..e7ff189 100644 --- a/working/context-engineering-rightsizing/metrics.org +++ b/working/context-engineering-rightsizing/metrics.org @@ -41,13 +41,36 @@ an afternoon instead of a week. *** T2 — Progressive disclosure preserves behavior (context-engineering post) -*Claim.* Moving guidance out of always-loaded context into on-demand skills +*Claim.* Moving guidance out of always-loaded context into on-demand loading doesn't degrade adherence. -*Test.* The Phase 1 pilot. Detailed in Part 2. +*ANSWERED 2026-07-27 for the deterministic half.* =/context= in a live work +session lists 17 generic rules under Memory files, with the three path-scoped +ones absent. Path-scoping is a glob match rather than a model judgment, so it +either fires or doesn't, and it fires. That half needs no trial and no miss-rate +metric. -*Metric.* Miss rate per exposure. Mechanical, since four of the six pilot rules -have a checker. +*Still open for the semantic half.* Rules whose condition can't be a glob +("when a commit is in play") route to skills, where triggering *is* a model +judgment. Everything in Part 2 applies there and only there. + +*A caution the answer doesn't cover.* Path-scoping fires when Claude *reads* a +matching file. A session that writes an org file without reading one first +never triggers =todo-format.md=. Edit requires a prior read, so edits to +existing files are safe; creating a new org file from scratch is the gap. Worth +watching rather than blocking on. + +*** T3a — The token baseline was wrong by 45% [SETTLED] + +Word counts converted at a guessed ~1.3 tokens per word. The live number is +2.28. =commits.md= is 12,800 tokens, not the ~7,000 estimated, and +=claude-rules/= was ~57,800 tokens per session before today rather than the +~33,000 implied. + +The lesson is narrower than "measure better." I had a real measurement +available the whole time — =/context= reports per-file token counts — and used +an estimate instead because the estimate was easier to compute from inside the +repo. Reach for the instrument that reports the actual quantity. *** T3 — Deliverables run long without explicit calibration (Opus 5 guide) @@ -63,7 +86,13 @@ call on whether anything useful was lost, which is the part I can't measure. *Baseline worth taking now,* since it costs one command and the before-number disappears the moment we change anything. -*** T4 — Lower effort holds quality on mechanical passes (Opus 5 guide) +*** T4 — Lower effort holds quality [WITHDRAWN 2026-07-27] + +Dropped with P4. The goal is output quality first, and this claim trades +quality for cost, so testing it would answer a question we've decided not to +act on either way. + +*** T4 (original text, retained for the record) *Claim.* =low= and =medium= produce strong quality at a fraction of the tokens. @@ -212,6 +241,21 @@ Cheap now, impossible to reconstruct later: Item 1 is done. The rest are one session's work and should happen before Phase 0 changes the review skill, since that change contaminates item 4. +* Instrument reliability — the day's second lesson + +The plan weights mechanical detectors over self-report because they have no +stake. Two failed on 2026-07-27, both reporting success while doing damage: +=wrap-org-table.el= reflowed a table into a worse shape and =lint-org= then +certified it clean, and the wrap-teardown hook consumed a two-hour-old sentinel +and killed a live work session that had done nothing wrong. + +Neither invalidates the preference for mechanical detectors, which is still +right. Both narrow the claim: a detector has no stake, but it can be +confidently wrong, and a green check from a guard that never looked at the +thing is indistinguishable from a green check that did. When a detector clears +a moved rule, the useful question is whether it actually evaluated it, not just +whether it reported clean. + * What this can and cannot tell us It can tell us whether on-demand loading fires reliably here, whether diff --git a/working/context-engineering-rightsizing/proposals.org b/working/context-engineering-rightsizing/proposals.org index 40bfcf8..1300a7b 100644 --- a/working/context-engineering-rightsizing/proposals.org +++ b/working/context-engineering-rightsizing/proposals.org @@ -8,33 +8,61 @@ Proposals only. Nothing here is applied. Source: three Anthropic posts Craig supplied 2026-07-27 — the Claude 5 context-engineering post (2026-07-24), the Opus 5 prompting guide, and the Fable 5 field guide (2026-07-06). -* The measurement - -Every session in every project loads this before the user's first word: - -| Surface | Words | -|--------------------------+--------| -| =claude-rules/= (20 files) | 25,386 | -|--------------------------+--------| -| =protocols.org= | 6,620 | -|--------------------------+--------| -| Project =CLAUDE.md= | 117 | -|--------------------------+--------| -| *Total* | 32,123 | -|--------------------------+--------| - -Roughly 40,000 tokens of always-on context. The context-engineering post reports -Anthropic cut over 80% of Claude Code's *system prompt* with no measurable loss -on their coding evals. That's their number on their surface, not a target for -this one, but the direction is the finding: this system is built in the "then" -style the post retired. - -The content isn't the problem. The post says skills are best when they "encode -particular opinions, knowledge, or best practices particular to you" — which is -exactly what these rules are. The problem is *load timing*: all of it is -resident on every turn, including =commits.md= (5,561 words) in a session that -never commits, and =todo-format.md= (4,494 words) in a session that never opens -an org file. +* Goal + +Output quality and results first. Token reduction is a real goal and worth +having, but it is the second one, and where the two conflict quality wins. +Craig's framing, 2026-07-27: the concern is agents having the freedom to +produce at their highest capacity, unbound by guardrails that constrain them or +work against them. + +That reordering matters more than it sounds. Anthropic's 80% figure was a +*finding* — they cut and quality held — not a target. Aimed at directly it +optimizes the thing we don't care about, and P4 below is the proposal that +falls to it. + +* The measurement (corrected 2026-07-27 from live =/context=) + +The original figures in this document were word counts converted at a guessed +ratio, and they understated by about 45%. A =/context= run in a live work +session gave the real numbers: + +| Surface | Tokens | +|------------------------------------+--------| +| =claude-rules/= before this session | 57,800 | +|------------------------------------+--------| +| =claude-rules/= now (17 files) | 44,410 | +|------------------------------------+--------| +| Path-scoped out (3 files) | 13,390 | +|------------------------------------+--------| + +The real ratio is 2.28 tokens per word, not the ~1.3 assumed. =commits.md= +alone is 12,800 tokens, not the ~7,000 estimated. Every earlier figure here +should be read as low. + +** Two loading paths, not one + +The original framing called 32,123 words "always-loaded" and was wrong to +lump them. They arrive by different mechanisms: + +- *Memory files* — =claude-rules/= (via the =~/.claude/rules/= symlinks), + =CLAUDE.md=, project =.claude/rules/=, and auto-memory's =MEMORY.md=. Loaded + at session start by the harness. This is the 44,410 above, plus CLAUDE.md and + project rules. +- *Read during startup* — =protocols.org= and the workflow files. These never + appear under Memory files in =/context=; the startup workflow reads them, so + they land in Messages. Still a real per-session cost, but a different lever: + they shrink by editing the workflow, not by scoping a rule. + +Conflating the two made =protocols.org= look like it competed with +=commits.md= for the same fix. It doesn't. + +** What the harness says on its own + +=/context= ends with its own suggestion: prune =commits.md= (12.8k), +=testing.md= (6.3k), and =MEMORY.md= (5.5k). That is the Phase 4 target list, +arrived at independently. Worth treating as corroboration rather than +coincidence. * Proposals @@ -106,7 +134,7 @@ always-loaded core — otherwise this proposal fights the one above it. *Confidence.* Medium-high on the practices being useful. Lower on the quiz, which may not fit how you actually work. -** P4 — Effort calibration on the unattended loops [medium value, low risk] +** P4 — Effort calibration [DROPPED 2026-07-27 — trades quality for cost] *Change.* Sentry, work-the-backlog, and the no-approvals speedrun run many passes at default effort. The Opus 5 guide says to use =low= and =medium= @@ -116,11 +144,11 @@ for demanding work. *Why.* Sentry fires hourly. Effort is the lever with the largest cost multiple, and most sentry passes are mechanical sweeps. -*Confidence.* Medium. The guide says to re-run an effort sweep on your own -evals rather than trusting carried-over defaults, and we have no evals — so -this wants a trial and a judgment, not a blanket change. +*Dropped.* Lowering effort buys tokens by spending quality, which is exactly +backwards under the goal above. Revisit only if a specific pass proves +genuinely mechanical and a cost problem shows up on its own. -** P5 — Positive framing over prohibition [low value, low risk, slow] +** P5 — Positive framing over prohibition [PROMOTED — now the top quality lever] *Change.* =commits.md= carries 41 prohibition markers (NEVER / DO NOT / MANDATORY / CRITICAL) across 5,561 words. =todo-format.md= 21, @@ -133,8 +161,11 @@ tend to be more effective than instructions about what not to do." *Keep as prohibitions:* the AI-attribution rule and anything else where the worst case is genuinely unacceptable. Those earn their emphasis. -*Confidence.* Medium. Real but diffuse, and it competes with P1 for the same -editing effort. Probably rides along with P1 rather than standing alone. +*Promoted 2026-07-27.* Ranked low when the score was token savings. Under a +quality-first goal this is the proposal that most directly targets guardrails +working against good output, which is the stated concern. It still rides along +with the Phase 4 edits rather than running as its own campaign, because every +file it touches is a file those phases open anyway. ** P6 — Deduplicate the two always-loaded surfaces [medium value, low risk] diff --git a/working/context-engineering-rightsizing/rollout.org b/working/context-engineering-rightsizing/rollout.org index 4afc27f..d641ce7 100644 --- a/working/context-engineering-rightsizing/rollout.org +++ b/working/context-engineering-rightsizing/rollout.org @@ -34,7 +34,73 @@ Two consequences: This is why the posts' value here is subtractive, not additive. -* Phase 0 — Free wins (this session, no dependencies) +* Status — 2026-07-27, end of first working session + +Phase 0 and the deterministic half of Phase 1 are done and verified in a live +session. The pilot's central question is answered, which changes what remains. + +** Shipped + +- =paths:= frontmatter on the three rules that already declared a file-type + scope in prose (=todo-format=, =org-tables=, =emacs=), plus a =lint.sh= + checker that catches the prose/frontmatter mismatch, plus a heading check + taught to skip frontmatter. Commit 0adcb1a. +- Generic rules no longer ship per project. =install-lang= stopped copying them + and =sync-language-bundle= sweeps what earlier installs left, guarded on the + global rule existing. Swept 20 files each from work and =.emacs.d=. Commit + 7ea1d7b. +- The live session anchor is gitignored, so rulesets stops reporting + sync-blocked for the whole of every session. Same commit. + +** Verified in a live work session + +=/context= lists 17 generic rules under Memory files. =todo-format.md=, +=org-tables.md=, and =emacs.md= are absent, and only =python-testing.md= and +=publishing.md= come from the project's own rules directory. + +So *path-scoping works at user level* and *the de-duplication holds*. Both were +open questions this morning. + +** What that changes + +Path-scoping is a glob match, not a model judgment. It is deterministic, so the +silent-miss risk the whole pilot was designed around does not apply to it. That +splits the remaining work in two: + +1. *Path-scopable* — any rule whose scope is a file type or directory. Ships + immediately, no trial, no detectors. =docs-lifecycle.md= is the obvious next + one (=docs/**=), and parts of =working-files.md= may qualify. +2. *Semantic* — rules whose condition can't be written as a glob ("any spec + with a non-trivial UI", "when a commit is in play"). These still need the + skills route, and they are the only place the pilot's detectors and stop + conditions apply. + +=commits.md= is the case that matters: 12,800 tokens, the single largest item, +and almost all of it is publish machinery that only applies when a commit is in +play. That is a task scope rather than a path scope, so it is the skills route +and the real test of the risky tier. + +** Correction carried from the live numbers + +Earlier phases in this document quote word counts converted at a guessed ratio +and understate by about 45%. The real ratio is 2.28 tokens per word. Read the +targets below as token figures needing that correction, and see proposals.org +for the corrected table. + +** A pattern worth designing around + +Two mechanical guards failed in the same day, both mine, both reporting success +while doing damage: =wrap-org-table.el= reflowed a table into a worse shape and +=lint-org= then certified it clean, and the wrap-teardown hook consumed a +two-hour-old sentinel and killed a live work session. + +This plan leans on mechanical detectors precisely because they have no stake in +the outcome. Both incidents say that is necessary but not sufficient — a +detector can be confidently wrong. Whatever Phase 3 decides, the verification +step should include "did the guard's own claim get checked," not just "did the +guard report clean." + +* Phase 0 — Free wins [DONE except /doctor] *Scope.* Three items that interact with nothing. @@ -58,7 +124,7 @@ filter step. =/doctor= output read and reconciled against this schedule. *Rollback.* Single revert; nothing downstream depends on it. -* Phase 1 — The pilot migration (next working session) +* Phase 1 — The pilot migration [deterministic half DONE; semantic half pending] *Scope.* Six rule files move from always-loaded to on-demand skills. Roughly 3,000 words, about 12% of the rules surface. @@ -243,14 +309,13 @@ rather than early. - *D6 — Which practices you actually want.* I have low confidence on the quiz fitting how you work, and medium-high on the rest. -* Phase 8 — Effort calibration (independent, any time after Phase 2) +* Phase 8 — Effort calibration [DROPPED 2026-07-27] *Scope.* Set effort levels for the unattended loops: sentry's hourly fires, work-the-backlog, the no-approvals speedrun. -*Reasoning.* Independent of everything else and safe to run in parallel. The -guide says to sweep effort on your own evals rather than trusting carried-over -defaults; we have no evals, so this is a trial and a judgment. +*Dropped.* Buys tokens by spending quality, which inverts the stated goal. +D7 is withdrawn with it. *Decision needed.* |
