diff options
Diffstat (limited to 'working')
| -rw-r--r-- | working/context-engineering-rightsizing/metrics.org | 54 | ||||
| -rw-r--r-- | working/context-engineering-rightsizing/proposals.org | 99 | ||||
| -rw-r--r-- | working/context-engineering-rightsizing/rollout.org | 77 |
3 files changed, 185 insertions, 45 deletions
diff --git a/working/context-engineering-rightsizing/metrics.org b/working/context-engineering-rightsizing/metrics.org index 9cced4b..e7ff189 100644 --- a/working/context-engineering-rightsizing/metrics.org +++ b/working/context-engineering-rightsizing/metrics.org @@ -41,13 +41,36 @@ an afternoon instead of a week. *** T2 — Progressive disclosure preserves behavior (context-engineering post) -*Claim.* Moving guidance out of always-loaded context into on-demand skills +*Claim.* Moving guidance out of always-loaded context into on-demand loading doesn't degrade adherence. -*Test.* The Phase 1 pilot. Detailed in Part 2. +*ANSWERED 2026-07-27 for the deterministic half.* =/context= in a live work +session lists 17 generic rules under Memory files, with the three path-scoped +ones absent. Path-scoping is a glob match rather than a model judgment, so it +either fires or doesn't, and it fires. That half needs no trial and no miss-rate +metric. -*Metric.* Miss rate per exposure. Mechanical, since four of the six pilot rules -have a checker. +*Still open for the semantic half.* Rules whose condition can't be a glob +("when a commit is in play") route to skills, where triggering *is* a model +judgment. Everything in Part 2 applies there and only there. + +*A caution the answer doesn't cover.* Path-scoping fires when Claude *reads* a +matching file. A session that writes an org file without reading one first +never triggers =todo-format.md=. Edit requires a prior read, so edits to +existing files are safe; creating a new org file from scratch is the gap. Worth +watching rather than blocking on. + +*** T3a — The token baseline was wrong by 45% [SETTLED] + +Word counts converted at a guessed ~1.3 tokens per word. The live number is +2.28. =commits.md= is 12,800 tokens, not the ~7,000 estimated, and +=claude-rules/= was ~57,800 tokens per session before today rather than the +~33,000 implied. + +The lesson is narrower than "measure better." I had a real measurement +available the whole time — =/context= reports per-file token counts — and used +an estimate instead because the estimate was easier to compute from inside the +repo. Reach for the instrument that reports the actual quantity. *** T3 — Deliverables run long without explicit calibration (Opus 5 guide) @@ -63,7 +86,13 @@ call on whether anything useful was lost, which is the part I can't measure. *Baseline worth taking now,* since it costs one command and the before-number disappears the moment we change anything. -*** T4 — Lower effort holds quality on mechanical passes (Opus 5 guide) +*** T4 — Lower effort holds quality [WITHDRAWN 2026-07-27] + +Dropped with P4. The goal is output quality first, and this claim trades +quality for cost, so testing it would answer a question we've decided not to +act on either way. + +*** T4 (original text, retained for the record) *Claim.* =low= and =medium= produce strong quality at a fraction of the tokens. @@ -212,6 +241,21 @@ Cheap now, impossible to reconstruct later: Item 1 is done. The rest are one session's work and should happen before Phase 0 changes the review skill, since that change contaminates item 4. +* Instrument reliability — the day's second lesson + +The plan weights mechanical detectors over self-report because they have no +stake. Two failed on 2026-07-27, both reporting success while doing damage: +=wrap-org-table.el= reflowed a table into a worse shape and =lint-org= then +certified it clean, and the wrap-teardown hook consumed a two-hour-old sentinel +and killed a live work session that had done nothing wrong. + +Neither invalidates the preference for mechanical detectors, which is still +right. Both narrow the claim: a detector has no stake, but it can be +confidently wrong, and a green check from a guard that never looked at the +thing is indistinguishable from a green check that did. When a detector clears +a moved rule, the useful question is whether it actually evaluated it, not just +whether it reported clean. + * What this can and cannot tell us It can tell us whether on-demand loading fires reliably here, whether diff --git a/working/context-engineering-rightsizing/proposals.org b/working/context-engineering-rightsizing/proposals.org index 40bfcf8..1300a7b 100644 --- a/working/context-engineering-rightsizing/proposals.org +++ b/working/context-engineering-rightsizing/proposals.org @@ -8,33 +8,61 @@ Proposals only. Nothing here is applied. Source: three Anthropic posts Craig supplied 2026-07-27 — the Claude 5 context-engineering post (2026-07-24), the Opus 5 prompting guide, and the Fable 5 field guide (2026-07-06). -* The measurement - -Every session in every project loads this before the user's first word: - -| Surface | Words | -|--------------------------+--------| -| =claude-rules/= (20 files) | 25,386 | -|--------------------------+--------| -| =protocols.org= | 6,620 | -|--------------------------+--------| -| Project =CLAUDE.md= | 117 | -|--------------------------+--------| -| *Total* | 32,123 | -|--------------------------+--------| - -Roughly 40,000 tokens of always-on context. The context-engineering post reports -Anthropic cut over 80% of Claude Code's *system prompt* with no measurable loss -on their coding evals. That's their number on their surface, not a target for -this one, but the direction is the finding: this system is built in the "then" -style the post retired. - -The content isn't the problem. The post says skills are best when they "encode -particular opinions, knowledge, or best practices particular to you" — which is -exactly what these rules are. The problem is *load timing*: all of it is -resident on every turn, including =commits.md= (5,561 words) in a session that -never commits, and =todo-format.md= (4,494 words) in a session that never opens -an org file. +* Goal + +Output quality and results first. Token reduction is a real goal and worth +having, but it is the second one, and where the two conflict quality wins. +Craig's framing, 2026-07-27: the concern is agents having the freedom to +produce at their highest capacity, unbound by guardrails that constrain them or +work against them. + +That reordering matters more than it sounds. Anthropic's 80% figure was a +*finding* — they cut and quality held — not a target. Aimed at directly it +optimizes the thing we don't care about, and P4 below is the proposal that +falls to it. + +* The measurement (corrected 2026-07-27 from live =/context=) + +The original figures in this document were word counts converted at a guessed +ratio, and they understated by about 45%. A =/context= run in a live work +session gave the real numbers: + +| Surface | Tokens | +|------------------------------------+--------| +| =claude-rules/= before this session | 57,800 | +|------------------------------------+--------| +| =claude-rules/= now (17 files) | 44,410 | +|------------------------------------+--------| +| Path-scoped out (3 files) | 13,390 | +|------------------------------------+--------| + +The real ratio is 2.28 tokens per word, not the ~1.3 assumed. =commits.md= +alone is 12,800 tokens, not the ~7,000 estimated. Every earlier figure here +should be read as low. + +** Two loading paths, not one + +The original framing called 32,123 words "always-loaded" and was wrong to +lump them. They arrive by different mechanisms: + +- *Memory files* — =claude-rules/= (via the =~/.claude/rules/= symlinks), + =CLAUDE.md=, project =.claude/rules/=, and auto-memory's =MEMORY.md=. Loaded + at session start by the harness. This is the 44,410 above, plus CLAUDE.md and + project rules. +- *Read during startup* — =protocols.org= and the workflow files. These never + appear under Memory files in =/context=; the startup workflow reads them, so + they land in Messages. Still a real per-session cost, but a different lever: + they shrink by editing the workflow, not by scoping a rule. + +Conflating the two made =protocols.org= look like it competed with +=commits.md= for the same fix. It doesn't. + +** What the harness says on its own + +=/context= ends with its own suggestion: prune =commits.md= (12.8k), +=testing.md= (6.3k), and =MEMORY.md= (5.5k). That is the Phase 4 target list, +arrived at independently. Worth treating as corroboration rather than +coincidence. * Proposals @@ -106,7 +134,7 @@ always-loaded core — otherwise this proposal fights the one above it. *Confidence.* Medium-high on the practices being useful. Lower on the quiz, which may not fit how you actually work. -** P4 — Effort calibration on the unattended loops [medium value, low risk] +** P4 — Effort calibration [DROPPED 2026-07-27 — trades quality for cost] *Change.* Sentry, work-the-backlog, and the no-approvals speedrun run many passes at default effort. The Opus 5 guide says to use =low= and =medium= @@ -116,11 +144,11 @@ for demanding work. *Why.* Sentry fires hourly. Effort is the lever with the largest cost multiple, and most sentry passes are mechanical sweeps. -*Confidence.* Medium. The guide says to re-run an effort sweep on your own -evals rather than trusting carried-over defaults, and we have no evals — so -this wants a trial and a judgment, not a blanket change. +*Dropped.* Lowering effort buys tokens by spending quality, which is exactly +backwards under the goal above. Revisit only if a specific pass proves +genuinely mechanical and a cost problem shows up on its own. -** P5 — Positive framing over prohibition [low value, low risk, slow] +** P5 — Positive framing over prohibition [PROMOTED — now the top quality lever] *Change.* =commits.md= carries 41 prohibition markers (NEVER / DO NOT / MANDATORY / CRITICAL) across 5,561 words. =todo-format.md= 21, @@ -133,8 +161,11 @@ tend to be more effective than instructions about what not to do." *Keep as prohibitions:* the AI-attribution rule and anything else where the worst case is genuinely unacceptable. Those earn their emphasis. -*Confidence.* Medium. Real but diffuse, and it competes with P1 for the same -editing effort. Probably rides along with P1 rather than standing alone. +*Promoted 2026-07-27.* Ranked low when the score was token savings. Under a +quality-first goal this is the proposal that most directly targets guardrails +working against good output, which is the stated concern. It still rides along +with the Phase 4 edits rather than running as its own campaign, because every +file it touches is a file those phases open anyway. ** P6 — Deduplicate the two always-loaded surfaces [medium value, low risk] diff --git a/working/context-engineering-rightsizing/rollout.org b/working/context-engineering-rightsizing/rollout.org index 4afc27f..d641ce7 100644 --- a/working/context-engineering-rightsizing/rollout.org +++ b/working/context-engineering-rightsizing/rollout.org @@ -34,7 +34,73 @@ Two consequences: This is why the posts' value here is subtractive, not additive. -* Phase 0 — Free wins (this session, no dependencies) +* Status — 2026-07-27, end of first working session + +Phase 0 and the deterministic half of Phase 1 are done and verified in a live +session. The pilot's central question is answered, which changes what remains. + +** Shipped + +- =paths:= frontmatter on the three rules that already declared a file-type + scope in prose (=todo-format=, =org-tables=, =emacs=), plus a =lint.sh= + checker that catches the prose/frontmatter mismatch, plus a heading check + taught to skip frontmatter. Commit 0adcb1a. +- Generic rules no longer ship per project. =install-lang= stopped copying them + and =sync-language-bundle= sweeps what earlier installs left, guarded on the + global rule existing. Swept 20 files each from work and =.emacs.d=. Commit + 7ea1d7b. +- The live session anchor is gitignored, so rulesets stops reporting + sync-blocked for the whole of every session. Same commit. + +** Verified in a live work session + +=/context= lists 17 generic rules under Memory files. =todo-format.md=, +=org-tables.md=, and =emacs.md= are absent, and only =python-testing.md= and +=publishing.md= come from the project's own rules directory. + +So *path-scoping works at user level* and *the de-duplication holds*. Both were +open questions this morning. + +** What that changes + +Path-scoping is a glob match, not a model judgment. It is deterministic, so the +silent-miss risk the whole pilot was designed around does not apply to it. That +splits the remaining work in two: + +1. *Path-scopable* — any rule whose scope is a file type or directory. Ships + immediately, no trial, no detectors. =docs-lifecycle.md= is the obvious next + one (=docs/**=), and parts of =working-files.md= may qualify. +2. *Semantic* — rules whose condition can't be written as a glob ("any spec + with a non-trivial UI", "when a commit is in play"). These still need the + skills route, and they are the only place the pilot's detectors and stop + conditions apply. + +=commits.md= is the case that matters: 12,800 tokens, the single largest item, +and almost all of it is publish machinery that only applies when a commit is in +play. That is a task scope rather than a path scope, so it is the skills route +and the real test of the risky tier. + +** Correction carried from the live numbers + +Earlier phases in this document quote word counts converted at a guessed ratio +and understate by about 45%. The real ratio is 2.28 tokens per word. Read the +targets below as token figures needing that correction, and see proposals.org +for the corrected table. + +** A pattern worth designing around + +Two mechanical guards failed in the same day, both mine, both reporting success +while doing damage: =wrap-org-table.el= reflowed a table into a worse shape and +=lint-org= then certified it clean, and the wrap-teardown hook consumed a +two-hour-old sentinel and killed a live work session. + +This plan leans on mechanical detectors precisely because they have no stake in +the outcome. Both incidents say that is necessary but not sufficient — a +detector can be confidently wrong. Whatever Phase 3 decides, the verification +step should include "did the guard's own claim get checked," not just "did the +guard report clean." + +* Phase 0 — Free wins [DONE except /doctor] *Scope.* Three items that interact with nothing. @@ -58,7 +124,7 @@ filter step. =/doctor= output read and reconciled against this schedule. *Rollback.* Single revert; nothing downstream depends on it. -* Phase 1 — The pilot migration (next working session) +* Phase 1 — The pilot migration [deterministic half DONE; semantic half pending] *Scope.* Six rule files move from always-loaded to on-demand skills. Roughly 3,000 words, about 12% of the rules surface. @@ -243,14 +309,13 @@ rather than early. - *D6 — Which practices you actually want.* I have low confidence on the quiz fitting how you work, and medium-high on the rest. -* Phase 8 — Effort calibration (independent, any time after Phase 2) +* Phase 8 — Effort calibration [DROPPED 2026-07-27] *Scope.* Set effort levels for the unattended loops: sentry's hourly fires, work-the-backlog, the no-approvals speedrun. -*Reasoning.* Independent of everything else and safe to run in parallel. The -guide says to sweep effort on your own evals rather than trusting carried-over -defaults; we have no evals, so this is a trial and a judgment. +*Dropped.* Buys tokens by spending quality, which inverts the stated goal. +D7 is withdrawn with it. *Decision needed.* |