aboutsummaryrefslogtreecommitdiff
path: root/working/context-engineering-rightsizing/metrics.org
diff options
context:
space:
mode:
authorCraig Jennings <c@cjennings.net>2026-07-27 13:46:32 -0500
committerCraig Jennings <c@cjennings.net>2026-07-27 13:46:32 -0500
commitd74d98d03e7321b6c69dae9ef427da66b0f6fbc6 (patch)
tree4d61b604085afc019b40903a9e8db9072b46147c /working/context-engineering-rightsizing/metrics.org
parent2c664cb5651fbf03bc52d7848add0c571708adcd (diff)
downloadrulesets-d74d98d03e7321b6c69dae9ef427da66b0f6fbc6.tar.gz
rulesets-d74d98d03e7321b6c69dae9ef427da66b0f6fbc6.zip
docs: correct the rightsizing docs against live measurements
I had the goal backwards in the lead. Craig's framing is output quality first, with token reduction real but second, and where they conflict quality wins. Anthropic's 80% was a finding rather than a target, and aiming at it directly optimizes the thing we don't care about. P4 falls to that and is dropped: lowering effort buys tokens by spending quality. P5 rises for the same reason, since positive framing over prohibition is what actually targets guardrails working against output. My token figures were low by about 45%. I converted word counts at a guessed ratio when /context reports real per-file numbers, so I used an estimate because it was easier to compute from inside the repo. The live ratio is 2.28 tokens per word. commits.md is 12,800 tokens rather than the 7,000 I estimated, and claude-rules was near 57,800 per session before today. I also conflated two loading paths. Memory files come in through the harness at session start. protocols.org and the workflows are read by startup and land in Messages, so they never appear under Memory files at all. They shrink by editing the workflow, not by scoping a rule, and treating them as one surface made protocols.org look like it competed with commits.md for the same fix. Path-scoping is confirmed working at user level, and the de-duplication holds. That splits the remaining work: anything scopable to a file type or directory ships with no trial, because a glob match is deterministic. Only the semantic rules need the skills route and the stop conditions, and commits.md is the one that matters there at 12,800 tokens of publish machinery that only applies when a commit is in play. I recorded one caution the confirmation doesn't cover. Path-scoping fires when a matching file is read, so creating a new org file from scratch never triggers todo-format.md. Edits are safe because Edit requires a prior read. I also wrote up the day's two guard failures, both mine. wrap-org-table reflowed a table into a worse shape and lint-org certified it clean, and the teardown hook consumed a stale sentinel and killed a live work session. The plan leans on mechanical detectors because they have no stake, which is still right, but a detector can be confidently wrong and a green check from a guard that never looked is indistinguishable from one that did.
Diffstat (limited to 'working/context-engineering-rightsizing/metrics.org')
-rw-r--r--working/context-engineering-rightsizing/metrics.org54
1 files changed, 49 insertions, 5 deletions
diff --git a/working/context-engineering-rightsizing/metrics.org b/working/context-engineering-rightsizing/metrics.org
index 9cced4b..e7ff189 100644
--- a/working/context-engineering-rightsizing/metrics.org
+++ b/working/context-engineering-rightsizing/metrics.org
@@ -41,13 +41,36 @@ an afternoon instead of a week.
*** T2 — Progressive disclosure preserves behavior (context-engineering post)
-*Claim.* Moving guidance out of always-loaded context into on-demand skills
+*Claim.* Moving guidance out of always-loaded context into on-demand loading
doesn't degrade adherence.
-*Test.* The Phase 1 pilot. Detailed in Part 2.
+*ANSWERED 2026-07-27 for the deterministic half.* =/context= in a live work
+session lists 17 generic rules under Memory files, with the three path-scoped
+ones absent. Path-scoping is a glob match rather than a model judgment, so it
+either fires or doesn't, and it fires. That half needs no trial and no miss-rate
+metric.
-*Metric.* Miss rate per exposure. Mechanical, since four of the six pilot rules
-have a checker.
+*Still open for the semantic half.* Rules whose condition can't be a glob
+("when a commit is in play") route to skills, where triggering *is* a model
+judgment. Everything in Part 2 applies there and only there.
+
+*A caution the answer doesn't cover.* Path-scoping fires when Claude *reads* a
+matching file. A session that writes an org file without reading one first
+never triggers =todo-format.md=. Edit requires a prior read, so edits to
+existing files are safe; creating a new org file from scratch is the gap. Worth
+watching rather than blocking on.
+
+*** T3a — The token baseline was wrong by 45% [SETTLED]
+
+Word counts converted at a guessed ~1.3 tokens per word. The live number is
+2.28. =commits.md= is 12,800 tokens, not the ~7,000 estimated, and
+=claude-rules/= was ~57,800 tokens per session before today rather than the
+~33,000 implied.
+
+The lesson is narrower than "measure better." I had a real measurement
+available the whole time — =/context= reports per-file token counts — and used
+an estimate instead because the estimate was easier to compute from inside the
+repo. Reach for the instrument that reports the actual quantity.
*** T3 — Deliverables run long without explicit calibration (Opus 5 guide)
@@ -63,7 +86,13 @@ call on whether anything useful was lost, which is the part I can't measure.
*Baseline worth taking now,* since it costs one command and the before-number
disappears the moment we change anything.
-*** T4 — Lower effort holds quality on mechanical passes (Opus 5 guide)
+*** T4 — Lower effort holds quality [WITHDRAWN 2026-07-27]
+
+Dropped with P4. The goal is output quality first, and this claim trades
+quality for cost, so testing it would answer a question we've decided not to
+act on either way.
+
+*** T4 (original text, retained for the record)
*Claim.* =low= and =medium= produce strong quality at a fraction of the tokens.
@@ -212,6 +241,21 @@ Cheap now, impossible to reconstruct later:
Item 1 is done. The rest are one session's work and should happen before
Phase 0 changes the review skill, since that change contaminates item 4.
+* Instrument reliability — the day's second lesson
+
+The plan weights mechanical detectors over self-report because they have no
+stake. Two failed on 2026-07-27, both reporting success while doing damage:
+=wrap-org-table.el= reflowed a table into a worse shape and =lint-org= then
+certified it clean, and the wrap-teardown hook consumed a two-hour-old sentinel
+and killed a live work session that had done nothing wrong.
+
+Neither invalidates the preference for mechanical detectors, which is still
+right. Both narrow the claim: a detector has no stake, but it can be
+confidently wrong, and a green check from a guard that never looked at the
+thing is indistinguishable from a green check that did. When a detector clears
+a moved rule, the useful question is whether it actually evaluated it, not just
+whether it reported clean.
+
* What this can and cannot tell us
It can tell us whether on-demand loading fires reliably here, whether