diff options
| author | Craig Jennings <c@cjennings.net> | 2026-07-13 20:17:53 -0500 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2026-07-13 20:17:53 -0500 |
| commit | 21d14e0752fa8438a7ea299402e6680c3bc25062 (patch) | |
| tree | cb1b9e7dc373a27d03f15101d45a19aedd0af8d8 | |
| parent | 49be354ff18d6739e478ee4f5ab46de6cb9e4f81 (diff) | |
| download | rulesets-21d14e0752fa8438a7ea299402e6680c3bc25062.tar.gz rulesets-21d14e0752fa8438a7ea299402e6680c3bc25062.zip | |
chore(todo): record the verified local-inference stack on the model-floor task
ollama 0.31.2 with the Vulkan backend forced by systemd override (ROCm first-touch loads were unusable at under 25 MB/s), gpt-oss:120b and qwen3-coder:30b verified at 37.9 and 76 tok/s on 100% GPU, four legacy dense models dropped, the orphaned user store purged. The child's remaining work is purely the instruction-following evaluation.
| -rw-r--r-- | todo.org | 4 |
1 files changed, 3 insertions, 1 deletions
@@ -314,7 +314,9 @@ Recorded in [[file:docs/design/2026-07-13-runtime-portability-inventories.org][t *** TODO Local model floor evaluation The workflows assume long-context instruction-following (startup's multi-file read; the commits.md publish chain). Establish the minimum viable local tier (likely strong-70B+/MoE, 100k+ context), and what compensations a weaker model needs: shortened protocols, more checklist gates, more hook-level enforcement. Feeds the spec's "default local runtime/server" and "first supported local editing CLI" blocker decisions. -Environment inventory done 2026-07-13 (KB node "Local LLM inference inventory — daily drivers"): ratio is the inference box — Strix Halo iGPU + 125 GiB unified RAM, ollama 0.17.7 on demand (no systemd unit), llama3.1:8b + llama3.3 70B already in the 45 GB store. velox is out of scope (Iris Xe, 60 GiB, no ollama). So the floor evaluation can run today on ratio against the on-disk 70B: drive a scripted startup + a small publish-flow transcript through it and grade instruction-following. Remaining work is that evaluation, not procurement. +Environment inventory done 2026-07-13 (KB node "Local LLM inference inventory — daily drivers"): ratio is the inference box — Strix Halo iGPU + 125 GiB unified RAM. velox is out of scope (Iris Xe, 60 GiB, no ollama). + +Environment BUILT and verified, evening of 2026-07-13: ollama 0.31.2 as a systemd service, Vulkan/RADV backend forced via systemd override (ROCm loaded a 61 GB model at <25 MB/s — 40+ min, unusable; Vulkan loads it in 44 s), models swapped to the MoE shape this platform wants: gpt-oss:120b (44 s load, 37.9 tok/s, 100% GPU) and qwen3-coder:30b (5 s load, 76 tok/s). All four legacy dense models dropped; the orphaned ~/.ollama user store purged. Gotcha for the eval harness: pass num_ctx 8-32K — gpt-oss's native 128K default balloons the load. Remaining work is now purely the evaluation: drive a scripted startup read + a publish-flow transcript through gpt-oss:120b (and qwen3-coder as the fast comparator), grade instruction-following against the protocol stack, and answer the spec's default-local-runtime + first-supported-CLI blockers. ** TODO [#C] Docs-lifecycle convention — manual validation :test: The human-eyes half of the docs-lifecycle acceptance surface. The convention shipped IMPLEMENTED 2026-07-04 (spec [[id:80b0787b-4a60-4c82-8a16-b383d3e3c8f2][docs-lifecycle]]); these checks confirm the human-facing behavior. A failed check promotes to a bug. |
