path: root/docs/todo-keywords-from-workflow-states-spec.org

#+TITLE: pearl — Derive Org TODO Keywords from Linear Workflow States Spec
#+AUTHOR: Craig Jennings
#+DATE: 2026-05-24
#+STARTUP: showall

* Status

READY — implementation-ready, awaiting Craig's final go. Reviews incorporated (2026-05-24, rounds 1–4); round 4 returned a Ready verdict with no blocking findings. Implements the =todo.org= task "Derive the org TODO keywords from the Linear workflow states". One implementation prerequisite remains (verify =WorkflowState.position=; see prerequisites).

* Problem

The generated file's =#+TODO:= line is fixed — either the hardcoded =TODO IN-PROGRESS IN-REVIEW BACKLOG BLOCKED | DONE= or a copy of the user's global =org-todo-keywords= — and =pearl-state-to-todo-mapping= is a static six-entry default. Neither reflects a team's real Linear states (Dev Review, PM Acceptance, Icebox, Grooming, …).

Org only cycles a heading to a keyword listed in the file's =#+TODO:=. So you cannot move a ticket to "Dev Review" by cycling its TODO keyword: the keyword isn't in the line, and there's no mapping entry for it. =pearl-set-state= already reaches any state (id-based, header-independent), but the keyword-cycle path — the natural org gesture — is stuck on the hardcoded six.

The =#+TODO:= line therefore becomes *generated infrastructure* for the sync-back write path. If it is missing a rendered keyword, stale after a merge refresh, or ambiguous after slugification, org-native state changes silently stop being a trustworthy write path. The header-and-reverse-lookup contract below is written to that bar.

* Current state

- =pearl--build-org-content= writes =#+TODO:= from =org-todo-keywords= (or the hardcoded fallback). It's a pure function.
- =pearl--map-linear-state-to-org= renders an issue's keyword by =assoc= on =pearl-state-to-todo-mapping= (fallback =TODO=).
- =pearl--map-org-state-to-linear= resolves a cycled keyword back to a Linear state *name* by =rassoc= on the same mapping; =pearl--process-heading-at-point= then calls =pearl--update-issue-state-async= with that name + team id.
- =pearl--get-todo-states-pattern= builds the full-file scan regex from the mapping's keywords (cached in =pearl-todo-states-pattern=).
- =pearl--team-states= is a *synchronous, cached* accessor (blocks via =pearl--wait-for= on first fetch, then serves from =pearl--cache-states=). =pearl-set-state= already calls it synchronously from command context.
- The team-states GraphQL query fetches =id name color= only. Issue queries already fetch state =type= (but not workflow-state =position=).
- The same-source refresh path (=pearl--merge-query-result= → =pearl--merge-issues-into-buffer=) updates issue subtrees in place and only rewrites the run-at / count / truncation header lines via =pearl--update-source-header=. It does *not* rebuild the file, so it does not touch =#+TODO:= today.

* Decisions

Settled inputs for v1 (A1 / B2 / C-yes agreed with Craig 2026-05-24; the remainder resolved from review):

- *A1 — union.* A file may hold issues from several teams. Build one =#+TODO:= line from the union of all involved teams' states, de-duplicated by slug.
- *B2 — derived replaces.* The keyword is always the slugified Linear state name. =pearl-state-to-todo-mapping= is *removed*, not layered. One source of truth (Linear), an honest header, a deterministic round-trip (keyword = =slugify(name)= everywhere, no stored reverse map).
- *C-yes — defer cross-team slug collisions.* Documented known limitation (see out of scope).
- *Done-side types.* =completed=, =canceled=, *and* =duplicate= render after the =|=; =triage=, =backlog=, =unstarted=, =started= before it. Split by =type=, never by name.
- *Header coverage guarantee.* Every keyword visible on a heading in the buffer must be declared in =#+TODO:= — that header powers org cycling and sync-back. The header is the slug-union of (a) every visible issue heading's state and (b) every team's full state set that was fetched successfully. (a) guarantees coverage even when a team's state fetch fails; (b) makes absent states cyclable when available. A failed team's only degradation is that you can't cycle to a state none of its visible headings is in. The hardcoded line is used only when there are no states at all. The "visible headings" set differs by path: a full rebuild reads the normalized issue list; a merge refresh reads the *final buffer* (so retained/skipped dirty subtrees are covered — see below).
- *Merge coverage via final-buffer scan.* For merge refresh the header is rebuilt from the final displayed buffer (scan every Linear issue heading's current TODO keyword) unioned with the fetched team states — not from the fetched issue list alone, which omits retained dirty subtrees. The buffer scan subsumes the fetched issues (they're in the buffer) and directly validates the invariant users see.
- *Issue-own / position-less ordering.* Full workflow states (from team-states) order by =position=; states drawn from headings or the issue list carry no workflow =position=, so they append in first-seen order within their active/done partition, de-duped by slug.
- *Slugify is Unicode-aware and locale-independent.*
- *Same-team slug collision.* The pure gather/derive layer returns collision metadata (the colliding slug + states); the render/sync layer logs it. The header de-dups; sync-back resolves the keyword to the *first state by =position=*. Returning the metadata keeps the lossy transform testable without capturing =message=.
- *Unknown keyword behavior splits by path.* Interactive current-heading sync (an =org-todo= cycle) whose keyword resolves to no team state raises a =user-error= naming the keyword + team. The full-file save scan reports and *skips* the unknown heading and continues to the rest — one stale heading must not abort syncing the others or surface as an after-save-hook error.
- *Store the state type in the drawer.* Render a =:LINEAR-STATE-TYPE:= drawer field (the issue query already fetches state =type=). The active/done side of a keyword is a function of =type=, and the merge final-buffer scan recovers a retained heading's keyword but not its type from name/id alone — so the type must travel with the heading. Classification on the merge scan: by the heading's own =:LINEAR-STATE-TYPE:= when present (deterministic); else (a legacy heading written before this field) preserve the keyword's current side from the buffer's parsed TODO config — done side if the keyword is in =org-done-keywords=, active otherwise; else (no parseable =#+TODO:= / keyword unknown to Org) default to the active side and log a warning naming the keyword and issue. Retained headings thus keep their old Org done/active semantics until a clean refresh re-derives them from Linear.

* Proposed design

** Slugify: state name → org keyword

A new pure helper =pearl--state-name-to-keyword=:

- Upcase (locale-independent — Emacs =upcase=, no locale-sensitive casing).
- Replace each run of characters *not* matched by =[[:alnum:]]= with a single hyphen. =[[:alnum:]]= is Unicode-aware in Emacs, so accented and non-Latin letters are preserved rather than stripped.
- Trim leading/trailing hyphens.
- If the result is empty (an all-punctuation/symbol name), fall back to =TODO=.

Expected outputs:

| Input              | Output            |
|--------------------+-------------------|
| =Dev Review=       | =DEV-REVIEW=      |
| =In Progress=      | =IN-PROGRESS=     |
| =Todo=             | =TODO=            |
| =PM Acceptance=    | =PM-ACCEPTANCE=   |
| =Backlog (prioritized)= | =BACKLOG-PRIORITIZED= |
| =Dev-Review=       | =DEV-REVIEW=      |
| =Ångström=         | =ÅNGSTRÖM=        |
| =!!!=              | =TODO= (empty fallback) |

Note =Dev Review= and =Dev-Review= both produce =DEV-REVIEW= — a same-team collision (handled below). The existing default mapping was effectively slugify already (=Todo=→=TODO=, =In Progress=→=IN-PROGRESS=, …), so slugify reproduces today's keywords for those states.

** Same-team slug collisions

When two states in one team slugify to the same keyword (=Dev Review= and =Dev-Review=), the =#+TODO:= line lists the keyword once (de-dup, first-seen by =position= wins its slot). Sync-back resolves that keyword to the *first state by =position=* and logs a one-line warning naming the colliding states, so the behavior is deterministic and visible. Cross-team collisions are deferred (out of scope) — sync still resolves correctly per the heading's own team, but the header can't distinguish them.

** Derive the =#+TODO:= line

A new pure helper =pearl--derive-todo-line= takes an ordered list of states (each =(:name :type :position)=) and returns the keyword string:

- Partition by =type=: done-side = =completed=/=canceled=/=duplicate=; active-side = everything else.
- Within each side, preserve the input order (the caller supplies states already ordered — see Multi-team ordering).
- Slugify each name; de-duplicate by slug, preserving first-seen order.
- Result: ="ACTIVE-1 ACTIVE-2 … | DONE-1 DONE-2 …"=.

When the state list is empty, return the hardcoded =TODO IN-PROGRESS IN-REVIEW BACKLOG BLOCKED | DONE= so the file is always valid.

** Multi-team ordering

=position= is meaningful within a team, not across teams. To keep the header deterministic:

1. Teams are ordered by *first-seen order in the sorted issue list* (the issues are already sorted before render).
2. Within each team, states are ordered by Linear =position=.
3. The union concatenates teams in that order, then de-dups by slug (first-seen wins), then the derive-line partitions by type.

So the header order is stable across runs regardless of hash/traversal order.

** Gather the states (pipeline)

Because =pearl--team-states= is synchronous-with-cache and =pearl-set-state= already calls it from command context, the render path gathers states synchronously without restructuring into async callbacks:

1. After issues are normalized and sorted, collect the distinct =LINEAR-TEAM-ID= values in first-seen order.
2. For each team, call =pearl--team-states= (cached after the first hit). Show a progress message while a fetch blocks; on a team's fetch failure, log it and continue (per the coverage guarantee).
3. Build the union: every displayed issue's own state, plus the full states of each team that fetched. Order per Multi-team ordering, de-dup by slug.
4. Hand the union to =pearl--build-org-content=.

=pearl--build-org-content= stays pure: it gains a =states= argument (the ordered union list) and writes the derived =#+TODO:= via =pearl--derive-todo-line=. The async layer does the synchronous gather just before calling it. One synchronous, cached team-states fetch per distinct team per session — usually one or two; multi-team views do N bounded blocking fetches.

The team-states query gains =type= and =position= (currently =id name color=), and the cache entry keeps them.

** Render each issue's keyword

=pearl--format-issue-as-org-entry= renders the heading keyword as =pearl--state-name-to-keyword(issue-state-name)= instead of =pearl--map-linear-state-to-org=. Safe because the header always includes each displayed issue's own state (coverage guarantee). It also writes a =:LINEAR-STATE-TYPE:= drawer field next to =:LINEAR-STATE-ID:= / =:LINEAR-STATE-NAME:=, so a later merge scan can classify a retained heading onto the correct side of the =|= from the heading itself (see the classification decision).

** Generated header update on refresh

The same-source merge refresh must keep the header honest: a refresh can add an issue from a team whose state keyword isn't yet declared, or surface a renamed/added/removed state — *and* it retains existing subtrees that the merge skips. =pearl--merge-issues-into-buffer= keeps a dirty existing subtree (unpushed body edits) without re-rendering it, and keeps a dirty issue that's absent from the refreshed result rather than dropping it. Those headings stay visible after the refresh, so their keywords must be declared too.

A new helper =pearl--update-derived-todo-header= rewrites the =#+TODO:= line in place (creating it if absent). It derives from the *final displayed buffer*: scan every Linear issue heading (one carrying =LINEAR-ID=) for its current TODO keyword *and its =:LINEAR-STATE-TYPE:= drawer*, union those with the fetched team states (per the ordering rules), classify each onto the active/done side (by type when known; otherwise the fallback in the classification decision), and rewrite the line. =pearl--merge-query-result= calls it after the merge and the state gather, alongside =pearl--update-source-header=. Scanning the final buffer — rather than building from the fetched issue list — is what guarantees retained/skipped subtrees are covered, and it directly validates the invariant: every keyword visible in the buffer is declared on the correct side of the bar.

** Sync-back: cycled keyword → Linear state

=pearl--process-heading-at-point= resolves the cycled keyword via the heading's team rather than the removed mapping:

1. Read the heading's TODO keyword + =LINEAR-TEAM-ID=.
2. =pearl--team-states= (cached) for that team; find the state whose =slugify(name)= equals the keyword (first by =position= on a collision).
3. If a state matches, push it via =pearl--update-issue-state-async= (unchanged; it resolves name → id per team).
4. If *no* state matches (stale buffer keyword after a workflow change, or an old mapped keyword from a pre-upgrade file), the behavior depends on the path. Interactive current-heading sync (an =org-todo= cycle) raises a =user-error= naming the keyword + team and suggesting a refresh or =pearl-clear-cache=. The full-file save scan (=pearl-sync-org-to-linear=, non-=org-todo= path) reports the unknown heading (a =pearl--log= / message) and *skips* it, continuing to the rest — one stale heading must not abort the scan or fail an after-save hook. Neither path ever silently no-ops or pushes a wrong state.

No persisted reverse map: the keyword is always =slugify(name)=, so the match recomputes from the team's live states.

** The full-file scan pattern

=pearl--get-todo-states-pattern= no longer builds from the mapping. The full-file sync scan (=pearl-sync-org-to-linear=, non-=org-todo= path) builds its keyword alternation from the buffer's own =org-todo-keywords-1= (what Org parsed from =#+TODO:=), so it matches whatever the derived header declared, with no stale cache. The =pearl-todo-states-pattern= / =pearl--todo-states-pattern-source= caches are removed with the mapping.

** User-facing errors

None of these fail silently: an unknown heading keyword on sync-back (=user-error= naming keyword + team), a missing =LINEAR-TEAM-ID= on a heading being synced, a team state fetch that fails during render (logged + progress/skip message), and a same-team slug collision (logged warning). Each names the offending value.

* Implementation prerequisites

- *Verify =WorkflowState.position=* against the current Linear schema or a live query before implementation. Issue queries fetch state =type= today but not workflow-state =position=. If =position= is unavailable, fall back to the order the API returns states in (still deterministic per fetch) and note it.
- Confirm the Linear state =type= enum is =triage/backlog/unstarted/started/completed/canceled/duplicate= (already verified in =docs/issue-query-spec.org=).
- Only =pearl--team-states= gains the =type=/=position= fields. =pearl-get-states-async= / =pearl-get-states= (used by creation flows) fetch =id name color= and are intentionally left unchanged — this feature doesn't depend on them. The two query shapes diverging is acceptable for v1; aligning them is optional follow-up cleanup.

* Phased implementation plan

Ordered so dependencies land first.

1. *Pure core.* =pearl--state-name-to-keyword= and =pearl--derive-todo-line= + their tests (no I/O). Everything else depends on these.
2. *Query + gather.* Add =type=/=position= to the team-states query (after the position verification); add the synchronous gather helper (distinct teams → union with coverage guarantee + multi-team ordering).
3. *Full rebuild.* =pearl--build-org-content= takes =states=, writes the derived header; =pearl--format-issue-as-org-entry= renders via slugify and adds the =:LINEAR-STATE-TYPE:= drawer field. Assert the rebuilt header declares every rendered heading keyword.
4. *Merge refresh.* =pearl--update-derived-todo-header= (final-buffer scan + active/done classification by the heading's =:LINEAR-STATE-TYPE:=, with the org-done-keywords fallback) wired into =pearl--merge-query-result=.
5. *Sync-back.* Team-aware resolve in =pearl--process-heading-at-point= with unknown-keyword refusal; scan pattern from =org-todo-keywords-1=.
6. *Remove the mapping.* Delete =pearl-state-to-todo-mapping=, =pearl--map-linear-state-to-org=, =pearl--map-org-state-to-linear=, =pearl-todo-states-pattern=, =pearl--todo-states-pattern-source=. Replace =test-pearl-mapping.el= with =test-pearl-keywords.el= (keep the regression class: a changed header/keyword set affects full-file scan with no stale cache).
7. *Docs.* README state-mapping section + customization table; migration notes.

* Test plan

- =pearl--state-name-to-keyword=: ASCII names, punctuation, repeated punctuation, high-ASCII/accented, double-byte, combining characters, emoji/symbol-only → =TODO=, empty string, =Dev Review= vs =Dev-Review= collision, a real =Todo= alongside an empty-derived =TODO=.
- =pearl--derive-todo-line=: active/done partition *including =duplicate= after the bar*, per-team =position= ordering, deterministic multi-team order, duplicate-slug first-wins, empty-states fallback.
- *Full rebuild*: =build-org-content= with a state set emits the derived =#+TODO=; the header declares every rendered heading keyword; an issue in =Dev Review= renders the keyword =DEV-REVIEW= on its level-2 heading.
- *Merge refresh*: a same-source refresh where a newly fetched issue introduces =DEV-REVIEW= updates the buffer's =#+TODO:= line (creates it if missing).
- *Merge refresh — retained dirty subtree (absent from result)*: an old dirty issue in =QA-REVIEW= is gone from the refreshed result, is kept by the merge, and the rewritten =#+TODO:= still declares =QA-REVIEW=.
- *Merge refresh — skipped dirty subtree (still in result)*: a dirty issue still present is skipped (not re-rendered), and the rewritten header still declares its current kept keyword even if it differs from the fetched issue's new state.
- *Merge refresh — done-side classification of a type-less retained heading*: a retained dirty heading whose keyword was on the done side, whose team-state fetch fails and which lacks =:LINEAR-STATE-TYPE:= (legacy), keeps its keyword *after* the =|= via the org-done-keywords fallback. Same setup for an active retained heading keeps it *before* the bar. A missing/unparseable old header defaults the unknown keyword to active and logs the ambiguity.
- *Drawer carries the type*: a freshly rendered issue's drawer includes =:LINEAR-STATE-TYPE:=, and a merge scan classifies it by that field without the fallback.
- *Partial failure*: a multi-team result where one team's state fetch fails still declares every rendered keyword (issue-own states in the header) and renders without error; the two issue-own fallback states from the failed team order deterministically (first-seen within partition).
- *Sync-back*: =DEV-REVIEW= resolves through the heading's team states; a same-team collision resolves to first-by-position.
- *Unknown keyword by path*: interactive current-heading sync of an unknown keyword raises =user-error=; the full-file scan reports and skips it and still syncs the other headings.
- *Collision metadata*: the pure gather/derive layer returns the colliding slug + states (asserted directly, no =message= capture).
- *Migration/regression*: an old mapped keyword no longer in the derived header does not silently push a wrong state (interactive refuses; scan skips).
- *Scan pattern*: the full-file scan matches a derived keyword present in the buffer's =#+TODO= with no stale-cache dependency.

* Migration / breaking change

Removing =pearl-state-to-todo-mapping= is a breaking change for anyone who set it. The package is pre-release (MELPA pending), so no deprecation cycle. The commit is =feat!:= with a =BREAKING CHANGE:= footer.

*Upgrade path:* after upgrading, *refresh a Pearl file before cycling TODO keywords on it.* An old file's header and headings may use custom-mapped keywords that no longer resolve; cycling one of those now *refuses* (unknown-keyword =user-error=) rather than silently pushing a wrong state, so the safe move is to re-fetch the file so its header and keywords become the derived set. The README migration note must state this because the defcustom is going away.

* Review dispositions

Only modified or rejected recommendations, and decisions worth recording, are listed; everything else from the reviews (2026-05-24, rounds 1–4) was accepted as written and woven into the body above.

** Round 4 (2026-05-24)

Ready verdict — no blocking findings. The round-3 classification blocker is confirmed resolved (=:LINEAR-STATE-TYPE:= drawer + legacy fallback). The sole caveat, verifying =WorkflowState.position=, was already recorded as an implementation prerequisite. Tidied the one org-lint nit (a literal double-star =DEV-REVIEW= example in the test plan) the reviewer flagged as harmless.

** Round 3 (2026-05-24)

- *HP1 "active/done classification of type-less scanned headings" — modified.* The review's fallback (preserve the keyword's side from the buffer's parsed =org-done-keywords=) is a recovery run on every merge. Modified to stop losing the type at the source: render a =:LINEAR-STATE-TYPE:= drawer field (free — the issue query already fetches =type=) so a scanned heading classifies by its own recorded type deterministically. The review's org-done-keywords-side preservation is kept as the *fallback* for legacy headings lacking the field, with default-active-and-log as the last resort. Fully addresses the concern and removes the per-merge reparse for go-forward files.
- *Open question 1 (preserve old side vs default-all-active) — resolved:* preserve the old side, primarily via the stored type, with the parsed-header fallback — not default-all-to-active.
- MP1 (two state-fetch API shapes) accepted: extend =pearl--team-states= only; leave =pearl-get-states-async= unchanged for v1.

** Round 2 (2026-05-24)

- *HP1 "retained dirty subtrees in the header" — accepted, option (b).* The review offered two implementations: thread retained/skipped state metadata out of =pearl--merge-issues-into-buffer=, or scan the final buffer. Chose the final-buffer scan — it directly validates the invariant ("every keyword visible is declared"), subsumes the fetched issues, and avoids widening the merge helper's return contract.
- *Open question 1 (scan: abort vs skip-and-continue) — resolved:* skip-and-continue with a report on the full-file scan; =user-error= only on interactive current-heading sync.
- *Open question 2 (final-buffer scan vs returned metadata for merge coverage) — resolved:* final-buffer scan (the HP1 option-(b) choice).
- MP1 (position-less issue-own ordering), MP2 (split unknown-keyword behavior), and MP3 (return collision metadata) accepted as written.

** Round 1 (2026-05-24)

- *HP3 "partial team-state fetch failure" — modified.* The review offered two rules: global hardcoded fallback on any failure, or fail the render and leave the file unchanged. Both discard information — the first drops real derived keywords for teams that succeeded; the second leaves the user with nothing. Adopted instead the *header coverage guarantee*: derive the header from the union of each displayed issue's own state (always available from the issue query) plus each successfully-fetched team's full states. This keeps every rendered keyword declared regardless of fetch outcome, and a failed team degrades only to "can't cycle to its absent states." The hardcoded line is reserved for the no-states-at-all case. The review's underlying safety requirement — "the render rule is only safe when the header contains that slug" — is met more completely this way.
- *Review open question 1 (leave-unchanged vs conservative fallback) — resolved* by the HP3 modify above: neither; the coverage-guarantee union.
- *Review open questions 2 and 3 — resolved as decisions:* slugify is Unicode-aware and locale-independent (Q2); same-team collisions resolve to first-by-position with a logged warning (Q3). Both now live in Decisions.

* vNext / out of scope

- *Cross-team slug collisions.* Two teams whose states slugify to the same keyword collapse to one keyword in a multi-team file; sync still resolves per the heading's own team, so the push is correct, but the header can't distinguish them. Disambiguation (team-prefixed keywords, per-team =#+TODO= sections) is deferred.
- *Automatic workflow-state cache staleness.* States are cached for the session; a mid-session Linear workflow change needs =pearl-clear-cache=. A TTL/auto-invalidation is deferred.
- *Label-color → tag-face mapping* and other presentation polish.