#+TITLE: Creating New Workflows
#+AUTHOR: Craig Jennings
#+DATE: 2025-11-01

* Overview

This document describes the meta-workflow for creating new workflows. When we identify a repetitive workflow or collaborative pattern, we use this process to formalize it into a documented workflow that we can reference and reuse.

Note the definitions: "sessions" are the time Claude spends with the user, i.e., the user starts a "session" with Claude, does some work, then ends the "session". A workflow is a routine or pattern of doing tasks within a session to accomplish a goal. 

Workflows are living documents that capture how we work together on specific types of tasks. They build our shared vocabulary and enable efficient collaboration across multiple work sessions.

* Problem We're Solving

Without a formal workflow creation process, we encounter several issues:

** Inefficient Use of Intelligence
- Craig leads the process based solely on his knowledge
- We don't leverage Claude's expertise to improve or validate the approach
- Miss opportunities to apply software engineering and process best practices

** Time Waste and Repetition
- Craig must re-explain the workflow each time we work together
- No persistent memory of how we've agreed to work
- Each session starts from scratch instead of building on previous work

** Error-Prone Execution
- Important steps may be forgotten or omitted
- No checklist to verify completeness
- Mistakes lead to incomplete work or failed goals

** Missed Learning Opportunities
- Don't capture lessons learned from our collaboration
- Can't improve processes based on what works/doesn't work
- Lose insights that emerge during execution

** Limited Shared Vocabulary
- No deep, documented understanding of what terms mean
- "Let's do a refactor workflow" has no precise definition
- Can't efficiently communicate about workflows

*Impact:* Inefficiency, errors, and lost opportunity to continuously improve our collaborative workflows.

* Exit Criteria

We know a workflow definition is complete when:

1. **Information is logically arranged** - The structure makes sense and flows naturally
2. **Both parties understand how to work together** - We can articulate the workflow
3. **Agreement on effectiveness** - We both agree that following this workflow will lead to exit criteria and resolve the stated problem
4. **Tasks are clearly defined** - Steps are actionable, not vague
5. **Problem resolution path** - Completing the tasks either:
   - Fixes the problem permanently, OR
   - Provides a process for keeping the problem at bay

*Measurable validation:*
- Can we both articulate the workflow without referring to the document?
- Do we agree it will solve the problem?
- Are the tasks actionable enough to start immediately?
- Does the workflow get used soon after creation (validation by execution)?

* When to Use This Workflow

Trigger this workflow creation workflow when:

- You notice a repetitive workflow that keeps coming up
- A collaborative pattern emerges that would benefit from documentation
- Craig says "let's create/define/design a workflow for [activity]"
- You identify a new type of work that doesn't fit existing workflows
- An existing workflow needs significant restructuring (treat as creating a new one)

Examples:
- "Let's create a workflow where we inbox zero"
- "We should define a code review workflow"
- "Let's design a workflow for weekly planning"

* Approach: How We Work Together
** Phase 0: Context Hygiene

Before starting, write out the session context file and check with Craig whether we could compact the context. This might be a long process. If the context window collapses, we may forget important details. Writing out the session context prevents this data loss. 

** Phase 1: Question and Answer Discovery

Walk through these four core questions collaboratively. Take notes on the answers.

*IMPORTANT: Save answers as you go!*

The Q&A phase can take time—Craig may need to think through answers, and discussions can be lengthy. To prevent data loss from terminal crashes or process quits:

1. Create a draft file at =docs/workflows/[name]-draft.org= after deciding on the name
2. After each question is answered, save the Q&A content to the draft file
3. If workflow is interrupted, you can resume from the saved answers
4. Once complete, the draft becomes the final workflow document

This protects against losing substantial thinking work if the workflow is interrupted.

*** Question 1: What problem are we solving in this type of workflow?

Ask Craig: "What problem are we solving in this type of workflow? What would happen without this workflow?"

The answer reveals:
- Overview and goal of the workflow
- Why this work matters (motivation)
- Impact/priority compared to other work
- What happens if we don't do this work

Example from refactor workflow:
#+begin_quote
"My Emacs configuration isn't resilient enough. There's lots of custom code, and I'm even developing some as Emacs packages. Yet Emacs is my most-used software, so when Emacs breaks, I become unproductive. I need to make Emacs more resilient through good unit tests and refactoring."
#+end_quote

*** Question 2: How do we know when we're done?

Ask Craig: "How do we know when we're done?"

The answer reveals:
- Exit criteria
- Results/completion criteria
- Measurable outcomes

*Your role:*
- Push back if the answer is vague or unmeasurable
- Propose specific measurements based on context
- Iterate together until criteria are clear
- Fallback (hopefully rare): "when Craig says we're done"

Example from refactor workflow:
#+begin_quote
"When we've reviewed all methods, decided which to test and refactor, run all tests, and fixed all failures including bugs we find."
#+end_quote

Claude might add: "How about a code coverage goal of 70%+?"

*** Question 3: How do you see us working together in this kind of workflow?

Ask Craig: "How do you see us working together in this kind of workflow?"

The answer reveals:
- Steps or phases we'll go through
- The general approach to the work
- How tasks flow from one to another

*Your role:*
- As steps emerge, ask yourself:
  - "Do these steps lead to solving the real problem?"
  - "What is missing from these steps?"
- If the answers aren't "yes" and "nothing", raise concerns
- Propose additions based on your knowledge
- Suggest concrete improvements

Example from refactor workflow:
#+begin_quote
"We'll analyze test coverage, categorize functions by testability, write tests systematically using Normal/Boundary/Error categories, run tests, analyze failures, fix bugs, and repeat."
#+end_quote

Claude might suggest: "Should we install a code coverage tool as part of this process?"

*** Question 4: Are there any principles we should be following while doing this?

Ask Craig: "Are there any principles we should be following while doing this kind of workflow?"

The answer reveals:
- Principles to follow
- Decision frameworks
- Quality standards
- When to choose option A vs option B

*Your role:*
- Think through all elements of the workflow
- Consider situations that may arise
- Identify what principles would guide decisions
- Suggest decision frameworks from your knowledge

Example from refactor workflow:
#+begin_quote
Craig: "Treat all test code as production code - same engineering practices apply."

Claude suggests: "Since we'll refactor methods mixing UI and logic, should we add a principle to separate them for testability?"
#+end_quote

** Phase 2: Assess Completeness

After the Q&A, ask together:

1. **Do we have enough information to formulate steps/process?**
   - If yes, proceed to Phase 3
   - If no, identify what's missing and discuss further

2. **Do we agree following this approach will resolve/mitigate the problem?**
   - Both parties must agree
   - If not, identify concerns and iterate

** Phase 3: Name the Workflow

Decide on a name for this workflow.

*Naming convention:* Action-oriented (verb form)
- Examples: "refactor", "inbox-zero", "create-workflow", "review-code"
- Why: Shorter, natural when saying "let's do a [name] workflow"
- Filename: =docs/workflows/[name].org=

** Phase 4: Document the Workflow

Write the workflow file at =docs/workflows/[name].org= using this structure:

*** Recommended Structure
1. *Title and metadata* (=#+TITLE=, =#+AUTHOR=, =#+DATE=)
2. *Overview* - Brief description of the workflow
3. *Problem We're Solving* - From Q&A, with context and impact
4. *Exit Criteria* - Measurable outcomes, how we know we're done
5. *When to Use This Workflow* - Triggers, circumstances, examples
6. *Approach: How We Work Together*
   - Phases/steps derived from Q&A
   - Decision frameworks
   - Concrete examples woven throughout
7. *Principles to Follow* - Guidelines from Q&A
8. *Living Document Notice* - Reminder to update with learnings

*** Important Notes
- Weave concrete examples into sections (don't separate them)
- Use examples from actual workflows when available
- Make tasks actionable, not vague
- Include decision frameworks for common situations
- Note that this is a living document

** Phase 5: Update Project State

Update =notes.org=:
1. Add new workflow to "Available Workflows" section
2. Include brief description and reference to file
3. Note creation date

Example entry:
#+begin_src org
,** inbox-zero
File: =docs/workflows/inbox-zero.org=

Workflow for processing inbox to zero:
1. [Brief workflow summary]
2. [Key steps]

Created: 2025-11-01
#+end_src

** Phase 6: Cleanup
Write out the session context file before proceeding any further

** Phase 7: Validate by Execution

*Critical step:* Use the workflow soon after creating it.

- Schedule the workflow for immediate use
- Follow the documented workflow
- Note what works well
- Identify gaps or unclear areas
- Update the workflow document with learnings

*This validates the workflow definition and ensures it's practical, not theoretical.*

* Principles to Follow

These principles guide us while creating new workflows:

** Collaboration Through Discussion
- Be proactive about collaboration
- Suggest everything on your mind
- Ask all relevant questions
- Push back when something seems wrong, inconsistent, or unclear
- Misunderstandings are learning opportunities

** Reviewing the Whole as Well as the Pieces
- May get into weeds while identifying each step
- Stop to look at the whole thing at the end
- Ask the big questions: Does this actually solve the problem?
- Verify all pieces connect logically

** Concrete Over Abstract
- Use examples liberally within explanations
- Weave concrete examples into Q&A answers
- Don't just describe abstractly
- "When nil input crashes, ask..." is better than "handle edge cases"

** Actionable Tasks Over Vague Direction
- Steps should be clear enough to know what to do next
- "Ask: how do you see us working together?" is actionable
- "Figure out the approach" is too vague
- Test: Could someone execute this without further explanation?

** Validate Early
- "Use it soon afterward" catches problems early
- Don't let workflow definitions sit unused and untested
- Real execution reveals gaps that theory misses
- Update immediately based on first use

** Decision Frameworks Over Rigid Steps
- Workflows are frameworks (principles + flexibility), not recipes
- Include principles that help case-by-case decisions
- "When X happens, ask Y" is a decision framework
- "Always do X" is too rigid for most workflows

** Question Assumptions
- If something doesn't make sense, speak up
- If a step seems to skip something, point it out
- Better to question during creation than discover gaps during execution
- No assumption is too basic to verify

* Living Document

This is a living document. As we create new workflows and learn what works (and what doesn't), we update this file with:

- New insights about workflow creation
- Improvements to the Q&A process
- Better examples
- Additional principles discovered
- Refinements to the structure

Every time we create a workflow, we have an opportunity to improve this meta-process.

** Updates and Learnings

*** 2025-11-01: Save Q&A answers incrementally
*Learning:* During emacs-inbox-zero workflow creation, we discovered that Q&A discussions can be lengthy and make Craig think deeply. Terminal crashes or process quits can lose substantial work.

*Improvement:* Added guidance in Phase 1 to create a draft file and save Q&A answers after each question. This protects against data loss and allows resuming interrupted workflows.

*Impact:* Reduces risk of losing 10-15 minutes of thinking work if workflow is interrupted.

*** 2025-11-01: Validation by execution works!
*Learning:* Immediately after creating the emacs-inbox-zero workflow, we validated it by actually running the workflow. This caught unclear areas and validated that the 10-minute target was realistic.

*Key insight from validation:* When Craig provides useful context during workflows (impact estimates, theories, examples), that context should be captured in task descriptions. This wasn't obvious during workflow creation but became clear during execution.

*Impact:* Validation catches what theory misses. Always use Phase 6 (validate by execution) soon after creating a workflow.

* Example: Creating the "Create-Workflow" Workflow

This very document was created using the process it describes (recursive!).

** The Q&A
- *Problem:* Time waste, errors, missed learning from informal processes
- *Exit criteria:* Logical arrangement, mutual understanding, agreement on effectiveness, actionable tasks
- *Approach:* Four-question Q&A, assess completeness, name it, document it, update notes.org, validate by use
- *Principles:* Collaboration through discussion, review the whole, concrete over abstract, actionable tasks, validate early, decision frameworks, question assumptions

** The Result
We identified what was needed, collaborated on answers, and captured it in this document. Then we immediately used it to create the next workflow (validation).

* Conclusion

Creating workflows is a meta-skill that improves all our collaboration. By formalizing how we work together, we:

- Build shared vocabulary
- Eliminate repeated explanations
- Capture lessons learned
- Enable continuous improvement
- Make our partnership more efficient

Each new workflow we create adds to our collaborative toolkit and deepens our ability to work together effectively.

*Remember:* Workflows are frameworks, not rigid recipes. They provide structure while allowing flexibility for case-by-case decisions. The goal is effectiveness, not perfection.