Architecture Overview

High-Level Architecture

Understanding how Spry processes Markdown into executable workflows through its layered architecture.

Spry CLI

axiom

(graph)

(runbook)

(sqlpage)

lib/axiom

[Semantic Graph Engine]

(loader)

remark

(plugins)

edge

(rules)

projection

(views)

Playbook/Task

Execution

orchestrate

(runner)

lib/spawn

[Unified Execution Framework]

shell

(process)

code-shell

(engine)

os-shell

(bash)

sql-shell

(postgres)

function-shell

(env/envrc)

factory.ts

(catalog & using API)

lib/universal

[Shared Utilities]

task

code

render

resource

directive

lib/tap

[Test Anything Protocol v14]

protocol

(TAP v14)

report

(formats)

task-event-bus

(integration)

Layer Boundaries

Data Flow

Uses/Depends

Processing Pipeline

Spry transforms Markdown through six distinct stages, each building upon the previous to create executable workflows.

Stage 1: Input/Loading

Location: lib/axiom/io/

The input stage loads resources from multiple sources including files, URLs, and glob patterns.

// Load resources from files, URLs, or globs
const resources = await markdownASTs({
  sources: ["runbook.md", "*.md", "https://example.com/doc.md"],
  cwd: process.cwd()
});

Key components:

resource.ts - Abstract resource loading
mod.ts - Pipeline orchestration
VFile integration for metadata tracking

Stage 2: Parsing

Location: lib/axiom/remark/

Markdown is parsed into MDAST (Markdown Abstract Syntax Tree) using the unified/remark ecosystem.

// Parser pipeline
unified()
  .use(remarkParse)
  .use(remarkFrontmatter)
  .use(remarkGfm)
  .use(remarkDirective)

The parser produces a tree structure:

interface Root {
  type: "root";
  children: Node[];
  data?: {
    documentFrontmatter?: Record<string, unknown>;
  };
}

interface Code {
  type: "code";
  lang: string;
  meta: string;
  value: string;
  data?: {
    codeFM?: ParsedCodeFrontmatter;
  };
}

Stage 3: AST Enrichment

Location: lib/axiom/remark/

Remark plugins enhance the AST with semantic information:

Plugin	Purpose
`doc-frontmatter.ts`	Parse YAML frontmatter
`code-directive-candidates.ts`	Identify PARTIAL directives
`actionable-code-candidates.ts`	Mark executable cells
`code-contribute.ts`	Handle file includes
`node-decorator.ts`	Add semantic decorations

Note: Plugin Order Matters

Plugins execute sequentially, with each building on previous AST transformations. The order is critical for proper semantic analysis.

Stage 4: Graph Construction

Location: lib/axiom/edge/

Edge rules create semantic relationships between document elements:

// Build graph from AST
const graph = buildGraph(root, typicalRules());

Edge Types:

containedInSection - Section hierarchy
codeDependsOn - Task dependencies
frontmatterClassification - Semantic roles
sectionSemanticId - Section identity
nodesClassification - Node roles

Stage 5: Projection

Location: lib/axiom/projection/

Projections transform the semantic graph into domain-specific models optimized for different use cases.

FlexibleProjection

UI-neutral view of document structure:

const flex = buildFlexibleProjection(graph);

// Returns:
interface FlexibleProjection {
  documents: ProjectedDocument[];
  nodes: ProjectedNode[];
  edges: ProjectedEdge[];
  hierarchies: ProjectedHierarchy[];
}

Use Cases:

Document analysis and querying
Building custom tooling
Exploring document structure
Creating visualizations

PlaybookProjection

Executable task model for runbook operations:

const playbook = buildPlaybookProjection(graph);

// Returns:
interface PlaybookProjection {
  executables: Executable[];
  materializables: Materializable[];
  directives: Directive[];
  tasks: ExecutableTask[];
}

Use Cases:

Running automated workflows
Executing multi-step processes
Building CI/CD pipelines
Creating interactive runbooks

TreeProjection

Hierarchical view of document structure:

const tree = buildTreeProjection(graph);

Use Cases:

Navigation interfaces
Document outlines
Hierarchical analysis

Stage 6: Execution

Location: lib/axiom/orchestrate/

Execute tasks from the playbook using a DAG-based execution engine:

// Build execution plan
const plan = executionPlan(tasks);

// Execute in topological order
await executeDAG(plan, {
  executor: shellExecutor,
  onTaskStart: (task) => console.log(`Starting: ${task.taskId()}`),
  onTaskComplete: (task, result) => console.log(`Done: ${task.taskId()}`),
});

Warning: Dependency Resolution

The execution planner uses Kahn's algorithm for topological sorting. Circular dependencies are detected and reported as errors before execution begins.

Key Modules

Understanding Spry's module organization helps you navigate the codebase and extend functionality.

lib/axiom

The semantic graph engine and core transformation pipeline.

Module	Description
`graph.ts`	Graph construction
`edge/`	Edge rules
`projection/`	View builders
`orchestrate/`	Execution engine
`io/`	Resource loading
`remark/`	AST plugins
`mdast/`	AST utilities and hooks
`text-ui/`	CLI interface

lib/axiom/
├── mod.ts              # Public exports
├── graph.ts            # Graph building
├── edge/               # Edge rules
│   ├── mod.ts
│   ├── orchestrate.ts  # Rule pipeline
│   ├── rule/           # Individual rules
│   └── pipeline/       # Rule compositions
├── io/                 # I/O and parsing
│   ├── mod.ts
│   └── resource.ts
├── mdast/              # AST utilities
│   ├── data-bag.ts
│   ├── node-issues.ts
│   └── ...
├── projection/         # Graph projections
│   ├── flexible.ts
│   ├── playbook.ts
│   └── tree.ts
├── remark/             # Remark plugins
│   ├── actionable-code-candidates.ts
│   ├── code-directive-candidates.ts
│   └── ...
├── text-ui/            # Terminal interfaces
└── web-ui/             # Web interface

lib/spawn

Unified execution framework providing language-agnostic code execution.

Module	Description
`shell.ts`	Low-level process spawning (thin wrapper around Deno.Command)
`code-shell.ts`	LanguageEngine interface and execution orchestration
`function-shell.ts`	In-process function engines (env, envrc)
`os-shell.ts`	OS shell engines (bash, sh, PowerShell, cmd, fish)
`sql-shell/`	SQL database engines (postgres, sqlite, duckdb, surveilr)
`factory.ts`	Catalog parsing and `using()` API for engine resolution

Key concepts:

Languages vs Runtimes: Languages are semantic (SQL, shell), runtimes are concrete (psql, bash)
Execution Modes: stdin, file, eval, auto - how code is passed to the runtime
Catalogs: YAML-based declarative runtime configuration
LanguageEngine: Typed interface describing how a runtime should be invoked

lib/spawn/
├── shell.ts            # Process spawning
├── code-shell.ts       # Engine interface
├── function-shell.ts   # Function engines
├── os-shell.ts         # OS shells
├── sql-shell/          # SQL engines
│   ├── postgres.ts
│   ├── sqlite.ts
│   ├── duckdb.ts
│   └── surveilr.ts
└── factory.ts          # Catalog & using API

lib/universal

Shared utilities used across the system.

Module	Description
`task.ts`	DAG execution and planning
`code.ts`	Language registry and specifications
`directive.ts`	Directive parsing
`posix-pi.ts`	CLI flag parsing
`render.ts`	Template interpolation
`resource.ts`	Resource abstraction

lib/universal/
├── task.ts             # DAG execution
├── code.ts             # Language specs
├── resource.ts         # Resource loading
├── directive.ts        # Directive parsing
├── event-bus.ts        # Event system
├── watcher.ts          # File watching
└── ...

lib/tap

Test Anything Protocol (TAP v14) support for test reporting and compliance.

Module	Description
`protocol.ts`	TAP v14 data structures and stringification
`report.ts`	Multi-format TAP output (canonical, HTML, Markdown, JSON)
`task-event-bus.ts`	Integration with task execution for TAP report generation
`compliance.ts`	TAP compliance utilities

lib/tap/
├── protocol.ts         # TAP v14 protocol
├── report.ts           # Multi-format output
├── task-event-bus.ts   # Task integration
└── compliance.ts       # TAP compliance

lib/playbook

Domain-specific playbook implementations.

Module	Description
`sqlpage/`	SQLPage integration

lib/playbook/
├── README.md           # Architecture docs
└── sqlpage/
    ├── cli.ts          # SQLPage CLI
    ├── content.ts      # Content generation
    ├── interpolate.ts  # Template interpolation
    └── orchestrate.ts  # Orchestration

Data Flow Example

Understanding data flow through the pipeline helps you debug issues and optimize workflows.

Markdown to Tasks

Step 1: Load Markdown File

runbook.md
    ↓ [io/resource.ts]
VFile with content

Step 2: Parse to AST

    ↓ [remark/parser pipeline]
MDAST (raw)
    ↓ [remark/plugins]
MDAST (enriched with codeFM, decorations)

Step 3: Build Semantic Graph

    ↓ [edge/orchestrate.ts]
Graph { root, edges: [...relationships] }

Step 4: Create Projection

    ↓ [projection/playbook.ts]
PlaybookProjection { tasks: [...] }

Step 5: Plan Execution

    ↓ [orchestrate/task.ts]
ExecutionPlan { layers: [[task1], [task2, task3], [task4]] }

Step 6: Execute Tasks

    ↓ [spawn/factory.ts → spawn/code-shell.ts → spawn/shell.ts]
LanguageEngine execution with catalog-based runtime resolution
    ↓
LanguageSpawnResult { stdout, stderr, exitCode, success }

Extension Points

Spry's architecture makes it straightforward to add new capabilities through well-defined extension points.

1. Custom Remark Plugins

Add new AST transformations:

const myPlugin: Plugin<[], Root> = () => {
  return (tree) => {
    visit(tree, "code", (node) => {
      // Transform code nodes
    });
  };
};

2. Custom Edge Rules

Add new relationship types:

const myRule: EdgeRule = {
  rel: "myRelationship",
  apply: function* (root) {
    // Yield edges
    yield { from: nodeA, to: nodeB, rel: "myRelationship" };
  },
};

3. Custom Projections

Create new domain views:

function myProjection(graph: Graph): MyProjection {
  // Transform graph into domain-specific view
  return { ... };
}

4. Custom Language Engines

Add new execution runtimes via lib/spawn:

import { LanguageEngine, ExecutionMode } from "./lib/spawn/code-shell.ts";

const myEngine: LanguageEngine<MyInit> = {
  engineIdentity: "my-runtime",
  supportedModes: ["stdin", "file"],

  plan: (init, input, mode) => ({
    argv: ["my-runtime", "--flag"],
    stdin: mode === "stdin" ? input.code : undefined,
    cleanup: async () => { /* cleanup temp files */ },
  }),
};

Engines can be registered in catalogs (YAML) for declarative configuration:

runtimes:
  my-db:
    engine: my-runtime
    connection: ${env.MY_DB_URL}

Type System

Key types used throughout Spry's architecture:

// From lib/axiom/
type Graph<Rel, Edge> = {
  root: Root;
  edges: Edge[];
};

type GraphEdge<Rel> = {
  rel: Rel;
  from: Node;
  to: Node;
};

// From lib/universal/
type Task<Baggage> = {
  taskId: () => string;
  taskDeps?: () => string[];
} & Baggage;

type ExecutionPlan<T> = {
  tasks: T[];
  layers: T[][];
  order: T[];
};

How is this guide?

Architecture Overview

On this page