Architecture Deep Dive

System Overview

Spry transforms Markdown into executable workflows through a multi-stage pipeline. Understanding this architecture helps you leverage Spry's full capabilities and extend it for your needs.

Phase 1: Parse

The parsing phase converts Markdown text into an Abstract Syntax Tree (AST) using the unified/remark ecosystem.

Location

lib/axiom/io/mod.ts

Components

import { mardownParserPipeline } from "./lib/axiom/io/mod.ts";

const pipeline = mardownParserPipeline();
// Uses unified() with a series of plugins

Plugin Pipeline

The parser runs a carefully ordered sequence of plugins, each building on the previous:

Output

An mdast Root node with enhanced data properties on relevant nodes, ready for semantic analysis.

Phase 2: Analyze (Axiom)

Axiom applies edge rules to build a semantic graph representing the relationships between elements in your Spryfile.

Location

lib/axiom/edge/

Edge Rules

Rules are functions that process the AST and yield relationship edges:

type GraphEdgesRule<Rel, Ctx, Edge> = (
  ctx: Ctx,
  prevEdges: Iterable<Edge>
) => Iterable<Edge> | false;

Core Rules

Rule Execution

function* astGraphEdges(root, { prepareContext, rules }) {
  const ctx = prepareContext(root);

  let current = [];
  for (const rule of rules(ctx)) {
    const produced = rule(ctx, current);
    if (produced !== false) {
      current = produced;
    }
  }

  for (const edge of current) {
    yield edge;
  }
}

Output

A Graph object containing:

• root - The mdast Root node
• edges - Array of relationship edges
• rels - Set of relationship types
• relCounts - Count per relationship type

Phase 3: Project

Projections transform the semantic graph into domain-specific models optimized for different use cases.

Location

lib/axiom/projection/

Available Projections

const model = await flexibleProjectionFromFiles(paths);

// Returns:
{
  documents: [...],      // Document metadata
  nodes: {...},          // Node lookup by ID
  hierarchies: {...},    // Section tree structures
  mdastStore: [...],     // AST node storage
}

const { tasks, directives, issues } = await playbooksFromFiles(paths);

// Returns:
{
  sources: [...],           // Source documents
  executables: [...],       // Executable cells
  materializables: [...],   // Materializable cells
  directives: [...],        // Directive cells
  tasks: [...],             // ExecutableTask objects
  issues: [...],            // Validation issues
}

Task Classification

Tasks are categorized by their nature and intended use:

Phase 4: Execute / Emit

The final phase executes tasks or emits artifacts using a DAG-based execution engine.

Location

lib/universal/task.ts, lib/axiom/orchestrate/task.ts

Execution Plan

const plan = executionPlan(tasks);

// Plan includes:
{
  ids: [...],         // Task identifiers
  byId: {...},        // Quick lookup map
  layers: [...],      // Parallel execution waves
  dag: [...],         // Topological order
  edges: [...],       // Dependency edges
  unresolved: [...],  // Cycle detection
}

DAG Execution

The task executor uses Kahn's algorithm for dependency resolution:

Task Runner

const runbook = tasksRunbook({ directives, shellBus, tasksBus });
const results = await runbook.execute(plan);

Execution Features:

• Event-driven progress reporting
• Streaming output capture
• Error handling and recovery
• Parallel execution support (via layers)

Library Structure

Understanding Spry's module organization helps you navigate the codebase and extend functionality.

lib/axiom

The semantic graph engine and core transformation pipeline.

lib/axiom/
├── mod.ts              # Public exports
├── graph.ts            # Graph building
├── edge/               # Edge rules
│   ├── mod.ts
│   ├── orchestrate.ts  # Rule pipeline
│   ├── rule/           # Individual rules
│   └── pipeline/       # Rule compositions
├── io/                 # I/O and parsing
│   ├── mod.ts
│   └── resource.ts
├── mdast/              # AST utilities
│   ├── data-bag.ts
│   ├── node-issues.ts
│   └── ...
├── projection/         # Graph projections
│   ├── flexible.ts
│   ├── playbook.ts
│   └── tree.ts
├── remark/             # Remark plugins
│   ├── actionable-code-candidates.ts
│   ├── code-directive-candidates.ts
│   └── ...
├── text-ui/            # Terminal interfaces
└── web-ui/             # Web interface

lib/universal

Shared utilities used across the system.

lib/universal/
├── task.ts             # DAG execution
├── shell.ts            # Shell commands
├── resource.ts         # Resource loading
├── code.ts             # Code parsing
├── directive.ts        # Directive parsing
├── event-bus.ts        # Event system
├── watcher.ts          # File watching
└── ...

lib/courier

Data movement protocol implementations.

lib/courier/
├── protocol.ts         # DataMP protocol
├── singer.ts           # Singer adapter
└── airbyte.ts          # Airbyte adapter

lib/playbook

Domain-specific playbook implementations.

lib/playbook/
├── README.md           # Architecture docs
└── sqlpage/
    ├── cli.ts          # SQLPage CLI
    ├── content.ts      # Content generation
    ├── interpolate.ts  # Template interpolation
    └── orchestrate.ts  # Orchestration

Data Flow Examples

Understanding data flow through the pipeline helps you debug issues and optimize workflows.

Markdown to Tasks

Markdown to SQLPage

Design Principles

Spry's architecture embodies several key design principles that guide development and usage.

Determinism

Same input always produces same output:

• Stable topological ordering - Tasks execute in a predictable sequence
• Definition-order tie-breaking - When multiple orders are valid, uses document order
• Reproducible graph generation - AST transformations are pure functions

Composability

Components mix freely without tight coupling:

• Projections stack - Multiple views of the same graph
• Rules extend - Add new relationships without modifying existing ones
• Playbooks share infrastructure - Reuse executors and utilities

Type Safety

TypeScript provides strong guarantees:

• Zod schemas - Runtime validation of configurations and data
• Generic types - Flexible while maintaining type safety
• Runtime checking - Catch errors before they cause problems

Extensibility

Plugin architecture throughout:

• Custom remark plugins - Transform Markdown in new ways
• Custom edge rules - Define new semantic relationships
• Custom projections - Create domain-specific models
• Custom executors - Support new languages and tools

Performance Considerations

Understanding performance characteristics helps you build efficient workflows.

Planning

• O(V + E) complexity - Linear in vertices and edges
• Stable sorting - Limited to frontier nodes only
• Non-destructive indegree - Preserves original graph structure

Execution

• Serial by default - Predictable and debuggable
• Parallelization available - Via execution layers when safe
• Event-driven reporting - Non-blocking progress updates
• Streaming output - Memory-efficient capture of large outputs

Memory

• AST nodes shared - Not copied, reducing memory footprint
• Lazy iteration - Process data on-demand where possible
• VFile-based resources - Efficient file handling

Extending Spry

Spry's architecture makes it straightforward to add new capabilities.

Adding a Custom Language

const myLang = {
  nature: "EXECUTABLE" as const,
  lang: "mylang",
  extensions: [".mylang"]
};

async function executeMyLang(cell: CodeCell, context: ExecutionContext) {
  // Your execution logic here
  const result = await runMyLanguage(cell.code);
  return { stdout: result, stderr: "", exitCode: 0 };
}

languageRegistry.register(myLang, executeMyLang);

Adding a Custom Projection

export function myCustomProjection(graph: Graph): MyDomainModel {
  // Extract relevant nodes and edges
  const relevantNodes = Array.from(graph.nodes())
    .filter(node => /* your criteria */);

  // Build your domain model
  return {
    // Your custom structure
  };
}

Adding a Custom Edge Rule

export function* myCustomRule(
  ctx: GraphContext,
  prevEdges: Iterable<Edge>
): Iterable<Edge> {
  // Analyze the AST
  const nodes = selectNodes(ctx.root);
  
  // Yield new edges
  for (const node of nodes) {
    yield {
      source: node.id,
      target: relatedNode.id,
      relationship: "myRelationship"
    };
  }
  
  // Pass through previous edges
  yield* prevEdges;
}