Architecture Overview

High-Level Architecture

Spry transforms Markdown documents into executable workflows through a multi-stage pipeline. The architecture separates concerns into distinct layers: loading, parsing, graph construction, projection, and execution.

Processing Pipeline

Stage 1: Input/Loading

Location: lib/axiom/io/

The loading stage accepts Markdown files from multiple sources and normalizes them into a unified resource format.

// Load resources from files, URLs, or globs
const resources = await markdownASTs({
  sources: ["runbook.md", "*.md", "https://example.com/doc.md"],
  cwd: process.cwd()
});

Key components:

• resource.ts - Abstract resource loading
• mod.ts - Pipeline orchestration
• VFile integration for metadata tracking

Stage 2: Parsing

Location: lib/axiom/remark/

Markdown is parsed into MDAST (Markdown Abstract Syntax Tree) using unified and remark plugins:

// Parser pipeline
unified()
  .use(remarkParse)
  .use(remarkFrontmatter)
  .use(remarkGfm)
  .use(remarkDirective)

The result is a tree structure:

interface Root {
  type: "root";
  children: Node[];
  data?: {
    documentFrontmatter?: Record<string, unknown>;
  };
}

interface Code {
  type: "code";
  lang: string;
  meta: string;
  value: string;
  data?: {
    codeFM?: ParsedCodeFrontmatter;
  };
}

This AST representation allows Spry to analyze and transform the document structure programmatically.

Stage 3: AST Enrichment

Location: lib/axiom/remark/

Remark plugins enhance the AST with semantic metadata and relationships:

These plugins transform a raw syntax tree into a semantically rich structure that understands the document's intent.

Stage 4: Graph Construction

Location: lib/axiom/edge/

Edge rules create semantic relationships between nodes, building a graph that represents the document's structure and dependencies:

// Build graph from AST
const graph = buildGraph(root, typicalRules());

Edge Types:

The graph representation enables powerful queries and transformations that would be difficult with raw AST traversal.

Stage 5: Projection

Location: lib/axiom/projection/

Projections create domain-specific views of the graph for different use cases:

// FlexibleProjection - UI-neutral view
const flex = buildFlexibleProjection(graph);

// PlaybookProjection - Execution view
const playbook = buildPlaybookProjection(graph);

// TreeProjection - Hierarchical view
const tree = buildTreeProjection(graph);

FlexibleProjection:

interface FlexibleProjection {
  documents: ProjectedDocument[];
  nodes: ProjectedNode[];
  edges: ProjectedEdge[];
  hierarchies: ProjectedHierarchy[];
}

PlaybookProjection:

interface PlaybookProjection {
  executables: Executable[];
  materializables: Materializable[];
  directives: Directive[];
  tasks: ExecutableTask[];
}

Projections isolate domain concerns, making it easy to add new views without modifying core graph logic.

Stage 6: Execution

Location: lib/axiom/orchestrate/

Execute tasks from the playbook in the correct order, respecting dependencies:

// Build execution plan
const plan = executionPlan(tasks);

// Execute in topological order
await executeDAG(plan, {
  executor: shellExecutor,
  onTaskStart: (task) => console.log(`Starting: ${task.taskId()}`),
  onTaskComplete: (task, result) => console.log(`Done: ${task.taskId()}`),
});

The execution engine handles:

• Dependency resolution
• Parallel execution of independent tasks
• Error handling and rollback
• Progress reporting

Key Modules

lib/axiom/

Core semantic processing layer:

lib/spawn/

Unified execution framework providing language-agnostic code execution:

Key concepts:

• Languages vs Runtimes: Languages are semantic (SQL, shell), runtimes are concrete (psql, bash)
• Execution Modes: stdin, file, eval, auto - how code is passed to the runtime
• Catalogs: YAML-based declarative runtime configuration
• LanguageEngine: Typed interface describing how a runtime should be invoked

lib/universal/

Shared utilities used across the system:

lib/playbook/

Domain-specific patterns and integrations:

Data Flow Example

Here's how a document flows through the entire pipeline:

runbook.md
    │
    ▼ [axiom/io/resource.ts]
VFile with content
    │
    ▼ [axiom/remark/parser pipeline]
MDAST (raw)
    │
    ▼ [axiom/remark/plugins]
MDAST (enriched with codeFM, decorations)
    │
    ▼ [axiom/edge/orchestrate.ts]
Graph { root, edges: [...relationships] }
    │
    ▼ [axiom/projection/playbook.ts]
PlaybookProjection { tasks: [...] }
    │
    ▼ [axiom/orchestrate/task.ts]
ExecutionPlan { layers: [[task1], [task2, task3], [task4]] }
    │
    ▼ [spawn/factory.ts → spawn/code-shell.ts → spawn/shell.ts]
LanguageEngine execution with catalog-based runtime resolution
    │
    ▼
LanguageSpawnResult { stdout, stderr, exitCode, success }

Each stage transforms the data into a form appropriate for the next stage, maintaining clear separation of concerns.

Extension Points

Spry's architecture provides multiple extension points for customization:

1. Custom Remark Plugins

Add new AST transformations:

const myPlugin: Plugin<[], Root> = () => {
  return (tree) => {
    visit(tree, "code", (node) => {
      // Transform code nodes
    });
  };
};

Use cases:

• Custom code block metadata
• Special directive handling
• Document validation rules

2. Custom Edge Rules

Add new relationship types:

const myRule: EdgeRule = {
  rel: "myRelationship",
  apply: function* (root) {
    // Yield edges
    yield { from: nodeA, to: nodeB, rel: "myRelationship" };
  },
};

Use cases:

• Custom dependency types
• Semantic relationships
• Cross-document references

3. Custom Projections

Create new domain views:

function myProjection(graph: Graph): MyProjection {
  // Transform graph into domain-specific view
  return { ... };
}

Use cases:

• Custom report formats
• Specialized workflows
• Domain-specific validations

4. Custom Language Engines

Add new execution runtimes via lib/spawn:

import { LanguageEngine, ExecutionMode } from "./lib/spawn/code-shell.ts";

const myEngine: LanguageEngine<MyInit> = {
  engineIdentity: "my-runtime",
  supportedModes: ["stdin", "file"],

  plan: (init, input, mode) => ({
    argv: ["my-runtime", "--flag"],
    stdin: mode === "stdin" ? input.code : undefined,
    cleanup: async () => { /* cleanup temp files */ },
  }),
};

Engines can be registered in catalogs (YAML) for declarative configuration:

runtimes:
  my-db:
    engine: my-runtime
    connection: ${env.MY_DB_URL}

Use cases:

• Custom database engines
• Domain-specific languages
• Integration with proprietary tools

Type System

Key types that define the system's structure:

// From lib/axiom/
type Graph<Rel, Edge> = {
  root: Root;
  edges: Edge[];
};

type GraphEdge<Rel> = {
  rel: Rel;
  from: Node;
  to: Node;
};

// From lib/universal/
type Task<Baggage> = {
  taskId: () => string;
  taskDeps?: () => string[];
} & Baggage;

type ExecutionPlan<T> = {
  tasks: T[];
  layers: T[][];
  order: T[];
};

The type system ensures type safety throughout the pipeline while remaining flexible enough for diverse use cases.

Architecture Principles

Spry's architecture follows these core principles:

Performance Considerations

Spry's architecture optimizes for:

• Lazy Evaluation: Projections are built on-demand, not eagerly
• Parallel Execution: Independent tasks execute concurrently
• Memory Efficiency: Streaming processing for large documents
• Caching: Parsed ASTs and graphs can be cached across invocations

For large-scale deployments, consider:

• Pre-parsing documents during CI/CD
• Caching projection results
• Parallelizing document processing across multiple cores

Summary

Spry's architecture transforms Markdown into executable workflows through a principled, multi-stage pipeline:

1. Load documents from diverse sources
2. Parse into abstract syntax trees
3. Enrich with semantic metadata
4. Build relationship graphs
5. Project into domain-specific views
6. Execute with dependency-aware orchestration

This architecture provides:

• Clarity: Each stage has a clear purpose
• Flexibility: Easy to extend at every level
• Reliability: Type-safe transformations throughout
• Performance: Optimized for real-world usage