Spawn Module

Concept and Purpose

The lib/spawn module is a language-agnostic execution framework. Its primary purpose is to provide a consistent, type-safe, and declarative way to run code, scripts, and commands regardless of the underlying language or runtime engine.

Core Philosophy

What vs How

The "what" (the code/script) is separated from the "how" (the runtime engine). This allows you to swap implementations without changing calling code.

Catalog-Based

Execution environments (e.g., a specific database connection, a shell with specific flags) are defined in "catalogs" rather than hardcoded in imperative logic.

Consistent Interface

Whether you are running a SQL query against Postgres, executing a Bash script, or running an in-process function, the API looks exactly the same to the developer.

This design is ideal for developer tooling, automation tasks, and orchestration where predictability and swap-ability of runtimes are important.

Key Components

The module is layered to separate low-level process spawning from high-level orchestration.

Architecture Layers

Component	Purpose	Layer
`shell.ts`	Process spawning primitive	Low-level
`code-shell.ts`	Execution orchestration	Mid-level
Engines	Language-specific adapters	Implementation
`factory.ts`	User-facing API	High-level

Core Modules

// The lowest level primitive
// Wraps Deno.Command to spawn processes
// Captures output and handles exit codes
// Knows nothing about "languages"

import { spawn } from "./lib/spawn/shell.ts";

const result = await spawn({
  cmd: ["echo", "hello"],
  cwd: "/tmp"
});

// The orchestration layer
// Introduces LanguageEngine concept
// Handles execution modes (stdin, file, eval)
// Coordinates the execution lifecycle

import { executeCode } from "./lib/spawn/code-shell.ts";

const result = await executeCode(engine, {
  kind: "text",
  text: "SELECT 1"
});

Available Engines

OS Shells: bash, sh, pwsh, cmd (os-shell.ts)
SQL Databases: postgres, sqlite, duckdb (sql-shell/*)
In-Process: env and others (function-shell.ts)

All engines implement the same LanguageEngine interface.

// Main user-facing entry point
// Handles catalog parsing
// Resolves engines
// Creates executable handles

import { using } from "./lib/spawn/factory.ts";

const executor = using(catalog, "my_runtime");

Workflow and Usage

The standard workflow for using lib/spawn involves four steps: Define, Parse, Select, and Execute.

Define a Catalog

Catalogs describe the available runtime environments. They can be defined as YAML strings or plain JavaScript objects.

spawn-catalog.yaml

spawnables:
  # A local Postgres database
  pg_local:
    engine: postgres
    host: 127.0.0.1
    port: 5432
    user: app
    dbname: appdb

  # An in-memory SQLite database
  sqlite_mem:
    engine: sqlite
    file: ":memory:"

  # A hardened Bash shell
  safe_bash:
    engine: bash
    env:
      RESTRICTED_MODE: "1"

const catalog = {
  spawnables: {
    pg_local: {
      engine: "postgres",
      host: "127.0.0.1",
      port: 5432,
      user: "app",
      dbname: "appdb"
    },
    sqlite_mem: {
      engine: "sqlite",
      file: ":memory:"
    },
    safe_bash: {
      engine: "bash",
      env: {
        RESTRICTED_MODE: "1"
      }
    }
  }
};

Parse the Catalog

Use catalogFromYaml to convert the configuration into a typed catalog object.

import { catalogFromYaml } from "./lib/spawn/factory.ts";

const yamlConfig = `...`; // from file or string
const catalog = catalogFromYaml(yamlConfig);

The catalog is validated at parse time, ensuring all required fields are present and types are correct.

Select a Runtime

Use the using function to bind a specific entry from the catalog to an executor.

import { using } from "./lib/spawn/factory.ts";

// Get a handle to the sqlite_mem runtime defined above
const db = using(catalog, "sqlite_mem");

Execute (Spawn)

Call .spawn() on the handle with your input. The input shape is consistent across all engines.

const sql = "CREATE TABLE items (id INT); INSERT INTO items VALUES (1);";

const result = await db.spawn({
  kind: "text",
  text: sql
});

if (result.success) {
  console.log("Output:", new TextDecoder().decode(result.stdout));
} else {
  console.error("Error:", new TextDecoder().decode(result.stderr));
}

Complete Example

Here is a complete example demonstrating the full workflow:

spawn-example.ts

import { catalogFromYaml, using } from "./lib/spawn/factory.ts";

async function main() {
  // 1. Define Catalog
  const config = `
spawnables:
  my_duckdb:
    engine: duckdb
    file: ":memory:"
  my_shell:
    engine: bash
`;

  // 2. Parse
  const catalog = catalogFromYaml(config);

  // 3. Select Runtime
  const duck = using(catalog, "my_duckdb");
  const bash = using(catalog, "my_shell");

  // 4. Execute (SQL)
  console.log("Running SQL...");
  const sqlRes = await duck.spawn({
    kind: "text",
    text: "SELECT 42 AS answer"
  });
  console.log("SQL Result:", new TextDecoder().decode(sqlRes.stdout));

  // 4. Execute (Shell)
  console.log("Running Shell...");
  const shRes = await bash.spawn({
    kind: "text",
    text: "echo 'Hello from Bash'"
  });
  console.log("Shell Result:", new TextDecoder().decode(shRes.stdout));
}

main();

Engine Configuration

SQL Engines

spawnables:
  my_postgres:
    engine: postgres
    host: localhost
    port: 5432
    user: myuser
    password: mypass
    dbname: mydb
    # Optional
    sslmode: prefer

spawnables:
  my_sqlite:
    engine: sqlite
    file: /path/to/database.db
    # Or in-memory
    # file: ":memory:"

spawnables:
  my_duckdb:
    engine: duckdb
    file: /path/to/database.duckdb
    # Or in-memory
    # file: ":memory:"

Shell Engines

spawnables:
  my_bash:
    engine: bash
    # Optional environment variables
    env:
      DEBUG: "1"
      PATH: "/custom/path:$PATH"

spawnables:
  my_pwsh:
    engine: pwsh
    # PowerShell-specific options
    env:
      POWERSHELL_TELEMETRY_OPTOUT: "1"

spawnables:
  my_cmd:
    engine: cmd
    # Windows CMD environment
    env:
      PROMPT: "$P$G"

Input Formats

The spawn module supports multiple input formats for maximum flexibility.

Text Input

// Direct text/code
await executor.spawn({
  kind: "text",
  text: "SELECT * FROM users;"
});

File Input

// Execute from file
await executor.spawn({
  kind: "file",
  path: "/path/to/script.sql"
});

Eval Mode

// Quick evaluation (engine-dependent)
await executor.spawn({
  kind: "eval",
  expression: "1 + 1"
});

Not all engines support all input formats. Check the engine documentation for supported modes.

Result Handling

All spawn operations return a consistent result structure.

Success Case

const result = await executor.spawn({ kind: "text", text: "..." });

if (result.success) {
  console.log("Exit code:", result.code); // 0
  console.log("Output:", new TextDecoder().decode(result.stdout));
  console.log("Errors:", new TextDecoder().decode(result.stderr));
}

const result = await executor.spawn({ kind: "text", text: "..." });

if (result.success) {
  // Parse JSON output
  const output = new TextDecoder().decode(result.stdout);
  const data = JSON.parse(output);
  console.log("Parsed:", data);
}

Error Case

const result = await executor.spawn({ kind: "text", text: "..." });

if (!result.success) {
  console.error("Failed with code:", result.code);
  console.error("Error output:", new TextDecoder().decode(result.stderr));
  
  // Handle specific error codes
  if (result.code === 127) {
    console.error("Command not found");
  }
}

Advanced Usage

Environment Variables

// Define runtime with custom environment
const config = `
spawnables:
  custom_bash:
    engine: bash
    env:
      API_KEY: "secret_key"
      DEBUG: "true"
      NODE_ENV: "production"
`;

const catalog = catalogFromYaml(config);
const bash = using(catalog, "custom_bash");

// Environment variables are available in the spawned process
await bash.spawn({
  kind: "text",
  text: "echo $API_KEY"
});

Working Directory

// Execute in specific directory
const result = await executor.spawn({
  kind: "text",
  text: "ls -la",
  cwd: "/path/to/directory" // Override working directory
});

Timeout Control

// Set execution timeout
const result = await executor.spawn({
  kind: "text",
  text: "long_running_script.sh",
  timeout: 30000 // 30 seconds
});

Streaming Output

// Handle output as it arrives
const result = await executor.spawn({
  kind: "text",
  text: "tail -f /var/log/app.log",
  onStdout: (chunk) => {
    console.log("Output:", new TextDecoder().decode(chunk));
  },
  onStderr: (chunk) => {
    console.error("Error:", new TextDecoder().decode(chunk));
  }
});

Best Practices

Catalog Organization

dev-catalog.yaml

spawnables:
  dev_db:
    engine: postgres
    host: localhost
    port: 5432
    user: dev
    dbname: app_dev
  
  local_shell:
    engine: bash
    env:
      DEBUG: "1"

prod-catalog.yaml

spawnables:
  prod_db:
    engine: postgres
    host: db.example.com
    port: 5432
    user: app
    dbname: app_prod
    sslmode: require
  
  prod_shell:
    engine: bash
    env:
      NODE_ENV: "production"

test-catalog.yaml

spawnables:
  test_db:
    engine: sqlite
    file: ":memory:"
  
  test_shell:
    engine: bash
    env:
      CI: "true"

Error Handling Strategy

async function safeExecute(executor: Executor, code: string) {
  try {
    const result = await executor.spawn({
      kind: "text",
      text: code
    });
    
    if (result.success) {
      return {
        ok: true,
        output: new TextDecoder().decode(result.stdout)
      };
    } else {
      return {
        ok: false,
        error: new TextDecoder().decode(result.stderr),
        code: result.code
      };
    }
  } catch (error) {
    return {
      ok: false,
      error: error.message,
      code: -1
    };
  }
}

Resource Cleanup

// Always clean up resources
const executor = using(catalog, "my_db");

try {
  const result = await executor.spawn({ kind: "text", text: "..." });
  // Process result
} finally {
  // Cleanup if needed (connection pooling, temp files, etc.)
  await executor.cleanup?.();
}

Security Considerations

Important Security Guidelines

Never pass untrusted user input directly to spawn
Validate and sanitize all inputs
Use parameterized queries for SQL engines
Restrict shell access in production
Use environment-specific catalogs
Avoid storing credentials in catalogs (use environment variables)

// Bad: SQL injection risk
const userInput = "'; DROP TABLE users; --";
await db.spawn({
  kind: "text",
  text: `SELECT * FROM users WHERE name = '${userInput}'`
});

// Good: Use parameterized queries or validation
function validateInput(input: string): boolean {
  // Implement proper validation
  return /^[a-zA-Z0-9_]+$/.test(input);
}

if (validateInput(userInput)) {
  await db.spawn({ kind: "text", text: safeQuery });
}

Testing

Unit Testing Executors

import { assertEquals } from "std/assert/mod.ts";
import { describe, it } from "std/testing/bdd.ts";
import { catalogFromYaml, using } from "./lib/spawn/factory.ts";

describe("Spawn Module", () => {
  it("should execute SQL successfully", async () => {
    const config = `
spawnables:
  test_db:
    engine: sqlite
    file: ":memory:"
`;
    
    const catalog = catalogFromYaml(config);
    const db = using(catalog, "test_db");
    
    const result = await db.spawn({
      kind: "text",
      text: "SELECT 42 AS answer"
    });
    
    assertEquals(result.success, true);
  });
  
  it("should handle errors gracefully", async () => {
    const catalog = catalogFromYaml(config);
    const db = using(catalog, "test_db");
    
    const result = await db.spawn({
      kind: "text",
      text: "INVALID SQL"
    });
    
    assertEquals(result.success, false);
  });
});

Integration Testing

describe("Integration Tests", () => {
  it("should work across multiple engines", async () => {
    const config = `
spawnables:
  db:
    engine: sqlite
    file: ":memory:"
  shell:
    engine: bash
`;
    
    const catalog = catalogFromYaml(config);
    
    // Test database
    const db = using(catalog, "db");
    const dbResult = await db.spawn({
      kind: "text",
      text: "CREATE TABLE test (id INT)"
    });
    assertEquals(dbResult.success, true);
    
    // Test shell
    const shell = using(catalog, "shell");
    const shellResult = await shell.spawn({
      kind: "text",
      text: "echo 'test'"
    });
    assertEquals(shellResult.success, true);
  });
});

Migration Guide

From Direct Command Execution

// Old: Direct Deno.Command usage
const cmd = new Deno.Command("psql", {
  args: ["-c", "SELECT 1"],
  env: { PGHOST: "localhost" }
});

const output = await cmd.output();

// New: Catalog-based spawn
const catalog = catalogFromYaml(`
spawnables:
my_db:
engine: postgres
host: localhost
`);

const db = using(catalog, "my_db");
const result = await db.spawn({
  kind: "text",
  text: "SELECT 1"
});

From Language-Specific Clients

// Old: Using node-postgres directly
import { Client } from "pg";

const client = new Client({
  host: "localhost",
  database: "mydb"
});

await client.connect();
const res = await client.query("SELECT 1");
await client.end();

// New: Unified spawn interface
const db = using(catalog, "my_db");

const result = await db.spawn({
  kind: "text",
  text: "SELECT 1"
});

// Same interface for any engine!

Troubleshooting

Common Issues

Issue	Cause	Solution
Engine not found	Invalid engine name	Verify engine name in catalog matches available engines
Connection failed	Wrong credentials/host	Check catalog configuration, verify network connectivity
Timeout	Long-running operation	Increase timeout or optimize query
Permission denied	Insufficient rights	Check file permissions or database privileges

Debug Mode

// Enable verbose logging
const executor = using(catalog, "my_runtime", {
  debug: true // Logs all spawn operations
});

const result = await executor.spawn({
  kind: "text",
  text: "...",
  verbose: true // Additional output
});

Validation Errors

// Catalog validation will throw on parse
try {
  const catalog = catalogFromYaml(invalidYaml);
} catch (error) {
  console.error("Catalog validation failed:", error.message);
  // Fix catalog configuration
}

Performance Considerations

Connection Pooling

// Reuse executors for better performance
const db = using(catalog, "my_db");

// Bad: Creating new executor each time
for (const query of queries) {
  const newDb = using(catalog, "my_db"); // Overhead!
  await newDb.spawn({ kind: "text", text: query });
}

// Good: Reuse executor
for (const query of queries) {
  await db.spawn({ kind: "text", text: query });
}

Batch Operations

// Execute multiple operations efficiently
const operations = [
  "INSERT INTO users VALUES (1, 'Alice')",
  "INSERT INTO users VALUES (2, 'Bob')",
  "INSERT INTO users VALUES (3, 'Charlie')"
];

// Combine into single execution when possible
const batchQuery = operations.join("; ");
await db.spawn({
  kind: "text",
  text: batchQuery
});

Deno Command API - Underlying process spawning
Zod Schema Validation - Catalog validation
YAML Specification - Catalog format reference

How is this guide?

Spawn Module

On this page