Architecture

Spooled is built for reliability, performance, and multi-tenant security. This guide explains the system architecture and design decisions.

System Overview

Spooled is a distributed job queue system built with Rust for maximum performance and safety. The architecture consists of several key layers:

flowchart TB
    subgraph clients["Client Layer"]
        SDK["SDKs<br/>Node / Python / Go / PHP"]
        HTTP["REST API<br/>OpenAPI 3.0"]
        GRPC["gRPC API"]
    end

    subgraph core["Core Services"]
        API["API Gateway<br/>Axum + Tower"]
        AUTH["Auth Service"]
        QUEUE["Queue Engine"]
        SCHED["Scheduler"]
        STREAM["Event Stream"]
    end

    subgraph storage["Data Layer"]
        PG[("PostgreSQL<br/>Jobs, org-scoped")]
        RD[("Redis<br/>Pub/Sub + Cache")]
    end

    subgraph workers["Worker Layer"]
        W1["Worker Pool 1"]
        W2["Worker Pool 2"]
        W3["Worker Pool N"]
    end

    subgraph observability["Observability"]
        PROM["Prometheus"]
        GRAF["Grafana"]
        LOGS["Structured Logs"]
    end

    SDK --> API
    HTTP --> API
    GRPC --> API
    API --> AUTH
    AUTH --> QUEUE
    QUEUE --> PG
    QUEUE --> RD
    SCHED --> QUEUE
    STREAM --> RD
    
    W1 --> API
    W2 --> API
    W3 --> API
    
    API --> PROM
    PROM --> GRAF

Core Components

Nodes from the system overview diagram, with technology, responsibility, and default ports (from backend config / compose; override via env).

Component	Technology	Responsibility	Port
API Gateway	Axum + Tower	REST (OpenAPI 3.0 / JSON), WebSocket/SSE, rate limiting, request validation	`8080` (`PORT`)
Auth Service	API keys, JWT, bcrypt	Validate credentials; bind `organization_id` into request context	— (in-process)
Queue Engine	PostgreSQL `FOR UPDATE SKIP LOCKED`	Job lifecycle, priority, leases, exponential backoff with jitter, scheduled jobs	— (in-process)
Scheduler	Tokio background tasks	Activate scheduled jobs, cron, lease recovery, dependencies, cleanup / retention	— (in-process)
Event Stream	Redis Pub/Sub	Real-time job status notifications to clients	— (via Redis)
gRPC Service	tonic (HTTP/2 + Protobuf)	Worker streaming (`StreamJobs`, `ProcessJobs`), health, reflection	`50051` (`GRPC_PORT`)
PostgreSQL	PostgreSQL (sqlx)	Durable job store; org-scoped queries; indexes for queue ops	`5432`
Redis	Redis	Pub/Sub, rate-limit counters, API key / org metadata cache	`6379`
Prometheus	prometheus crate	Metrics scrape endpoint for observability	`9090` (`METRICS_PORT`)

Scheduler task intervals

Task	Interval	Purpose
Activate scheduled jobs	5s	Move scheduled→pending
Process cron schedules	10s	Create jobs from cron
Recover expired leases	10s	Handle worker failures (secondary safety net)
Update job dependencies	10s	Unblock child jobs
Update metrics	15s	Refresh aggregate metrics
Cleanup stale workers	5m	Mark offline workers
Data retention	5m	Delete old data (same cleanup ticker)

Job Lifecycle

stateDiagram-v2
    [*] --> Pending: Enqueue
    Pending --> Processing: Worker claims
    Processing --> Completed: Success
    Processing --> Failed: Error
    Failed --> Pending: Retry
    Failed --> DeadLetter: Max retries
    Completed --> [*]
    DeadLetter --> Pending: Manual retry

Job States

State	Description
`pending`	Ready to be claimed by a worker
`processing`	Claimed by a worker, in progress
`completed`	Successfully processed
`failed`	Failed, will retry if retries remaining
`dead_letter`	Failed after exhausting all retries
`cancelled`	Manually cancelled

Multi-Tenant Security

Every API request is scoped to a single organization at the query layer: the organization id resolved from your API key is bound into every SQL statement.

flowchart LR
    subgraph request["Incoming Request"]
        TOKEN["API Key"]
    end

    subgraph auth["Authentication"]
        VALIDATE["Validate Key"]
        EXTRACT["Extract org_id"]
    end

    subgraph pg["Query Scoping"]
        SCOPE["org_id bound into every query"]
        DATA["Org's Data Only"]
    end

    TOKEN --> VALIDATE
    VALIDATE --> EXTRACT
    EXTRACT --> SCOPE
    SCOPE --> DATA

How Tenant Isolation Works

API key is validated and organization ID extracted
The request context carries that organization id end to end
Every query filters by organization_id — reads and writes alike
This scoping is exercised directly by our security audits

Security Layers

1. Network Security

TLS, Firewall, Network Policies

2. Rate Limiting

Per-IP, Per-API-Key, Per-Queue

3. Authentication

API Keys, JWT, bcrypt hashing

4. Authorization

Org-scoped queries, Queue ACLs

5. Input Validation

Schema validation, Sanitization

6. Security Headers

CSP, HSTS, X-Frame-Options

Retry Mechanism

Failed jobs automatically retry with configurable exponential backoff. The retry system ensures reliable delivery while preventing thundering herd problems.

sequenceDiagram
    participant W as Worker
    participant S as Spooled
    participant DLQ as Dead Letter Queue

    W->>S: Claim job
    S-->>W: Job data
    W->>W: Process (fails)
    W->>S: Fail job
    S->>S: Check retry count
    alt retries remaining
        S->>S: Schedule retry (backoff)
        Note over S: Wait 2^n seconds
        S-->>W: Job available again
    else max retries exceeded
        S->>DLQ: Move to DLQ
        Note over DLQ: Manual review
    end

Backoff Formula

Default backoff uses exponential delay with jitter:

delay = min(base_delay * 2^attempt + random_jitter, max_delay)

Where:

base_delay = 1 second
max_delay = 1 hour
random_jitter = 0-500ms

Scalability

flowchart TB
    LB["Load Balancer"]
    
    subgraph backends["Backend Pods"]
        B1["Backend 1"]
        B2["Backend 2"]
        B3["Backend N"]
    end
    
    subgraph data["Data Layer"]
        PGB["PgBouncer<br/>(Connection Pool)"]
        PG[("PostgreSQL<br/>Primary")]
        RD[("Redis<br/>Cluster")]
    end
    
    LB --> B1
    LB --> B2
    LB --> B3
    
    B1 --> PGB
    B2 --> PGB
    B3 --> PGB
    B1 --> RD
    B2 --> RD
    B3 --> RD
    
    PGB --> PG

Connection Pooling

PgBouncer in transaction mode
Each backend: 25 connections to bouncer
Bouncer: 100 connections to PostgreSQL
Result: 10 backends × 25 = 250 virtual connections

Caching Strategy

Data	Cache	TTL
API Keys	Redis	1 hour
Queue Config	Redis	5 minutes
Job Status (hot)	Redis	30 seconds
Rate Limit Counters	Redis	Per window

Performance Characteristics

10,000+

Jobs per second per node

<50ms

P99 enqueue latency

Isolated

Per-tenant scoping on every query

Rust

Memory-safe, zero-cost abstractions

Benchmarks

Operation	P50	P95	P99
Health Check	1ms	5ms	10ms
Job Enqueue	5ms	20ms	50ms
Job Dequeue	10ms	30ms	100ms
Job Complete	5ms	15ms	30ms
List Jobs (100)	20ms	50ms	100ms

Capacity Planning

Component	1K req/s	10K req/s	100K req/s
Backend Pods	2	10	50
PostgreSQL	1 primary	1 primary + 2 read	Cluster
Redis	1 node	Sentinel	Cluster
PgBouncer	1	2	4

Request Limits

Resource	Limit
Request body	5MB
Job payload	1MB
gRPC payload	1MB
Webhook payload	5MB
List page size	100
Bulk enqueue	100 jobs

Deployment Options

Option	Best For	Maintenance
Managed Cloud	Most teams	Zero maintenance
Docker Compose	Development, small deployments	Basic ops required
Kubernetes/Helm	Large scale, air-gapped	Full ops team

Next Steps

Deployment guide — Self-hosting instructions
API reference — Complete endpoint documentation
Real-time API — WebSocket, SSE, and gRPC streaming
Open source — Contributing and licensing