Deployment

Spooled Cloud can be deployed as a managed SaaS service or self-hosted on your own infrastructure. This guide covers all deployment options and production best practices.

Deployment Architecture

A production Spooled deployment consists of stateless API servers behind a load balancer, with PostgreSQL for persistence and Redis for real-time pub/sub.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#ecfdf5', 'primaryTextColor': '#065f46', 'primaryBorderColor': '#10b981', 'lineColor': '#6b7280'}}}%%
flowchart TB
  subgraph lb["Load Balancer"]
    LB["Nginx / Cloud LB"]
  end

  subgraph api["API Tier (Stateless)"]
    A1["Spooled API #1"]
    A2["Spooled API #2"]
    A3["Spooled API #N"]
  end

  subgraph data["Data Tier"]
    PG[(PostgreSQL)]
    RD[(Redis)]
  end

  subgraph workers["Worker Pool"]
    W1["Worker 1"]
    W2["Worker 2"]
    WN["Worker N"]
  end

  LB --> A1
  LB --> A2
  LB --> A3
  
  A1 --> PG
  A2 --> PG
  A3 --> PG
  
  A1 --> RD
  A2 --> RD
  A3 --> RD
  
  W1 --> LB
  W2 --> LB
  WN --> LB

Managed (spooled.cloud)

The easiest way to get started. Sign up at spooled.cloud/signup and start queuing jobs immediately.

• ✓ No infrastructure to manage
• ✓ Automatic scaling and updates
• ✓ Built-in monitoring and alerting
• ✓ 99.9% SLA (Pro and Enterprise plans)
• ✓ Security-focused architecture and operational monitoring

Self-Hosted: Docker

The quickest way to self-host Spooled for development or small deployments.

Quick Start

# Clone the repository
git clone https://github.com/spooled-cloud/spooled-backend.git
cd spooled-backend

# Copy example environment
cp .env.example .env

# Start with Docker Compose
docker compose up -d

# Check status
docker compose ps

# View logs
docker compose logs -f spooled-api

Docker Run (Single Container)

docker run -d \
  --name spooled-backend \
  -p 8080:8080 \    # REST API
  -p 9090:9090 \    # Prometheus metrics
  -p 50051:50051 \  # gRPC API (self-hosted) (HTTP/2 + Protobuf)
  -e DATABASE_URL="postgres://user:pass@host:5432/spooled" \
  -e REDIS_URL="redis://redis:6379" \
  -e JWT_SECRET="your-production-secret" \
  -e RUST_ENV="production" \
  spooled-backend:latest

Ports

Docker Compose Configuration

The docker-compose.yml file configures the following services:

• spooled-api — REST API on port 8080, gRPC API on port 50051 (self-hosted)
• postgres — PostgreSQL 16 database for queue storage
• pgbouncer — Connection pooler for production workloads
• redis — Redis 7 for pub/sub and rate limiting
• prometheus — Metrics collection
• grafana — Dashboards on port 3000

See the GitHub repository for the full docker-compose.yml.

Self-Hosted: Kubernetes

For production deployments at scale, use Kubernetes with our Kustomize manifests.

Prerequisites

• Kubernetes 1.24+
• kubectl configured
• Helm 3.10+ (optional, for dependencies)
• PostgreSQL 14+ (managed like RDS/Cloud SQL or self-hosted)
• Redis 7+ (managed like ElastiCache/Memorystore or self-hosted)

Installation

# Create namespace
kubectl create namespace spooled-cloud

# Create secrets
kubectl create secret generic spooled-secrets \
  --namespace spooled-cloud \
  --from-literal=database-url='postgres://user:pass@postgres:5432/spooled' \
  --from-literal=redis-url='redis://redis:6379' \
  --from-literal=jwt-secret='your-production-secret'

# Deploy with Kustomize
kubectl apply -k k8s/overlays/production

# Verify deployment
kubectl get pods -n spooled-cloud

Helm Chart

# Helm chart status
# The official Helm chart is not published yet.
# Track progress / updates in GitHub Discussions:
# https://github.com/orgs/Spooled-Cloud/discussions
#
# For Kubernetes today, use Docker Compose as a baseline and translate to:
# - Deployment for spooled-api (stateless)
# - Service + Ingress (WebSocket/SSE compatible)
# - Managed PostgreSQL + Redis (recommended)
#
# If you need a chart urgently, open a discussion and we'll prioritize it.

Configuration

Create a custom values.yaml to configure:

• replicaCount — Number of API replicas
• ingress — Ingress settings for external access
• resources — CPU and memory limits
• autoscaling — HPA configuration
• secrets — External secrets reference

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

Self-Hosted: Bare Metal

For maximum control, run Spooled directly on servers.

Build from Source

# Requirements: Rust 1.75+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Clone and build
git clone https://github.com/spooled-cloud/spooled-backend.git
cd spooled-backend
cargo build --release

# Binary at target/release/spooled-backend

System Service

Configure systemd to manage the Spooled service with automatic restart on failure. See the repository for example systemd unit files.

Reverse Proxy

Configure Nginx or Caddy as a reverse proxy with TLS termination. Ensure:

• WebSocket upgrade support for /api/v1/ws
• Buffering disabled for SSE at /api/v1/events
• Proper proxy headers (Host, X-Real-IP, X-Forwarded-For)
• gRPC proxying: expose your internal gRPC service on 443 (HTTP/2 required) in production

Cloudflare Tunnel for gRPC

When using Cloudflare Tunnel to expose your gRPC server, HTTPS is required for HTTP/2 connections. Cloudflare Tunnel does not support HTTP/2 over plaintext HTTP.

Environment Variables

Monitoring & Alerting

Prometheus Metrics

Metrics are exposed at /metrics on port 9090. Key metrics to monitor:

Recommended Alerts

Scaling Guidelines

Resource Guidelines

Database Scaling

Production Checklist

Database Requirements

PostgreSQL

• Version: 14+ (16 recommended)
• Extensions: pgcrypto (auto-enabled)
• Connections: At least 20 per API instance
• Storage: SSD recommended for queue performance

Redis

• Version: 7+ (for function support)
• Memory: At least 256MB for small deployments
• Persistence: RDB or AOF for durability
• HA: Redis Sentinel or Cluster for production

Troubleshooting

Database Connection Errors

• Verify DATABASE_URL is correct
• Check network connectivity to database
• Ensure database is accepting connections
• Check connection pool settings

Redis Connection Errors

• Verify REDIS_URL is correct
• Application will run without Redis (degraded mode)
• Real-time features require Redis

Slow Job Processing

• Check spooled_job_max_age_seconds metric
• Add more workers if queue is backing up
• Review job timeout settings
• Check for database bottlenecks