Durable job queues on PostgreSQL
Spooled stores every job in Postgres, retries failures with exponential backoff, and parks exhausted work in a dead-letter queue you can inspect and replay. Enqueue over REST or gRPC; run workers in any language. Idempotency keys stop duplicates.
See it in code ↓- 100
- jobs / bulk enqueue
- up to 100
- attempts before DLQ
- 6-field
- cron, second precision
- REST + gRPC
- APIs over HTTP/2
One request, up to 100 jobs
Exponential backoff, then dead-letter
Timezone-aware, with history
Scales to 100,000 jobs/day on Pro
Up and running in minutes
Three calls: enqueue, claim, complete. The same examples work in cURL, Node.js, Python, Go, and PHP.
Enqueue a job
POST a queue name and JSON payload. Pass an idempotency key and duplicate submissions collapse into one job. Bulk enqueue accepts up to 100 jobs per request.
Jobs & queues docs →import { SpooledClient } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
const userId = 'usr_123';
// Create a job
const { id } = await client.jobs.create({
queueName: 'email-notifications',
payload: {
to: 'user@example.com',
subject: 'Welcome!',
template: 'welcome',
},
idempotencyKey: `welcome-${userId}`,
maxRetries: 5,
});
console.log(`Created job: ${id}`);Run a worker
Workers run on your infrastructure. They claim jobs, run your code, and report complete or fail. Failures retry with exponential backoff — up to 100 attempts — then land in the dead-letter queue.
Worker docs →import { SpooledClient, SpooledWorker } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
const worker = new SpooledWorker(client, {
queueName: 'email-notifications',
concurrency: 10,
});
worker.process(async (ctx) => {
const { to, subject, body } = ctx.payload;
await sendEmail({ to, subject, body });
console.log(`Sent email to ${to}`);
return { sent: true };
});
await worker.start();Stream the results
Subscribe over Server-Sent Events or WebSocket instead of polling. The dashboard shows throughput, latency, failures, and DLQ replays from the same streams.
Real-time API docs →import { SpooledClient } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
// SSE realtime client (uses /api/v1/events with Authorization header under the hood)
const realtime = await client.realtime({ type: 'sse' });
realtime.on('job.created', (data) => {
console.log('job.created:', data);
});
await realtime.connect();
await realtime.subscribe({ queueName: 'orders' });Watch jobs flow through the queue
Spooled stores every job in Postgres. Your workers claim and process them in parallel, failures auto-retry, and the whole pipeline streams live. Here it is running.
Pipeline preview
DemoIllustrative motion. Pro enqueues at 100 requests/sec (burst 200) and retains completed and failed jobs for 30 days.
The call that feeds the queue
import { SpooledClient } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
const userId = 'usr_123';
// Create a job
const { id } = await client.jobs.create({
queueName: 'email-notifications',
payload: {
to: 'user@example.com',
subject: 'Welcome!',
template: 'welcome',
},
idempotencyKey: `welcome-${userId}`,
maxRetries: 5,
});
console.log(`Created job: ${id}`);Works with your stack
Any HTTP source in, workers in any language out.
- Stripe
- GitHub
- Shopify
- PostgreSQL
- Redis
- Node.js
- Python
- Go
- PHP
The safety net around every job
Queue mechanics you would otherwise build and babysit yourself. All of it on every plan, including Free.
Reliability
What happens when things fail
- Automatic retries Exponential backoff, configurable delays, up to 100 attempts per job.
- Dead-letter queue Exhausted jobs keep payload and error history; bulk retry or purge.
- Idempotency keys The same key enqueues once — redeliveries stop duplicating work.
- Rate limits you can see Per-plan API limits with clear errors; inspect usage via the API.
Control
How you steer the queue
- Bulk operations Enqueue up to 100 jobs per request; batch status in one call.
- Queue controls Pause and resume during deploys or incidents; boost priority live.
- Cron schedules 6-field expressions, second precision, timezone-aware, with history.
- Workflows Chain jobs with dependencies; parallel or sequence; retry end-to-end.
Visibility
How you watch it run
- Real-time streams WebSocket and SSE job updates, plus Prometheus metrics.
- Tags & filtering Tag jobs by customer, environment, or any dimension; filter later.
- REST + gRPC REST for integrations; gRPC HTTP/2 streaming for hot worker loops.
- Official SDKs Node.js, Python, Go, and PHP with typed clients and workers.
Watch a job retry, live
Failed jobs retry on their own with exponential backoff. Press play and follow one job through its retries into the dead-letter queue — the same walk your jobs take in production.
job a1f9c… · queue payment-processing
Ready — press play to run the retry flow.
-
Initial attempt
t = 0sWorker claims the job and runs it
Failed · HTTP 500 - 1
Retry #1
t + 1mFirst backoff delay — 1 minute
Failed · Timeout - 2
Retry #2
t + 3mDelay doubles — 2 minutes
Failed · Connection refused - 3
Retry #3
t + 7mDelay doubles again — 4 minutes
Failed · HTTP 503 -
Dead-letter queue
exhaustedPayload and error history preserved for inspect & replay
Parked · inspect & replay
Backoff schedule
delay_minutes = min(2 ** retry_count, 60)
Retries are scheduled in minutes (1m, 2m, 4m …), doubling each time and capped at 60m, for up to 100 attempts. Exhausted jobs land in the dead-letter queue with their payload and error history intact — inspect and replay them from the dashboard or the API.
Inspect and replay the dead-letter queue
Exhausted jobs are not lost. List them with their payloads and errors, fix the cause, then replay them — one at a time or in bulk.
// List jobs in dead-letter queue
const dlqJobs = await client.jobs.dlq.list({
queueName: 'payment-processing',
limit: 100,
});
for (const job of dlqJobs) {
console.log(`DLQ Job: ${job.id}, status: ${job.status}`);
console.log(`Retry count: ${job.retryCount}, created: ${job.createdAt}`);
}// Retry DLQ jobs
const result = await client.jobs.dlq.retry({
queueName: 'payment-processing',
limit: 50,
});
console.log(`Retried ${result.retriedCount} jobs`);Chain jobs into dependency graphs
Declare which jobs depend on which, and Spooled runs the graph for you. Roots go first, independent branches run in parallel, and a failed parent cancels everything downstream. Here is a workflow executing.
Workflow run
LiveThe workflow behind it
import { SpooledClient } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
// Create a workflow with job dependencies
const workflow = await client.workflows.create({
name: 'user-onboarding',
jobs: [
{
key: 'create-account',
queueName: 'users',
payload: { email: 'user@example.com', plan: 'pro' },
},
{
key: 'send-welcome-email',
queueName: 'emails',
dependsOn: ['create-account'], // Waits for this job
payload: { template: 'welcome' },
},
{
key: 'setup-defaults',
queueName: 'users',
dependsOn: ['create-account'], // Also waits
payload: { settings: {} },
},
],
});
console.log(`Created workflow: ${workflow.workflowId}`);Automatic ordering
Declare depends_on
and jobs run in topological order. No manual coordination.
Parallel branches
Independent jobs fan out and run at once across your workers, then rejoin downstream.
Failure cascades
If a parent exhausts its retries, its dependents are cancelled instead of running on bad state.
Illustrative motion. Workflows are a paid feature — Pro runs up to 25 concurrent workflows; the Free tier does not include them.
Urgent work jumps the queue
Give a job a priority and workers claim the highest first — high
(10)
before normal
(0)
before low
(-10).
Same enqueue call, one extra field.
Waiting to be claimed
6 jobs- VIP order High · +10
- Pager alert High · +10
- Send receipt Normal · 0
- Send invite Normal · 0
- Nightly cleanup Low · -10
- Rebuild report Low · -10
Processed in order
0 done- VIP order #1
- Pager alert #2
- Send receipt #3
- Send invite #4
- Nightly cleanup #5
- Rebuild report #6
Priority only orders the claim — every job still runs. On the Free plan up to 10 jobs run at once, so higher priority simply gets there first.
The one extra field
import { SpooledClient } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
// High priority - VIP customer order (processed first)
await client.jobs.create({
queueName: 'orders',
payload: { orderId: 789, customerTier: 'vip' },
priority: 10, // High priority
});
// Normal priority (default)
await client.jobs.create({
queueName: 'orders',
payload: { orderId: 790 },
priority: 0, // Default
});
// Low priority - background cleanup
await client.jobs.create({
queueName: 'maintenance',
payload: { task: 'cleanup' },
priority: -10, // Low priority
});
// Workers claim jobs: High → Normal → LowPut jobs on a schedule
Daily reports, subscription renewals, cleanup sweeps — describe when with a 6-field cron expression and Spooled enqueues the job for you, second-precise and timezone-aware. Pick a pattern and watch the next runs fill in.
Pick a schedule
Six fields: second, minute, hour, day-of-month, month, day-of-week.
Next runs
Computed- 1 Tomorrow, 9:00 AMin 6 hours
- 2 Fri, 9:00 AMin 1 day
- 3 Sat, 9:00 AMin 2 days
- 4 Sun, 9:00 AMin 3 days
Preview computed in your browser. Free includes 1 schedule, Pro allows 50 with 30-day job history.
The call that creates the schedule
// Create a cron schedule
const schedule = await client.schedules.create({
name: 'Daily Report',
cronExpression: '0 0 9 * * *',
timezone: 'America/New_York',
queueName: 'reports',
payloadTemplate: { type: 'daily_report' },
});Watch your workers claim jobs
Run your own workers wherever your code already lives. They claim jobs from Spooled, process them, and mark each complete or failed — in parallel, with no coordination. Row-level locking makes sure no two workers ever grab the same job.
Your worker pool
Claiming from queue emails
on api-eu-1
on api-us-2
on worker-box
Incoming queue
The loop your worker runs
- Claim a batch — Postgres row-level locking hands each job to exactly one worker.
- Do the work in your own code, on your own servers.
- Complete on success, or fail to retry with backoff, then dead-letter.
Scale out by starting more processes: Pro allows up to 25 workers across 50 queues. Spooled coordinates; you never write a scheduler.
import { SpooledClient, SpooledWorker } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
const worker = new SpooledWorker(client, {
queueName: 'email-notifications',
concurrency: 10,
});
worker.process(async (ctx) => {
const { to, subject, body } = ctx.payload;
await sendEmail({ to, subject, body });
console.log(`Sent email to ${to}`);
return { sent: true };
});
await worker.start();One worker, thousands of jobs a second
gRPC bidirectional streaming keeps a worker and Spooled talking over one open HTTP/2 connection, so a single worker can process thousands of jobs per second — no per-job round trip, no polling.
Publish once, stream everywhere
Enqueue a job over REST and Spooled fans it out in real time. Dashboards subscribe over Server-Sent Events, and clients that need two-way traffic connect over WebSocket. One publish, every subscriber updated. Here it is running.
Live pub / sub
Liveorders GET /api/v1/events GET /api/v1/ws Illustrative counters. Every publish is one enqueue against your 100 requests/sec Pro limit (burst 200); fan-out to subscribers is not rate-limited.
Subscribe over SSE
One-way stream, perfect for dashboards and live views. Reconnects on its own.
import { SpooledClient } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
// SSE realtime client (uses /api/v1/events with Authorization header under the hood)
const realtime = await client.realtime({ type: 'sse' });
realtime.on('job.created', (data) => {
console.log('job.created:', data);
});
await realtime.connect();
await realtime.subscribe({ queueName: 'orders' });Subscribe over WebSocket
Two-way channel when you need to send filters and commands back up the wire.
import { SpooledClient } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
// WebSocket realtime client (uses /api/v1/ws?token=... under the hood)
const realtime = await client.realtime({ type: 'websocket' });
realtime.on('job.created', (data) => {
console.log('job.created:', data);
});
realtime.on('job.completed', (data) => {
console.log('job.completed:', data);
});
await realtime.connect();
await realtime.subscribe({ queueName: 'orders' });See your queues live
The hosted dashboard subscribes to the very streams you saw fan out above — so what you watch here is exactly what your own clients receive, with no polling.
In the dashboard
Same streams, your clients
- Server-Sent Events One-way stream for dashboards and live views. Reconnects on its own.
- WebSocket Two-way channel when you need to send filters and commands back.
Common workloads, one queue
Webhook processing
Queue events from any source. For Stripe, GitHub, or Shopify, a small adapter verifies signatures and enqueues; idempotency keys absorb provider redeliveries.
In practice: Payment events, deploy-on-push, SMS replies
High-volume jobs
Bulk enqueue 100 jobs per request and check batch status in one call. Tag jobs to trace and filter them later.
In practice: Mass email, data migrations, bulk notifications
Scheduled tasks
Cron with second precision (6-field expressions), timezone-aware execution, manual triggering, and per-schedule history.
In practice: Daily reports, subscription renewals, cleanup
Multi-step workflows
Chain jobs with dependencies, run steps in parallel or sequence, and retry a failed workflow end-to-end with full visibility.
In practice: Order processing, onboarding, approval flows
Outgoing webhooks
Get notified the moment a job finishes. Register a global endpoint that fires for every job, or attach a per-job callback URL to a single enqueue — the same durable queue delivers both.
Global endpoint
One URL, called for every job your workers complete.
import { SpooledClient } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
// Setup webhook for job events
await client.webhooks.create({
name: 'Slack Notifications',
url: 'https://hooks.slack.com/...',
events: ['job.completed', 'job.failed', 'queue.paused'],
secret: 'your-hmac-secret', // For signature verification
});
// ✓ Spooled POSTs to your URL
// ✓ Automatic retries
// ✓ Delivery history in dashboardPer-job callback
A one-off URL attached to a single job at enqueue time.
import { SpooledClient } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
// Get notified when THIS job completes
await client.jobs.create({
queueName: 'exports',
payload: { reportId: 12345 },
completionWebhook: 'https://your-app.com/webhooks/export-done',
});
// When job completes, Spooled POSTs to your URL:
// POST https://your-app.com/webhooks/export-done
// {status: "completed", jobId: "...", result: {...}}Events you can subscribe to
-
job.created -
job.started -
job.completed -
job.failed -
job.cancelled -
queue.paused -
queue.resumed -
worker.registered -
worker.deregistered -
schedule.triggered
Division of labor
Spooled never runs your code. It stores jobs, schedules retries, and streams status; your workers do the work, wherever they already run.
Spooled handles
- Durable job storage in Postgres
- Retries with exponential backoff
- Deduplication via idempotency keys
- Dead-letter queues and replay
- Schedules, workflows, priorities
- Live status over SSE / WebSocket
You provide
- Workers — your code, your servers
- Email/SMS providers you already use
- Storage (S3, R2, GCS…)
- External APIs your jobs call
- The business logic itself
Your workers stay on your infrastructure and keep your secrets — Spooled only sees the job payloads you send it.
See it in real code
Three patterns you can copy today. Each pairs the enqueue side with the worker side — the same durable queue sits between them. Tap a tab, then expand any block to read the full snippet.
Reliable Stripe checkout
Ack the Stripe webhook in milliseconds, then let a durable job fulfil the order — retried automatically if fulfilment fails.
Receive the event
Verify the signature, enqueue a job, return 200 fast.
import { SpooledClient } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
// Express.js webhook handler
app.post('/webhooks/stripe', async (req, res) => {
const sig = req.headers['stripe-signature'];
try {
// Verify and parse the webhook
const event = stripe.webhooks.constructEvent(
req.body,
sig,
process.env.STRIPE_WEBHOOK_SECRET
);
// Queue in Spooled for reliable processing
await client.jobs.create({
queueName: 'stripe-events',
payload: event,
idempotencyKey: event.id,
});
// Return 200 immediately - Spooled handles retries
res.status(200).send('Queued');
} catch (error) {
res.status(400).send(`Webhook Error: ${error.message}`);
}
});Fulfil in a worker
Claim → do the work → complete or fail, on your own servers.
import { SpooledClient, SpooledWorker } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
const worker = new SpooledWorker(client, {
queueName: 'email-notifications',
concurrency: 10,
});
worker.process(async (ctx) => {
const { to, subject, body } = ctx.payload;
await sendEmail({ to, subject, body });
console.log(`Sent email to ${to}`);
return { sent: true };
});
await worker.start();Fan out a batch import
Turn one CSV upload into thousands of independent jobs — each row gets its own retry, idempotency key, and worker.
Enqueue per row
Read the file once, enqueue one job for every record.
import { SpooledClient } from '@spooled/sdk';
import fs from 'node:fs/promises';
const client = new SpooledClient({ apiKey: process.env.SPOOLED_API_KEY! });
const csv = await fs.readFile('users.csv', 'utf8');
const [headerLine, ...rows] = csv.trim().split(/\r?\n/);
const headers = headerLine.split(',').map((s) => s.trim());
for (const row of rows) {
if (!row.trim()) continue;
const values = row.split(',').map((s) => s.trim());
const payload: Record<string, string> = {};
headers.forEach((h, i) => (payload[h] = values[i] ?? ''));
await client.jobs.create({
queueName: 'csv-import',
payload,
idempotencyKey: payload.email ? `csv-${payload.email}` : undefined,
});
}
console.log('✅ Enqueued CSV jobs');Process each job
A minimal worker — swap the body for your real work.
import { SpooledClient, SpooledWorker } from '@spooled/sdk';
const client = new SpooledClient({ apiKey: process.env.SPOOLED_API_KEY! });
const worker = new SpooledWorker(client, {
queueName: 'my-queue',
concurrency: 1,
});
worker.process(async (ctx) => {
console.log('Job ID:', ctx.jobId);
console.log('Payload:', ctx.payload);
return { ok: true };
});
await worker.start();Push-to-deploy webhooks
A GitHub push enqueues a deploy job; a worker runs build, test, and release with retries instead of a flaky inline script.
On every push
Take the GitHub payload and enqueue a deploy job.
import { SpooledClient } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
// GitHub webhook handler
app.post('/webhooks/github', async (req, res) => {
const event = req.headers['x-github-event'];
const delivery = req.headers['x-github-delivery'];
// Queue for processing
await client.jobs.create({
queueName: 'github-events',
payload: {
event,
data: req.body,
},
idempotencyKey: delivery, // Use delivery ID for deduplication
});
res.status(200).send('Queued');
});Run the deploy
A worker claims the job and drives the pipeline.
import { SpooledClient, SpooledWorker } from '@spooled/sdk';
const client = new SpooledClient({
apiKey: process.env.SPOOLED_API_KEY!,
});
const worker = new SpooledWorker(client, {
queueName: 'email-notifications',
concurrency: 10,
});
worker.process(async (ctx) => {
const { to, subject, body } = ctx.payload;
await sendEmail({ to, subject, body });
console.log(`Sent email to ${to}`);
return { sent: true };
});
await worker.start();Rust core, Postgres durability, open end to end
The API and workers are Rust. PostgreSQL is the source of truth — every job, retry, and dead-letter entry is a row you could query, with row-level security isolating tenants. Redis carries only real-time pub/sub for live dashboards and streams; losing it never loses a job.
The core is Apache-2.0: backend, queue engine, and all four SDKs. Self-host it against your own Postgres with no platform limits, or use the hosted cloud and skip the operating. Same API either way.
How work flows through Spooled
Questions engineers ask
What is Spooled Cloud?
Is Spooled a job queue without Redis?
How do retries and the dead-letter queue work?
How does Spooled prevent duplicate jobs?
Does Spooled support cron and scheduled jobs?
Can I chain jobs into workflows?
Which languages have official SDKs?
Where does my worker code run?
Is Spooled open source? Can I self-host it?
How is Spooled different from BullMQ, Celery, or Sidekiq?
How do I watch jobs in real time?
Queue your first job today
The Free plan includes 1,000 jobs a day with retries, dead-letter queues, and real-time streams.
- No credit card required
- Free tier forever
- Open source — Apache-2.0