Retries & Dead-Letter Queue

Job failures are inevitable. Spooled provides automatic retries with exponential backoff and a dead-letter queue for jobs that can't be processed.

Related guides:

Jobs & Queues Building workers

How Retries Work

When a job fails, Spooled automatically schedules a retry with exponential backoff. This prevents overwhelming downstream services and gives transient failures time to resolve.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#fef3c7', 'primaryTextColor': '#92400e', 'primaryBorderColor': '#f59e0b', 'lineColor': '#6b7280'}}}%%
stateDiagram-v2
    [*] --> pending
    pending --> claimed: Worker claims
    claimed --> completed: Success
    claimed --> failed: Error
    failed --> pending: Retry (attempts left)
    failed --> dlq: Max retries exceeded
    dlq --> pending: Manual replay
    completed --> [*]

Retry Flow

Worker claims a job and attempts to process it
Processing fails (exception, timeout, or explicit failure)
Spooled checks if retry attempts remain
If retries remain: job is scheduled for later with backoff delay
If max retries exceeded: job moves to dead-letter queue (DLQ)

Backoff Strategies

By default, Spooled uses exponential backoff with jitter. This spreads out retries and prevents thundering herd problems.

Default Exponential Backoff

With default settings (base=1s, max=1h):

Attempt	Delay	Cumulative Time
1	1s	1s
2	2s	3s
3	4s	7s
4	8s	15s
5	16s	31s
6	32s	~1 min
7	64s	~2 min
8	128s	~4 min
9	256s	~8 min
10	512s	~17 min

Available Strategies

Exponential (default) — Delay doubles each attempt: 1s, 2s, 4s, 8s...
Linear — Constant delay increase: 1s, 2s, 3s, 4s...
Fixed — Same delay every time: 5s, 5s, 5s, 5s...
Custom — Provide your own delay function

Retry Configuration

Configure retry behavior when creating jobs:

Job with retry configuration

curl -X POST https://api.spooled.cloud/api/v1/jobs \
  -H "Authorization: Bearer sp_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "queue_name": "my-queue",
    "payload": {
      "event": "user.created",
      "user_id": "usr_123",
      "email": "alice@example.com"
    },
    "idempotency_key": "user-created-usr_123"
  }'

import { SpooledClient } from '@spooled/sdk';

const client = new SpooledClient({
  apiKey: process.env.SPOOLED_API_KEY!,
});

const userId = 'usr_123';

// Create a job
const { id } = await client.jobs.create({
  queueName: 'email-notifications',
  payload: {
    to: 'user@example.com',
    subject: 'Welcome!',
    template: 'welcome',
  },
  idempotencyKey: `welcome-${userId}`,
  maxRetries: 5,
});

console.log(`Created job: ${id}`);

from spooled import SpooledClient
import os

client = SpooledClient(api_key=os.environ["SPOOLED_API_KEY"])

image_id = "img_123"

# Create a background job
result = client.jobs.create({
    "queue_name": "image-processing",
    "payload": {
        "image_url": "https://example.com/image.jpg",
        "operations": ["resize", "compress"],
        "output_format": "webp"
    },
    "idempotency_key": f"process-image-{image_id}",
    "max_retries": 3
})

print(f"Created job: {result.id}")

client.close()

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/spooled-cloud/spooled-sdk-go/spooled"
	"github.com/spooled-cloud/spooled-sdk-go/spooled/resources"
)

func ptr[T any](v T) *T { return &v }

func main() {
	client, err := spooled.NewClient(spooled.WithAPIKey(os.Getenv("SPOOLED_API_KEY")))
	if err != nil {
		panic(err)
	}

	resp, err := client.Jobs().Create(context.Background(), &resources.CreateJobRequest{
		QueueName:      "my-queue",
		Payload:        map[string]any{"key": "value"},
		IdempotencyKey: ptr("unique-key"),
		MaxRetries:     ptr(3),
	})
	if err != nil {
		panic(err)
	}

	fmt.Printf("Created job: %s\n", resp.ID)
}

<?php

use Spooled\SpooledClient;
use Spooled\Config\ClientOptions;

$client = new SpooledClient(new ClientOptions(
    apiKey: getenv('SPOOLED_API_KEY'),
));

$userId = 'usr_123';

// Create a job
$job = $client->jobs->create([
    'queue' => 'email-notifications',
    'payload' => [
        'to' => 'user@example.com',
        'subject' => 'Welcome!',
        'template' => 'welcome',
    ],
    'idempotencyKey' => "welcome-{$userId}",
    'maxRetries' => 5,
]);

echo "Created job: {$job->id}\n";

Configuration Options

Option	Type	Default	Description
`max_retries`	number	settings default (typically 3)	Maximum retry attempts before moving to DLQ. Omitted fields use server queue defaults. REST allows explicit `0`; gRPC maps `<=0` to the settings default.
`timeout_seconds`	number	settings default (typically 300)	Job timeout (seconds). Same omit→defaults behavior on REST, gRPC, schedules, and incoming webhooks.
`priority`	number	0	Job priority (-100 to 100)

Dead-Letter Queue (DLQ)

Jobs that exhaust all retry attempts land in the dead-letter queue. The DLQ preserves the full job payload and failure history for debugging and replay.

Dashboard Tip

Dashboard → Jobs → Dead Letter Queue

What to look for:

→ Job payload and metadata
→ Last error message
→ Retry count and failure history
→ Original queue name

Actions:

Retry individual jobs
Bulk retry all DLQ jobs
Purge old failed jobs

List DLQ Jobs

# List jobs in dead-letter queue
curl -X GET "https://api.spooled.cloud/api/v1/jobs/dlq?queue_name=my-queue&limit=100" \
  -H "Authorization: Bearer sp_live_YOUR_API_KEY"

// List jobs in dead-letter queue
const dlqJobs = await client.jobs.dlq.list({
  queueName: 'payment-processing',
  limit: 100,
});

for (const job of dlqJobs) {
  console.log(`DLQ Job: ${job.id}, status: ${job.status}`);
  console.log(`Retry count: ${job.retryCount}, created: ${job.createdAt}`);
}

# List jobs in dead-letter queue
dlq_jobs = client.jobs.dlq.list({
    "queue_name": "payment-processing",
    "limit": 100
})

for job in dlq_jobs:
    print(f"DLQ Job: {job.id}, status: {job.status}")
    print(f"Retry count: {job.retry_count}, created: {job.created_at}")

import (
	"context"
	"github.com/spooled-cloud/spooled-sdk-go/spooled"
	"github.com/spooled-cloud/spooled-sdk-go/spooled/resources"
)

client, _ := spooled.NewClient(spooled.WithAPIKey("sp_live_YOUR_API_KEY"))

dlqJobs, err := client.Jobs().DLQ().List(context.Background(), &resources.ListDLQJobsParams{
	QueueName: stringPtr("payment-processing"),
	Limit:     intPtr(100),
})
if err != nil {
	panic(err)
}

for _, job := range dlqJobs {
	fmt.Printf("DLQ Job: %s, status: %s\n", job.ID, job.Status)
}

<?php

// List jobs in dead-letter queue
$dlqJobs = $client->jobs->dlq->list([
    'queue' => 'payment-processing',
    'limit' => 100,
]);

foreach ($dlqJobs as $job) {
    echo "DLQ Job: {$job->id}, status: {$job->status}\n";
    echo "Retry count: {$job->retryCount}, created: {$job->createdAt}\n";
}

Retry DLQ Jobs

# Retry jobs from DLQ
curl -X POST https://api.spooled.cloud/api/v1/jobs/dlq/retry \
  -H "Authorization: Bearer sp_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "queue_name": "my-queue",
    "limit": 50
  }'

import (
	"context"
	"github.com/spooled-cloud/spooled-sdk-go/spooled"
	"github.com/spooled-cloud/spooled-sdk-go/spooled/resources"
)

client, _ := spooled.NewClient(spooled.WithAPIKey("sp_live_YOUR_API_KEY"))

result, err := client.Jobs().DLQ().Retry(context.Background(), &resources.RetryDLQJobsRequest{
	QueueName: "payment-processing",
	Limit:     intPtr(50),
})
if err != nil {
	panic(err)
}

fmt.Printf("Retried %d jobs\n", result.RetriedCount)

Purge DLQ

# Purge DLQ jobs (requires confirm: true)
curl -X POST https://api.spooled.cloud/api/v1/jobs/dlq/purge \
  -H "Authorization: Bearer sp_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "queue_name": "my-queue",
    "confirm": true
  }'

// Purge old DLQ jobs
const result = await client.jobs.dlq.purge({
  queueName: 'my-queue',
  olderThan: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000), // 7 days
  confirm: true,
});

console.log(`Purged ${result.purgedCount} jobs`);

from datetime import datetime, timedelta

# Purge old DLQ jobs
result = client.jobs.dlq.purge({
    "queue_name": "my-queue",
    "older_than": (datetime.utcnow() - timedelta(days=7)).isoformat() + "Z",
    "confirm": True
})

print(f"Purged {result.purged_count} jobs")

import (
	"context"
	"github.com/spooled-cloud/spooled-sdk-go/spooled"
	"github.com/spooled-cloud/spooled-sdk-go/spooled/resources"
)

client, _ := spooled.NewClient(spooled.WithAPIKey("sp_live_YOUR_API_KEY"))

result, err := client.Jobs().DLQ().Purge(context.Background(), &resources.PurgeDLQJobsRequest{
	QueueName: "my-queue",
	Confirm:   true,
})
if err != nil {
	panic(err)
}

fmt.Printf("Purged %d jobs\n", result.PurgedCount)

<?php

use DateTime;

// Purge old DLQ jobs
$result = $client->jobs->dlq->purge([
    'queue' => 'my-queue',
    'olderThan' => (new DateTime('-7 days'))->format(DateTime::ATOM),
    'confirm' => true,
]);

echo "Purged {$result->purgedCount} jobs\n";

DLQ Best Practices

• Set up alerts when jobs enter the DLQ
• Review DLQ jobs regularly (daily for critical queues)
• Fix the root cause before replaying jobs
• Consider archiving old DLQ jobs to cold storage

Handling Failures in Workers

Workers should explicitly mark jobs as failed with a reason. This helps with debugging and determines retry behavior.

Failing a job

# Fail a job (will retry if retries remaining; echo lease_id from claim)
curl -X POST https://api.spooled.cloud/api/v1/jobs/job_xyz123/fail \
  -H "Authorization: Bearer sp_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "worker_id": "worker-1",
    "lease_id": "lease_abc",
    "error": "Connection timeout"
  }'

// Fail the job (will retry with exponential backoff)
await client.jobs.fail(job.id, {
  workerId: 'worker-1',
  leaseId: job.leaseId ?? undefined,
  error: 'Connection timeout to downstream service',
});

import (
	"context"
	"github.com/spooled-cloud/spooled-sdk-go/spooled"
	"github.com/spooled-cloud/spooled-sdk-go/spooled/resources"
)

client := spooled.NewClient(spooled.WithAPIKey("sp_live_YOUR_API_KEY"))

err := client.Jobs().Fail(context.Background(), jobID, &resources.FailJobRequest{
	WorkerID: stringPtr("worker-1"),
	LeaseID:  job.LeaseID,
	Error:    stringPtr("Connection timeout to downstream service"),
})
if err != nil {
	panic(err)
}

fmt.Println("Job failed and will retry")

<?php

// Fail the job (will retry with exponential backoff)
$client->jobs->fail($job->id, [
    'workerId' => 'worker-1',
    'leaseId' => $job->leaseId,
    'error' => 'Connection timeout to downstream service',
]);

Debug Failed Jobs

Dashboard → Jobs → Failed

What to look for:

→ Error message in last_error field
→ Retry count vs max_retries
→ Job payload for invalid data
→ Timestamps to correlate with logs

Actions:

Check worker logs for stack traces
Verify external service availability
Test payload manually

Next Steps

Jobs & queues — Understand the job lifecycle
Building workers — Production worker patterns
Real-time updates — Monitor job status in real-time