Skip to content

Jobs & Queues

Jobs are the fundamental building blocks of Spooled. Learn how to queue, process, and manage asynchronous work reliably at any scale.

Job Lifecycle

Every job follows a predictable lifecycle from creation to completion. Understanding this lifecycle is key to building reliable systems. 

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#ecfdf5', 'primaryTextColor': '#065f46', 'primaryBorderColor': '#10b981', 'lineColor': '#6b7280'}}}%%
stateDiagram-v2
    [*] --> pending: Job created
    pending --> claimed: Worker claims
    claimed --> completed: Success
    claimed --> failed: Error
    failed --> pending: Retry
    failed --> dlq: Max retries exceeded
    dlq --> pending: Replay
    completed --> [*]

Job States

State Description Transitions
pending Job is waiting to be processed → claimed (by worker)
claimed Worker is processing the job → completed, failed
completed Job finished successfully Terminal state
failed Job failed, may retry → pending (retry), dlq
dlq Job in dead-letter queue → pending (replay)

Creating Jobs

Note: Code examples use SDK syntax (under development). For production today, use the equivalent REST API endpoints.

Use the enqueue method to create jobs. Each job belongs to a queue and carries a payload that your workers will process. 

const job = await client.jobs.enqueue({
  queue: 'order-processing',
  payload: {
    order_id: 'ord_123',
    customer_email: 'alice@example.com',
    items: [
      { sku: 'PROD-001', quantity: 2 },
      { sku: 'PROD-042', quantity: 1 }
    ],
    total_amount: 9999
  },
  idempotencyKey: 'process-order-ord_123',
  maxRetries: 5,
  scheduledFor: null, // Process immediately
  priority: 0,        // Default priority (higher = processed first)
});

Job Properties

Property Type Description
queue string Queue name (alphanumeric + hyphens)
payload object JSON payload (up to 1MB on paid plans)
idempotencyKey string Prevents duplicate jobs (optional but recommended)
maxRetries number Max retry attempts (default: 3)
scheduledFor Date When to process (null = immediately)
priority number Higher values processed first (default: 0)

Claiming & Processing Jobs

Workers claim jobs to process them. Claimed jobs are locked for a visibility timeout to prevent duplicate processing. 

const jobs = await client.jobs.claim({
  queue: 'order-processing',
  limit: 10,           // Claim up to 10 jobs
  visibility: 30,      // Lock for 30 seconds
  filter: {
    priority: { gte: 5 }  // Optional: only high-priority jobs
  }
});

for (const job of jobs) {
  try {
    await processOrder(job.payload);
    await job.complete({ result: { processed: true } });
  } catch (error) {
    await job.fail({ reason: error.message });
  }
}

Important: Visibility Timeout

If a worker doesn't complete or fail a job within the visibility timeout, the job becomes available for other workers. Set this higher than your expected processing time.

Batch Operations

Queue multiple jobs in a single API call for better performance. All jobs in a batch share the same idempotency key scope. 

const jobs = await client.jobs.enqueueBatch([
  { queue: 'notifications', payload: { type: 'email', to: 'a@example.com' } },
  { queue: 'notifications', payload: { type: 'sms', to: '+1234567890' } },
  { queue: 'notifications', payload: { type: 'push', token: 'abc123' } },
], {
  idempotencyKey: 'notify-order-123',
});

Scheduled Jobs

Schedule jobs for future execution. Perfect for reminders, delayed notifications, or recurring tasks. 

// Send reminder in 24 hours
await client.jobs.enqueue({
  queue: 'reminders',
  payload: { userId: 'usr_123', type: 'cart-abandoned' },
  scheduledFor: new Date(Date.now() + 24 * 60 * 60 * 1000),
  idempotencyKey: 'reminder-usr_123-cart',
});

Idempotency

Idempotency keys prevent duplicate job processing. When you enqueue a job with an idempotency key:

  • If no job exists with that key, a new job is created
  • If a job exists and is still pending/processing, the existing job is returned
  • If a job exists and completed, the completed job is returned (no duplicate)

Best Practice

Always use meaningful idempotency keys derived from your business logic, like process-order-{orderId} or send-welcome-email-{userId}.

Working with Queues

Queues are logical groupings of related jobs. Use separate queues for different job types to enable independent scaling and monitoring.

Queue Naming

  • Use lowercase letters, numbers, and hyphens
  • Maximum 64 characters
  • Be descriptive: payment-processing, email-notifications

Recommended Queue Structure

  • High-priority queue — Critical operations (payments, security alerts)
  • Default queue — Standard background jobs
  • Low-priority queue — Analytics, cleanup, non-urgent tasks

Next Steps