Troubleshooting

Find answers to common questions and solutions to frequently encountered issues. If you can't find what you're looking for, contact support.

Jobs Not Processing

Jobs stay in "pending" status

Symptoms: Jobs are created successfully but never move to "processing" state.

Common causes:

• No workers running: Ensure you have at least one worker connected to the queue
• Workers on different queue: Verify workers are listening to the correct queue name
• Workers paused: Check if the queue or workers are paused in the dashboard
• Scheduled for future: Jobs with scheduled_at won't process until that time

How to diagnose:

# Check if workers are registered
curl -H "Authorization: Bearer YOUR_API_KEY" \
  https://api.spooled.cloud/api/v1/workers

# Check job details
curl -H "Authorization: Bearer YOUR_API_KEY" \
  https://api.spooled.cloud/api/v1/jobs/JOB_ID

Jobs stuck in "processing" state

Symptoms: Jobs move to processing but never complete.

Common causes:

• Worker crashed: Worker died without completing the job
• Long-running job: Job is taking longer than expected
• Heartbeat timeout: Worker didn't send heartbeats (lease expired)

Solution: Jobs with expired leases are automatically returned to pending. If this happens frequently:

• Increase heartbeat frequency in your worker
• Implement proper heartbeating for long-running jobs
• Check worker logs for crashes

Authentication Errors

401 Unauthorized

Symptoms: API requests return 401 Unauthorized.

Common causes:

• Missing API key: Ensure Authorization: Bearer YOUR_API_KEY header is present
• Invalid API key: Key may be revoked or incorrectly copied
• Wrong environment: Using test key with production endpoint or vice versa

Quick check:

# Verify your API key works
curl -H "Authorization: Bearer YOUR_API_KEY" \
  https://api.spooled.cloud/api/v1/health

403 Forbidden

Symptoms: API requests return 403 Forbidden.

Common causes:

• Organization mismatch: Accessing resources from a different organization
• Insufficient permissions: API key doesn't have required scope
• Rate limited: Too many failed authentication attempts

JWT Token Expired

Symptoms: Dashboard sessions expire frequently, workers can't authenticate.

Solution:

• For dashboard: Clear cookies and log in again
• For workers: Ensure workers refresh tokens before expiration
• Check server clock synchronization (NTP)

Webhook Issues

Webhooks not being received

Symptoms: Jobs complete but webhook notifications never arrive.

Checklist:

• Verify webhook URL is correctly configured in organization settings
• Ensure your endpoint is publicly accessible (not localhost)
• Check that your endpoint returns 2xx status code
• Verify SSL certificate is valid (we don't deliver to invalid certs in production)

Webhook signature validation failing

Symptoms: Receiving webhooks but signature verification fails.

Common causes:

• Wrong secret: Using incorrect webhook secret for verification
• Encoding issues: Not using raw request body for signature verification
• Timestamp tolerance: Clock drift causing timestamp validation to fail

Correct verification pattern:

// Node.js example
const crypto = require('crypto');

function verifyWebhook(payload, signature, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(payload, 'utf8')
    .digest('hex');

  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expected)
  );
}

Webhook delivery timeouts

Symptoms: Webhooks marked as failed with timeout errors.

Solution:

• Ensure your endpoint responds within 10 seconds
• Process webhooks asynchronously (respond 200 immediately, process later)
• Check your server isn't overloaded

Worker Problems

Workers disconnecting frequently

Symptoms: Workers show as unhealthy, reconnecting constantly.

Common causes:

• Network instability: Unstable connection to Spooled Cloud
• Resource exhaustion: Worker running out of memory/CPU
• Firewall issues: gRPC connections being terminated

Solutions:

• Implement reconnection logic with exponential backoff
• Monitor worker resource usage
• Ensure firewalls allow long-lived HTTP/2 connections

Workers not receiving jobs

Symptoms: Worker is connected but never receives jobs.

Checklist:

• Verify worker is subscribed to the correct queue name(s)
• Check there are pending jobs in that queue
• Ensure worker concurrency is set correctly (not 0)
• Verify worker health status in dashboard

Duplicate job processing

Symptoms: Same job processed multiple times.

Causes and solutions:

• Worker crashed during processing: Implement idempotent job handlers
• Network partition: Job lease expired, use longer lease durations
• Missing acknowledgment: Always complete or fail jobs explicitly

Rate Limiting

429 Too Many Requests

Symptoms: API requests return 429 status code.

Default limits:

• API requests: 100 requests/second per API key
• Burst: Up to 200 requests in a burst

Solutions:

• Implement exponential backoff on 429 responses
• Use bulk operations (e.g., bulkEnqueue) instead of individual requests
• Contact support for higher limits if needed

Handling rate limits:

// Exponential backoff example
async function apiCallWithRetry(fn, maxRetries = 5) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (error.status === 429 && i < maxRetries - 1) {
        const delay = Math.pow(2, i) * 1000; // 1s, 2s, 4s, 8s, 16s
        await new Promise((r) => setTimeout(r, delay));
        continue;
      }
      throw error;
    }
  }
}

Common Error Codes

Debugging Tips

Enable verbose logging

For SDK debugging, enable verbose mode:

// Node.js SDK
const client = new SpooledClient({
  apiKey: 'YOUR_API_KEY',
  debug: true,
});

# Python SDK
import logging

logging.basicConfig(level=logging.DEBUG)
client = SpooledClient(api_key='YOUR_API_KEY')

Use the Dashboard

The Spooled Dashboard provides:

• Real-time job status and history
• Worker health monitoring
• Queue metrics and throughput graphs
• DLQ management for failed jobs
• Webhook delivery logs

Still Stuck?

When contacting support, please include:

• Your organization ID (visible in Dashboard)
• Relevant job IDs or timestamps
• Error messages (full response if possible)
• Steps to reproduce the issue
• SDK version (if using an SDK)