Background Jobs and Long-Running Operations

A background job moves work that does not fit the user request's latency budget, or that can be retried, to queues and workers. Saying only “started” is not enough: the contract also needs a job ID, state, retries, progress, and a result-delivery mechanism.

Quick Decision

Situation	Approach	Watch out
Short work with an immediate result	Synchronous request	Timeout budget
Long or retryable work	Queue + worker	Idempotency and DLQ
Regularly triggered work	Scheduler + durable job record	Overlap and missed runs
User needs progress	Job status + progress event	Polling and notification load

Production Checklist

• Problem: Why is the work leaving the request path, and when is success final?
• Solution: Are job ID, state machine, retry, timeout, priority, progress, and ownership defined?
• Trade-off: The user response becomes faster, but eventual consistency and worker operations are added.
• Failure mode: Handle duplicate execution, stuck jobs, poison input, scheduler overlap, and worker loss.
• Measurement: Track queue age, job duration, success/failure, retries, active workers, and completion SLO.
• Security/cost: Minimize job payloads; enforce tenant authorization and worker concurrency.

Job Lifecycle

Suggested states are queued, running, succeeded, failed, retrying, cancelled, and expired. Allow only valid transitions; use a version or optimistic lock so two workers cannot claim the same job.

An endpoint starting long work can return 202 Accepted and Location: /jobs/{id}:

{
  "id": "job_123",
  "status": "running",
  "progress": { "completed": 40, "total": 100 },
  "statusUrl": "/api/jobs/job_123"
}

The status endpoint must enforce authorization. Deliver completion through polling, webhook, or push notification; document event IDs and duplicate behavior for each.

Worker Pools and Scheduling

Measure worker concurrency for CPU and I/O workloads. Unlimited threads create connection-pool exhaustion rather than throughput. Bound queue capacity and define rejection/backpressure behavior.

With Spring @Scheduled or a durable scheduler, document timezone, misfire, leader election, idempotent execution, and overlapping runs:

@Scheduled(cron = "0 */5 * * * *", zone = "UTC")
public void enqueueDueJobs() {
    jobRepository.findDueJobs(Instant.now())
        .forEach(job -> queue.publish(new JobCommand(job.id())));
}

The scheduler should create triggers; job state belongs in a queue and durable database. With multiple instances, use a unique claim or distributed lock to avoid enqueuing a job twice.

Retry, Priority, and SLA

Use bounded exponential backoff with jitter for transient errors. Do not retry permanent validation errors; quarantine poison jobs in a DLQ. Preserve the same idempotency key so a retry is not treated as a new business job.

A priority queue can reduce critical-job wait time, but low-priority work may starve; use aging or reserved capacity. Do not let tenants or users set unlimited priority.

Split the job SLA into submission-to-completion and queue-wait time. Check downstream API/database budgets and cost before increasing workers.

Progress and Notifications

Use a percentage only when total work is reliable. Otherwise use phase, completed, estimated, and updatedAt. Publishing progress too frequently increases broker, database, and client cost, so throttle it.

Store the completion result as an immutable artifact or resource; the status endpoint must not repeat expensive work on every read. Return stable error codes instead of technical stack traces.

Reliability and Operations

Test worker restarts, broker redelivery, database timeouts, graceful shutdown, and scheduler-leader loss. Stuck jobs need lease expiry, heartbeats, and operator replay/cancel procedures.

See Messaging Patterns and Async Processing for message, retry, and DLQ details.