Automation, Workers, and Event-Driven Workflows

The Constraint: Why Synchronous Processing Fails

Every HTTP request in a Laravel application runs inside a process with finite memory, a configured timeout, and a user waiting on the other end. For most requests (fetching a page, saving a form, returning JSON from an API), the lifecycle is fine. The work finishes in milliseconds. The problems start when the work is slow, unpredictable, or both.

PDF generation for a complex invoice might take 3 seconds or 30, depending on how many line items it contains. Sending a batch of 500 emails through an SMTP relay takes as long as the relay takes. Importing a CSV with 50,000 rows means 50,000 database writes, each with validation, event dispatch, and potential webhook callbacks to external systems.

Running any of these inside a web request creates three categories of failure.

The naive fix is to increase timeouts, raise memory limits, and add more workers. This delays the problem. It does not fix it. The fix is to move slow work out of the request lifecycle entirely: accept the request, dispatch a job to a queue, return a response immediately, and let a separate worker process handle the heavy lifting. Background jobs are not a feature. They are infrastructure.

Queue Architecture and Driver Selection

Laravel's queue system abstracts the transport layer behind a consistent API. You dispatch jobs the same way regardless of whether they end up in Redis, Amazon SQS, a database table, or Beanstalkd. The driver choice matters for operations, not for application code.

Redis (the default for production)

Redis is the most common queue driver for Laravel applications, and for good reason. It is fast (sub-millisecond latency in typical configurations), supports blocking pops (workers do not poll), and integrates with Laravel Horizon for monitoring and metrics. We use Redis for the majority of our queue infrastructure.

The trade-off: Redis is an in-memory store. If the Redis instance restarts without persistence configured, queued jobs disappear. In production, this means either Redis with AOF persistence, Redis Sentinel for failover, or a managed service like AWS ElastiCache with Multi-AZ.

Amazon SQS

SQS is the right choice when durability matters more than latency. Messages are replicated across multiple availability zones. You do not manage the infrastructure. The trade-off is that SQS does not support blocking pops (workers poll), message ordering is best-effort (FIFO queues exist but add complexity), and Horizon does not support SQS. We use SQS for jobs where message loss is unacceptable, such as webhook-triggered jobs from payment providers: you cannot afford to lose a Stripe webhook because Redis restarted.

Database queues

The database driver stores jobs in a jobs table. No additional infrastructure is needed. For applications with low job volume (fewer than 1,000 jobs per day), this is a reasonable starting point. It fails at scale because every job dispatch and every job pickup is a database write, competing with your application's transactional queries for connection pool capacity. On PostgreSQL, we have seen connection pool contention begin at around 5,000 jobs per day, though the exact threshold depends on your application's transactional load and pool size.

Driver comparison at a glance

Queue topology

A single default queue is where most applications start. It is also where most applications should stop pretending they have thought about queue design. In production, we typically configure three to five named queues with explicit priorities.

Workers are then assigned to queues with priority ordering. A password reset email will never wait behind a 50,000-row CSV import. The critical queue drains first. Always.

Job Dispatching Patterns

Laravel provides several dispatch mechanisms, each suited to different situations. The choice affects execution timing, failure isolation, and how jobs relate to each other.

Synchronous dispatch. dispatch_sync() runs the job inline, in the current process. Use this for testing and for local development. Running jobs synchronously in feature tests catches integration bugs that mocking hides: database state, event dispatch ordering, and exception handling. The speed trade-off is real, but the coverage gain is worth it for critical job classes. Never use synchronous dispatch in production for slow work.

Standard async dispatch. The job serialises onto the configured queue and returns immediately. A worker picks it up when capacity is available. This is the default and the right choice for most jobs.

Delayed dispatch. The job sits on the queue but is not available for processing until the delay expires. Useful for scheduled follow-ups, rate-limited API calls, and retry windows. Be aware that delayed jobs on Redis use sorted sets, which consume memory proportional to the number of delayed jobs.

Job chaining

Each job in a chain runs only if the previous one succeeded. If the first step fails, the validation and notification jobs never execute. This is the correct pattern for multi-step workflows where later steps depend on earlier results, applying a form of command-query separation where the dispatch side (commands) is decoupled from the processing side. Compare this with workflow engines, which handle more complex branching and conditional logic.

Job batching

Batching runs jobs concurrently with aggregate callbacks. Use it for parallelisable work: processing rows in a CSV, generating thumbnails for uploaded images, or sending notifications to a list of recipients. The batch tracks progress, so you can show the user a percentage complete via a real-time dashboard.

Job middleware

Middleware wraps job execution with cross-cutting concerns. The two we use most frequently:

Designing Idempotent Jobs

A job is idempotent if running it twice with the same input produces the same result as running it once. This property is not optional. It is a requirement for any job that might be retried. Queue systems provide at-least-once delivery, not exactly-once semantics, so every job must be safe to re-execute.

Retries happen constantly in production. A worker crashes mid-job. Redis fails over. A deployment restarts all workers. The queue system re-dispatches the job because job acknowledgement never reached the broker. If that job already wrote half its data to the database, the retry writes it again. Without idempotency, you get duplicate records, double-charged customers, or emails sent twice.

Idempotency is not a library you install. It is a design discipline applied to every job class. We review job idempotency during code review the same way we review database migrations: as infrastructure that must be correct. The deduplication logs and completion records that idempotent jobs produce also feed into audit trails, providing a verifiable history of what work was performed, when, and whether it was a first execution or a safe retry.

Failed Job Handling and Dead Letter Queues

Jobs fail. Connections drop. External APIs return 500s. A CSV contains a row with a malformed date that crashes the parser. The question is not whether jobs will fail but what happens when they do.

Retry strategies

Laravel's default retry behaviour is configurable per job. You can set the number of attempts and a backoff schedule (for example, 10 seconds, then 60, then 300). The backoff prevents a job from hammering an external service that is already struggling.

For transient failures (network timeouts, rate limits), retries with backoff usually resolve the issue. For permanent failures (invalid data, missing dependencies), retries waste resources. The job needs to distinguish between the two.

Dead letter queues

When a job exhausts all retries, Laravel moves it to the failed_jobs table. This is Laravel's built-in failed job store, distinct from a true dead letter queue (such as an SQS DLQ), though it serves a similar purpose. These jobs need attention. They represent work the system could not complete. The dead letter queue pattern, borrowed from enterprise integration patterns, treats failed jobs as messages that require human or automated intervention.

In our systems, we implement three layers of handling.

The worst failure mode is silent failure. A job fails, nobody notices, and the customer never receives their invoice. Dead letter queue monitoring prevents this.

Monitoring, Horizon, and Operational Visibility

A queue without monitoring is a queue where problems go undetected until a customer reports them.

Laravel Horizon

Laravel Horizon provides a dashboard and configuration layer for Redis-based queues. It shows real-time metrics for every queue and worker: jobs per minute, job runtime with percentile breakdowns, failed jobs with full exception traces, wait time before a worker picks up a job, and worker status across all processes.

Horizon also manages worker processes through its supervisor configuration, automatically scaling workers up or down based on queue depth. This is more reliable than manually managing queue:work processes with Supervisor.

What to monitor and alert on

Beyond Horizon's dashboard, we set up alerts for specific conditions that require human intervention. These thresholds integrate with the broader security and operations monitoring stack, where queue metrics sit alongside uptime checks, slow query alerts, and application performance data.

Process management

Queue workers are long-running PHP processes. They do not restart between jobs. This makes them susceptible to memory leaks, stale database connections, and accumulated state. In production, we configure workers to restart regularly: process up to 1,000 jobs or run for one hour, whichever comes first, then exit cleanly. Supervisor or systemd restarts the worker immediately. This limits the impact of memory leaks and ensures workers pick up code changes after deployments.

Production Failure Modes

Tutorials show you how to dispatch a job and process it. They do not show you what happens when things go wrong at scale. These are the failure modes that surface repeatedly in production Laravel applications.

The timeout triad

Three timeout values interact in Laravel queue processing, and misconfiguring any one of them creates failures that are difficult to diagnose.

The critical rule: worker --timeout must exceed job $timeout. If they are equal (or the worker timeout is lower), the worker process dies before the job's exception handler can run. No failed() method fires. No cleanup happens. The job simply vanishes from the worker's perspective and is eventually retried by the queue, potentially causing duplicate execution.

The second critical rule: retry_after must exceed maximum job execution time. If a job takes 90 seconds but retry_after is set to 60, the queue makes the job available again while the first worker is still processing it. Two workers now process the same job concurrently. Without idempotency, this causes data corruption.

Webhook-triggered job patterns

External systems (Stripe, Xero, CRM platforms) send webhooks to your application. Each webhook should dispatch a job rather than processing inline. This ensures the webhook endpoint returns a 200 quickly and isolates the processing from the HTTP request.

The challenge with webhook jobs is idempotency. Stripe sends the same webhook event multiple times as a reliability measure. Your job must handle receiving the same event three times without creating three payment records. Store the webhook event ID and check it before processing, following the same idempotency patterns described above. This pattern connects to our broader approach to API integrations, where incoming data from external systems flows through queued jobs with validation, deduplication, and error handling at each stage.

Symptom-to-cause diagnostic reference

When something goes wrong with queue processing, the symptom rarely points directly at the cause. This reference maps common production symptoms to their root causes and fixes.

Deployment Safety and Queue Worker Restarts

Deploying new code to an application with active queue workers introduces a failure mode that tutorials never cover: payload compatibility between code versions.

When a job is dispatched, Laravel serialises the job class and its properties. When a worker picks the job up, it deserialises that payload and calls handle(). If the code changed between dispatch and processing (because a deployment happened in between), the deserialisation can fail. Renamed classes throw ClassNotFoundException. Changed constructor signatures cause property mismatches. Removed or renamed properties produce silent null values that cascade into application errors.

The risk window is small but real: any job sitting on the queue at the moment of deployment was serialised by the old code and will be processed by the new code.

For applications where job loss is unacceptable (payment processing, order fulfilment), we version job classes explicitly. The old class stays in the codebase until all pending jobs have processed, and new dispatches use the updated class. This adds complexity but eliminates the deployment risk window entirely.

Backpressure and Queue Depth Management

Most queue documentation covers dispatching and processing but ignores what happens when producers outpace consumers. Queue depth grows. Memory consumption rises. Eventually the system degrades, and the degradation pattern depends on your driver.

With Redis, delayed jobs use sorted sets. Each delayed job consumes memory proportional to its serialised payload. A burst of 100,000 delayed jobs with large payloads can push Redis memory usage past its configured limit, triggering eviction policies that silently drop queued jobs. With the database driver, a growing jobs table increases query time for both dispatch and pickup operations, and the table's indexes bloat. With SQS, the default in-flight message limit (120,000 messages per standard queue) acts as a constraint, though this quota can be increased via AWS support.

Backpressure strategies prevent these failure modes.

When Background Jobs Change How a Business Operates

The technical patterns above are infrastructure. The business impact of well-designed background jobs is what makes them worth the engineering investment.

• ✓
  Reports that generate themselves Users click "Generate" and receive an email with the finished report. No waiting, no timeouts, no 504 errors.
• ✓
  Data imports with progress tracking Batch jobs with progress bars replace cron jobs that blocked other scheduled tasks and left operations guessing.
• ✓
  Payment webhooks that never go missing Dedicated queues with SQS durability, dead letter monitoring, and automatic retry. Revenue stops leaking through infrastructure gaps.
• ✓
  Responsive applications under load Heavy work happens in the background. Users never wait for email servers or report generation. Pages respond instantly.

These are patterns we have implemented across order management systems, financial operations platforms, and service delivery tools, deployed via Laravel Forge or Laravel Vapor depending on the infrastructure requirements. Background job processing connects to workflow engines for complex multi-step processes, to real-time dashboards for operational visibility, and to infrastructure decisions about how workers are deployed and scaled.