Message Queue Development: Redis, RabbitMQ & Kafka

A message queue lets one part of your system hand off work to another without waiting for it to finish. Instead of sending the confirmation email, generating the invoice PDF, resizing product images and notifying the warehouse while the customer stares at a spinner, your application drops a message on a queue and responds immediately; background workers pick the jobs up, retry them if something fails and report when they are done.

At CodeWiz we design, build and operate message queue systems with Redis, RabbitMQ and Apache Kafka – from a simple Laravel or Node.js job queue on one server to replicated Kafka clusters that stream millions of events a day. We choose the broker that fits your workload, write the producers and workers, and make sure messages are not lost, duplicated or silently stuck when servers restart or third-party APIs go down.

This service is for online stores that must never lose an order, SaaS products with heavy background processing, platforms receiving webhooks from payment providers and partners, and data teams that need a reliable stream of events for analytics. If your requests time out under load, cron jobs overlap, or a slow external API takes your whole application with it, a properly designed queue is usually the fix.

Message: Laptop showing source code on a desk next to a stack of books

When Your Application Needs a Message Queue

Not every system needs a broker, but the warning signs are consistent:

Slow requests: users wait for work they do not need to see finished – emails, exports, thumbnails, search indexing.
Traffic spikes: a campaign or a flash sale produces more work than the database or an external API can handle at once. A queue absorbs the spike and lets workers process it at a safe rate.
Unreliable dependencies: payment providers, couriers, ERPs and email APIs fail or rate-limit you. Queued jobs retry with backoff instead of failing the user’s request.
Fan-out: one action (“order paid”) must trigger several independent reactions – invoicing, stock, analytics, CRM.
Scheduled and long-running work: imports, scrapers and report generation that outgrow cron and need visibility, concurrency limits and retries.

Redis Queues for Background Jobs

Redis queues are the fastest way to add reliable background processing to a web application. Redis is often already in your stack for caching or sessions, it handles tens of thousands of operations per second on modest hardware, and mature job libraries sit on top of it in every major language.

Libraries we use

Laravel Queues with Horizon for PHP projects: job classes, retries, rate limiting, batches and a dashboard with throughput and failed jobs.
BullMQ for Node.js: delayed and repeatable jobs, priorities, parent–child flows, concurrency control and global rate limits.
Celery, RQ or Dramatiq for Python services, including AI and scraping pipelines.
Redis Streams with consumer groups when you want log-like semantics – acknowledgements, pending entry lists and replay – without running Kafka.

A typical BullMQ producer and worker, with retries and a deterministic job ID that prevents duplicates:

import { Queue, Worker } from 'bullmq';

const connection = { host: 'redis', port: 6379 };
const emails = new Queue('emails', { connection });

await emails.add('order-confirmation', { orderId: 1042 }, {
  jobId: 'order-confirmation-1042',
  attempts: 5,
  backoff: { type: 'exponential', delay: 2000 },
  removeOnComplete: 1000,
});

new Worker('emails', async (job) => {
  await sendOrderConfirmation(job.data.orderId);
}, { connection, concurrency: 10 });

Making Redis safe for jobs

Redis configured as a cache will happily evict your jobs when memory fills up. For queues we use a dedicated instance or database with maxmemory-policy noeviction, AOF persistence (appendonly yes) so a restart does not wipe pending work, and Redis Sentinel or a managed service for failover. Workers run under Supervisor or as containers with graceful shutdown, so a deployment finishes the current job instead of killing it halfway. The BullMQ documentation is a good reference for the Node.js side.

RabbitMQ Setup and Message Routing

RabbitMQ is a dedicated message broker built for flexible routing and per-message reliability. Producers publish to exchanges, and bindings decide which queues receive each message – so one “order paid” event can reach the invoicing, warehouse and analytics queues without the producer knowing any of them exist.

What a production RabbitMQ setup includes

Exchange design: direct exchanges for commands, topic exchanges with routing keys like order.paid.eu for events, fanout for broadcasts.
Quorum queues (Raft-replicated) on a three-node cluster instead of classic mirrored queues, so a node failure does not lose messages.
Publisher confirms on the producer side and manual acknowledgements on the consumer side – a message is removed only after it has actually been processed.
Prefetch limits (basic.qos) so one consumer does not grab thousands of messages and starve the others.
Dead-letter exchanges for messages that are rejected, expire or exceed their delivery limit, with delayed retry queues built on TTL.
TLS, separate virtual hosts and users per application, and the management plugin plus Prometheus metrics for visibility.

Policies keep this configuration out of application code. For example, every queue starting with orders. gets a dead-letter exchange and a delivery limit:

rabbitmqctl set_policy orders-dlx "^orders\." \
  '{"dead-letter-exchange":"orders.dlx","delivery-limit":5}' \
  --apply-to queues

We run RabbitMQ in Docker or Kubernetes (using the official Cluster Operator), on dedicated VMs, or as a managed service such as Amazon MQ or CloudAMQP when you prefer not to operate it. The official RabbitMQ documentation covers quorum queues and reliability guarantees in depth.

Motion-blurred train passing an illuminated modern station platform at night

Apache Kafka for High-Throughput Messaging

Apache Kafka is not a queue in the classic sense but a distributed, replicated commit log. Messages are appended to partitions and kept for a configured retention period, whether or not anyone has read them. Each consumer group tracks its own offset, so the same stream can feed a billing service, a data warehouse and a search indexer independently – and a new consumer can replay history from the beginning.

How we set Kafka up

KRaft mode without ZooKeeper (the only option since Kafka 4.0), with at least three brokers for production.
Partitioning by key – for example by customer or order ID – so related events stay in order while load spreads across consumers.
Durability settings: replication factor 3, min.insync.replicas=2, and producers with acks=all and idempotence enabled.
Schemas in Avro, Protobuf or JSON Schema with a schema registry, so a producer cannot break every consumer with an unannounced field change.
Retention and compaction per topic: time-based retention for event history, log compaction for “latest state per key” topics.

kafka-topics.sh --bootstrap-server kafka:9092 --create \
  --topic orders --partitions 12 --replication-factor 3 \
  --config min.insync.replicas=2 --config retention.ms=604800000

# producer.properties
acks=all
enable.idempotence=true
compression.type=zstd
linger.ms=10

Kafka is powerful but operationally heavier than Redis or RabbitMQ. We self-host it when control and cost matter, consider Redpanda as a Kafka-compatible alternative with a simpler footprint, or use managed offerings such as Confluent Cloud, Amazon MSK or Aiven. See the Apache Kafka documentation for the full configuration reference.

Event Streaming and Real-Time Data Pipelines

Event streaming goes beyond moving jobs from A to B: data is processed continuously as it arrives. Typical projects we build:

Change data capture (CDC) with Debezium: every insert, update and delete in MySQL or PostgreSQL becomes an event, without touching application code. Ideal for feeding search indexes, caches and analytics from a legacy database.
Stream processing with Kafka Streams, Apache Flink or ksqlDB: rolling aggregates, fraud and anomaly rules, sessionisation and enrichment in real time.
Kafka Connect pipelines into ClickHouse, Elasticsearch, Typesense, S3 or a data warehouse, with no custom glue code to maintain.
Real-time product features: live dashboards, notifications and activity feeds pushed to browsers over WebSockets.

Streams also make systems auditable and recoverable: if a consumer had a bug, you fix it, reset its offset and reprocess the affected events. That same idea – capture everything, make it replayable – is at the core of our approach to zero data loss e-commerce.

Redis vs RabbitMQ vs Kafka: Choosing a Message Broker

Criterion	Redis (BullMQ, Horizon, Streams)	RabbitMQ	Apache Kafka
Model	Job queue or lightweight stream	Broker with exchanges and queues	Partitioned, replicated log
Message after consumption	Removed (Streams can keep it)	Removed after acknowledgement	Kept until retention expires
Routing	Basic, by queue name	Rich: direct, topic, fanout, headers	By topic and partition key
Throughput	High	High	Very high, built for big volumes
Replay	Limited	No (except RabbitMQ Streams)	Yes, from any offset
Operational effort	Low	Medium	High
Best for	Web app background jobs, delays, scheduling	Task distribution, complex routing, RPC	Event streaming, analytics, CDC, event history

Many systems combine them: Redis or RabbitMQ for application jobs, Kafka for the event backbone. Cloud-native alternatives such as Amazon SQS, Google Pub/Sub or NATS JetStream are good choices when you are already committed to that platform or want minimal operations.

Reliability Patterns: No Lost or Duplicated Messages

The broker is only half the story; reliability comes from how producers and consumers use it.

At-least-once delivery plus idempotent consumers: messages may arrive twice, so every handler checks an idempotency key (order ID, event ID) before acting. Exactly-once is mostly a property you build, not one you buy.
Transactional outbox: the database change and the outgoing message are written in one transaction, then published by a relay – no “order saved, event lost” gaps.
Retries with exponential backoff and jitter, capped attempts, and a clear distinction between temporary errors (retry) and permanent ones (fail fast).
Dead-letter queues with alerts and a tool to inspect, fix and replay failed messages instead of losing them.
Ordering where it matters: per-key ordering through partition keys or single-active consumers, not global ordering that kills throughput.
Graceful shutdown of workers on every deployment, and visibility timeouts long enough for the slowest legitimate job.

Cars moving in orderly lanes through a busy city road at night

Monitoring and Scaling Queue Workers

A queue hides problems until it suddenly does not. We monitor the metrics that actually predict trouble: queue depth, age of the oldest message, consumer lag in Kafka, processing time per job type, failure rate and dead-letter queue size. Dashboards come from Laravel Horizon, Bull Board, the RabbitMQ management UI or Kafka UI, with Prometheus exporters and Grafana alerts on top.

Workers scale on those same signals. On Kubernetes, KEDA adds consumers when the backlog grows and removes them when it drains; on simpler setups we size Supervisor or container worker pools per queue, so a flood of low-priority exports never delays password-reset emails.

Common Message Queue Mistakes We Fix

Using a cache-configured Redis for jobs, so eviction or a restart silently deletes pending work.
Auto-acknowledging messages before processing, so a crash loses them.
Infinite retries of a poison message that blocks the queue and floods logs.
Putting huge payloads in messages instead of a reference to the data in storage or the database.
Choosing Kafka for a small job queue – or Redis for a permanent event history.
No alerting on backlog, discovered only when customers complain about missing emails.

How We Work

Discovery: we map the work your system does asynchronously today (or should), its volumes, latency requirements and failure modes.
Design: we choose the broker, define queues, exchanges or topics, message formats, retry and dead-letter policies, and delivery guarantees.
Build: we implement producers, workers and the outbox where needed, with idempotency built in, and provision the broker with infrastructure as code.
Test: load tests with realistic spikes, plus failure tests – killing brokers and workers mid-flight to prove that nothing is lost.
Launch: gradual rollout with dashboards and alerts for depth, lag and failures in place from day one.
Support: runbooks for replaying dead letters and scaling workers, plus optional ongoing monitoring and upgrades.

Why CodeWiz

Full-stack ownership: we write the Laravel, Node.js and Python code that produces and consumes the messages, not just the broker configuration.
The right tool, not the trendy one: tailored solutions – Redis when it is enough, Kafka when you genuinely need a log.
Open-source first: Redis, RabbitMQ, Kafka, Debezium and Grafana run anywhere, with no licence fees or lock-in.
Fair pricing: premium development without premium prices, from a team in Banja Luka working with clients locally and internationally.
Experience with real workloads: payments, e-commerce, large-scale data scraping and automation all depend on background processing we build every day.

Frequently Asked Questions

Should I use Redis, RabbitMQ or Kafka?

For background jobs in a web application, Redis-based queues are usually the simplest and fastest choice. RabbitMQ fits when you need flexible routing and strong per-message guarantees across services. Kafka is the right tool for high-volume event streams, replay and analytics. We recommend based on your volumes and team, not on fashion.

Can messages get lost?

With correct configuration – persistence, replication, publisher confirms, manual acknowledgements and an outbox on the producer side – lost messages become extremely unlikely. Most losses we see come from misconfiguration, such as auto-acknowledgement or a Redis instance that evicts keys.

How much does a message queue implementation cost?

Adding a Redis job queue to an existing Laravel or Node.js app is a small project. A replicated RabbitMQ or Kafka cluster with CDC pipelines and monitoring is considerably larger. We estimate after a short review of your workloads.

Do I need Kafka for event-driven architecture?

No. Many event-driven systems run perfectly well on RabbitMQ, Redis Streams, NATS or a cloud service. Kafka pays off when you need very high throughput, long retention or replay for many independent consumers.

Can you add queues to an existing application?

Yes, and it is one of our most common projects. We usually start by moving the slowest or most failure-prone tasks – emails, webhooks, exports, payment provider calls such as those in our payment gateway integrations – into jobs, then expand step by step.

Ready to Make Background Processing Reliable?

Tell us what your application does in the background today and where it breaks, and we will design a queue setup that fits your load and budget. Contact us to discuss your project, or see the full scope of our message queue services.

Message Queue Development: Redis, RabbitMQ & Kafka

Contents