Technology
Make Kafka predictable under load and during change
Kafka tolerates shortcuts for a long time, then fails at a peak, in a rebalance or during an upgrade. We review partitions, keys, clients, schemas and the cluster itself, and fix them in order of risk.
- Duration
- 4 to 12 weeks
- Model
- Review, fix and hand over
- Output
- Healthy cluster, schema rules and runbooks
Signs you need this
Each points to a choice made early and never revisited.
- Consumer lag climbs at every peak, and adding consumers stops helping once there is one per partition.
- Consumer groups rebalance several times an hour, and processing pauses each time.
- One partition is far busier than the rest because a few keys carry most of the traffic.
- A renamed field in one producer broke consumers downstream, because schema compatibility is not checked.
- Duplicates appear after deployments and rebalances, and nobody is sure what exactly-once covers in your setup.
- Partition counts were fixed on day one, and raising them now would change which partition each key lands on.
- The cluster still runs on ZooKeeper, several versions behind, and nobody wants to own the upgrade.
Why the obvious fixes backfire
Kafka is easy to start with. Default settings, a partition count chosen on day one and keys picked by whoever wrote the first producer carry a system a long way. The problems appear later: at peak load, during a deployment, or when a second team starts reading the same topics.
The obvious fixes often make things worse. Adding partitions changes which partition a key maps to, so ordering breaks for keys already in flight. Adding consumers to a group does nothing beyond one per partition. Transactions give exactly-once processing inside Kafka, but a consumer that writes to a database can still apply the same message twice. A managed service removes broker operations, not design decisions.
This service is about Kafka itself: the cluster, clients, schemas and tooling. Deciding which flows become events, and how services stay consistent, is Event-Driven Architecture; teams often do that first and then use this service to make the platform carry it.
What is included, and what is not
Included
- Topic, partition, key, retention and compaction design
- Producer settings: idempotence, acknowledgements, batching and retries
- Consumer group behaviour, rebalancing and lag alerting
- Schema registry and compatibility rules per topic
- Transactional outbox and change data capture, for example with Debezium on Kafka Connect
- Cluster health, security and upgrades, including the move from ZooKeeper to KRaft
- Managed or self-hosted: Amazon MSK, Confluent Cloud, the Azure Event Hubs Kafka endpoint or your own brokers
Not included
- Deciding which business flows become events
- Building stream processing and analytics applications
- Replacing a working RabbitMQ setup without a measured reason
- Running the cluster day to day after handover
How the work runs
- Weeks 1 to 2Map the platform. With read-only access to cluster metrics, topic configuration, client settings and schemas, we map each topic to its producers, consumers, keys and owner. Lag, rebalances and partition skew are measured across real peaks.
- Week 3Rank the risks. Findings are ordered by what fails first: settings that can lose data, such as
acks=1or a lowmin.insync.replicas, broken ordering assumptions, missing schema checks and capacity limits. - Week 4 onwardFix in reversible steps. Client, topic, schema and connector changes go through your pipeline one at a time, with lag and error rates compared before and after. Write access granted for a stage is removed when the stage ends.
- ThroughoutRehearse failure. Broker loss, a consumer crash, a poison message and a replay are tested with your team, so the runbooks describe what actually happens.
- Final weekHand over. Everything is walked through with the engineers who will own it.
What you get
All outputs belong to you, including configuration and the source files of every diagram.
When this is not the right service
- Systems with modest volumes that RabbitMQ or a managed queue already handles well. Kafka adds operational weight that has to pay for itself.
- Teams whose real problem is consistency between services rather than the broker. That starts with Distributed Systems Design.
- Organisations looking for someone to operate Kafka around the clock. We fix, document and hand over.
Frequently asked questions
Should we move from RabbitMQ to Kafka?
Only if you need what Kafka does differently: replayable event history, many independent consumers of one stream, or throughput your current broker cannot sustain. If RabbitMQ is doing its job, a move adds cost and new failure modes. We measure first and say so if the answer is no.
Managed Kafka or our own cluster?
Amazon MSK, Confluent Cloud and the Kafka endpoint of Azure Event Hubs remove most broker operations, but they differ in supported features, versions, limits and how costs grow with traffic. Self-hosting, for example on Kubernetes with Strimzi, gives full control and full responsibility. We compare them against your workload and team.
Can you promise exactly-once delivery?
Not end to end. Idempotent producers and transactions give exactly-once processing from topic to topic inside Kafka. Once a consumer writes to a database or calls an API, duplicates must be handled there, with idempotent writes or deduplication keys. We make those boundaries explicit.
Our cluster still runs on ZooKeeper. Is that a problem?
Yes. Kafka 4.0 removed ZooKeeper support, so a ZooKeeper-based cluster must move to KRaft on a 3.x release before upgrading to 4.x. We plan and rehearse that migration on a copy, then run it with a way back until the final step.
What happens after the engagement?
Your team owns the configuration, dashboards and runbooks. Many teams continue alone; others keep an embedded architect 2 to 3 days a week while they move more flows onto the platform.
Start with a short technical conversation
Thirty minutes. You describe your Kafka setup and where it hurts, and we tell you where we would start.