Kafka Library

Pure-Haskell Language Package

This is a pure-Haskell language package for producing to and consuming from Kafka clusters. The wire protocol format is derived directly and automatically from the Kafka protocol specification. It has structured logging and explicit error handling (no surprises). It is exposed to the user via a constant-memory streaming API (similar to Kafka Streams) in Haskell, which also inherently mitigates backpressure. Supports concurrent streaming to different partitions via normal Haskell green threads out of the box. It is well tested at the protocol format level up to the protocol level with its test suite making use of ephemeral Zookeeper clusters.

Benefits

Pure-Haskell: make use of existing powerful concurrency in Haskell, avoid the pitfalls of talking to C library, support the latest and greatest protocol by being more nimble, avoid common exploits and segmentation faults seen in C libraries.
Streaming API: connect pieces of your project together via streams, anywhere from HTTP to Websocket to Kafka to the database; you can pipe these together in a uniform way. Allows "fire and forget" style logging/metrics to Kafka.
Efficient Kafka streaming is an art (latency vs throughput) and you want a language that is high-level like Haskell to work in that context. You want something that helps you manage backpressure easily (Conduit, upon which our Kafka library is based).
Structured logs, not strings. You may want to perform logic (rebalancing or scaling up/down) based on conditions that happen when talking to the cluster, our logging system allows to easily implement "hooks" to pick up on those events.
Errors are explicit: things will never go wrong silently; you have to handle error conditions, meaning you have greater confidence that your Kafka stream will not go down mysteriously.