Event-Driven Architecture at Exchanges: Kafka, Flink and the Data Pipeline of the Trading World

One message, millions of times a second

In 2023, CME Group reported that it processes an average of 23 million messages a day — and on peak days (such as the start of the month) that can reach 120 million messages a day. Every one of those messages — a buy order, a sell order, a cancel, an amend — has to be validated, entered into the order book and, if there is a match, turned into a trade with a confirmation sent back. All of it within 50-100 microseconds.

This is not a problem you can solve with a standard REST API and a relational database. Modern exchanges are built on event-driven architectures in which every action is an event, and every event flows through a pipeline of processors working in real time.

A real case: LSEG (London Stock Exchange Group)

In 2022, LSEG completed the migration of its trading system to an event-driven system built on Aeron and Real Logic. The result: latency fell from 200 microseconds to 30 microseconds, throughput rose fivefold and total operating cost dropped by 40%. All that despite a large initial investment ($150 million); the company expects the investment to pay for itself within 3 years.

Key point: LSEG did not replace the existing system that had been in use since the 1990s. The process took 5 years — but the results are dramatic.

The communication protocols: FIX, FAST and SBE

FIX (Financial Information eXchange) is the oldest protocol still in wide use. It is text-based (tag=value), human-readable and supported by almost every exchange in the world. The problem? Decoding a FIX message means parsing text, and that is slow.

FAST: binary compression

The FAST protocol (FIX Adapted for Streaming) was designed to solve FIX's performance problem. It uses binary encoding, delta encoding of repeating fields and a fixed message structure. The result: messages that are 60-80% smaller and decoding that is 5-10 times faster.

SBE: speed at the expense of flexibility

SBE (Simple Binary Encoding) is the protocol CME chose for iLink 3.0. It defines a fixed binary structure — no parsing, no dynamic memory allocation, no garbage collection. An SBE message is simply a struct in memory. The advantage: deterministic latency at the nanosecond level. The downside: any change to the protocol requires recompilation.

Apache Kafka in trading infrastructure

Kafka is not an ordinary message queue. It is a distributed log that supports replay, retention and multiple consumers reading at the same pace or at different paces. In trading infrastructure, Kafka acts as the backbone of the data pipeline — but not on the critical path of the matching engine.

The reason: Kafka (even with linger.ms=0) adds 1-5 milliseconds of latency. In a matching engine that has to respond within microseconds, that is too much. So Kafka is used downstream — for market data ingestion, audit trail and real-time analytics.

A typical architecture

At a modern exchange, the data flow looks roughly like this:

1. A trading order comes in through the order gateway (SBE/FIX) straight to the matching engine
2. The matching engine processes it and returns a response — with no Kafka in between
3. In parallel, the trade event is written to a Kafka topic
4. Different consumers read from the topic: the market data disseminator, the risk engine, the audit log, analytics
5. Apache Flink (or another stream processor) performs aggregations and alerting in real time

Apache Flink: real-time stream processing

Apache Flink is a CNCF-graduated stream processing framework that works on an event-at-a-time model (as opposed to the micro-batch model of Spark Streaming). In trading infrastructure, Flink is used for tasks such as:

• Detecting anomalies in real time — for example an order that is unusual in its size or frequency
• Calculating VWAP (Volume Weighted Average Price) over a sliding time window
• Aggregating market data from multiple exchanges
• Monitoring traders' risk exposure in real time

Flink's advantage over alternatives such as Kafka Streams is its ability to manage complex state (keyed state, operator state) with exactly-once semantics. When you are dealing with financial data, “approximately” is not good enough — every event has to be processed exactly once.

DataStream<TradeEvent> trades = env.addSource(new TradeSource());

trades
    .keyBy(TradeEvent::getSymbol)
    .window(TumblingEventTimeWindows.of(Time.seconds(60)))
    .aggregate(new VWAPAggregator())
    .addSink(new MarketDataSink());

Flink with SQL

Flink also has the Table API and Flink SQL — a SQL interface that lets you write queries over streams just as you would over ordinary tables. It is very useful for traders and risk specialists who are not Java/Scala developers — they can write queries such as:

SELECT symbol, 
       AVG(price * volume) / SUM(volume) as vwap
FROM trades
GROUP BY TUMBLE(ts, INTERVAL '1' MINUTE), symbol

Eurex T7 versus CME iLink 3.0

The industry's two leading platforms represent different approaches to the same problem.

Eurex T7

The T7 platform from Eurex (part of Deutsche Börse) is built on Java and C++. It supports a range of protocols — FIX, FAST and proprietary binary. The approach: flexibility above all. T7 lets participants choose the protocol that suits them, and provides a rich API for risk management.

T7's matching engine runs at 3-5 microseconds of latency (p99). That is fast, but not the fastest in the industry. Choosing Java allows rapid development and easy maintenance, but it adds garbage collection overhead — which T7 handles with aggressive tuning and allocation-free code paths.

CME iLink 3.0

CME went the other way: raw speed at the expense of flexibility. iLink 3.0 uses SBE only, with FPGA acceleration for the matching engine. The latency: under 1 microsecond in some cases.

The trade-off: any change to the protocol requires a firmware update on the FPGA. That means CME has to plan changes months ahead, while T7 can update Java code and deploy within days. For high-frequency traders, the latency is worth it. For someone who trades once a day, it makes no difference.

Food for thought: the lowest latency in the industry belongs to the exchanges that offer colocation — they let HFT firms place their servers physically next to the matching engine. In that setup, network latency is under 1 microsecond and the bottleneck becomes the processing time of the FPGA itself.

FPGA in matching: why hardware and not software

An FPGA (Field-Programmable Gate Array) is a chip that can be programmed at the level of digital logic. Unlike a CPU, which runs instructions in sequence, an FPGA performs operations in parallel at the hardware level.

In a matching engine, that means: instead of running a loop of “receive message → decode → look up in the order book → match → send response”, the FPGA executes all of those stages in a parallel pipeline. While one message is being decoded, the previous message is already being matched, and the one before that has already been sent.

The downside: FPGA development requires expertise in Verilog or VHDL, a long cycle time and debugging that is far harder than ordinary software. Not every organisation can afford it.

Aeron: messaging without compromise

Aeron is a messaging library developed by Real Logic (Martin Thompson) specifically for low-latency systems. It uses memory-mapped files, zero-copy and UDP multicast to achieve latency in the hundreds of nanoseconds with a throughput of millions of messages a second.

Aeron is used at several exchanges and HFT firms as a replacement for Kafka on the low-latency side of the pipeline. It does not replace Kafka — it complements it. Aeron handles critical-path communication, and Kafka handles the downstream data pipeline.

Conclusion: how to choose an architecture

There is no single “correct” architecture. The choice depends on latency requirements, budget and team:

• If you are building an HFT system with a nanosecond latency requirement: FPGA + SBE + Aeron
• If you are building a general trading system with microsecond latency: Java/C++ + FIX/FAST + Kafka + Flink
• If you are building an analytics platform: Kafka + Flink + a data lake

The common principle: event-driven, an append-only log, and separation between the critical path and the data pipeline.