If you've ever wondered how trading systems receive real-time market data—price updates, trades, order book changes—at speeds measured in microseconds, this post is for you. We're open-sourcing a CME MDP3.0 market data feed handler written in C++ that achieves sub-microsecond latency from packet receipt to application callback.
This handler is certified on CME AutoCert for futures, and with minor modifications works for spreads, options, BTEC, and EBS products.
Why build your own feed handler?
GitHub Repository: CME-Market-Data-Handler
Market data is the continuous stream of information flowing from exchanges about trading activity:
For a futures contract like ES (S&P 500 E-mini), the exchange might send thousands of updates per second during active trading.
In electronic trading, receiving market data faster than competitors provides an edge:
The difference between 1 microsecond and 100 microseconds can determine profitability.
CME Group is the world's largest derivatives exchange, operating CME, CBOT, NYMEX, and COMEX. They trade:
MDP3.0 (Market Data Platform 3.0) is CME's binary market data protocol. Key characteristics:
Unlike text-based protocols (FIX, JSON), MDP3.0 uses Simple Binary Encoding (SBE)—a binary format designed for ultra-low-latency messaging. Binary data requires no parsing; you read values directly from memory.
Market data is distributed via UDP multicast—a one-to-many network protocol. The exchange sends each packet once, and all subscribers receive it simultaneously. CME provides redundant A and B channels for reliability.
MDP3.0 defines dozens of message types:
Every packet has a sequence number. Gaps indicate lost packets requiring recovery.
SBE is the FIX protocol's binary encoding standard, designed for financial messaging where nanoseconds matter.
Key properties:
Compare to JSON parsing, which requires:
SBE skips all of this—you just read bytes from known offsets.
CME publishes XML schema files defining all message types. The Real Logic SBE tool generates C++ headers from these schemas.
The process:
java -jar sbe-all.jar schema.xmlmktdata/ directoryEach message type becomes a C++ class:
// From mktdata/MDIncrementalRefreshBook46.h (auto-generated)
class MDIncrementalRefreshBook46 {
static const std::uint16_t SBE_TEMPLATE_ID = 46; // Message type identifier
static const std::uint16_t SBE_SCHEMA_VERSION = 9;
// Zero-copy: wraps existing buffer, no allocation
MDIncrementalRefreshBook46 &wrapForDecode(
char *buffer,
const std::uint64_t offset,
const std::uint64_t actingBlockLength,
const std::uint64_t actingVersion,
const std::uint64_t bufferLength);
// Direct memory access - no parsing overhead
std::uint64_t transactTime() const {
return *((std::uint64_t *)(m_buffer + m_offset + 0));
}
// Repeating groups for order book entries
class NoMDEntries {
bool hasNext() const;
NoMDEntries &next();
int64_t mDEntryPx(); // Price
int32_t mDEntrySize(); // Size
// ... more fields
};
};The mktdata/ directory contains 59 generated files:
MDIncrementalRefreshBook46.h, MDIncrementalRefreshTradeSummary48.h, etc.Decimal9.h, PRICE9.h, AggressorSide.hThe handler has a clean, layered architecture:
┌─────────────────────────────────────────────────────────────────┐
│ CME Multicast │
│ (Channels A & B, UDP) │
└──────────────────────────┬──────────────────────────────────────┘
│
┌────────────▼────────────┐
│ MessageProcessor │ ← Receives packets, tracks sequences
│ (message thread) │
└────────────┬────────────┘
│
│ Gap detected?
│ │
┌────────────▼──────▼─────┐
│ RecoveryProcessor │ ← Fetches snapshots on gaps
│ (recovery thread) │
└────────────┬────────────┘
│
┌────────────▼────────────┐
│ DataDecoder │ ← Parses SBE messages
└────────────┬────────────┘
│
┌────────────▼────────────┐
│ CallBackIF │ ← YOUR CODE HERE
│ (application logic) │
└─────────────────────────┘Threading model:
Handles the live data feed:
// Pseudocode for the main loop
void MessageProcessor::operator()() {
while (running) {
// Non-blocking read from channels A and B
read_sockets(); // Uses MSG_DONTWAIT
// Process messages in sequence order
processq(current_time);
}
}Key responsibilities:
MSG_DONTWAIT for low latencyTransforms SBE binary data into application callbacks:
void DataDecoder::mbo_data(const char* buffer, size_t len, uint64_t recv_time) {
// Read packet header
auto MsgSeqNum = *(uint32_t*)(buffer);
auto SendingTime = *(uint64_t*)(buffer + 4);
// Read message header
auto blockLength = *(uint16_t*)(buffer + 12);
auto templateId = *(uint16_t*)(buffer + 14);
// Dispatch based on template ID
switch (templateId) {
case 46: // MDIncrementalRefreshBook (MBP)
decode_book_46(buffer, recv_time, MsgSeqNum, SendingTime);
break;
case 47: // MDIncrementalRefreshOrderBook (MBO)
decode_book_47(buffer, recv_time, MsgSeqNum, SendingTime);
break;
case 48: // MDIncrementalRefreshTradeSummary
decode_trade_48(buffer, recv_time, MsgSeqNum, SendingTime);
break;
// ... other message types
}
}| Supported message types: | Template ID | Message Type | Description |
|---|---|---|---|
| 46 | MDIncrementalRefreshBook | Level-based book updates (MBP) | |
| 47 | MDIncrementalRefreshOrderBook | Order-based updates (MBO) | |
| 48 | MDIncrementalRefreshTradeSummary | Trade executions | |
| 51 | MDIncrementalRefreshSessionStatistics | Session stats | |
| 53 | SnapshotFullRefreshOrderBook | Full book snapshot (recovery) | |
| 54 | MDInstrumentDefinitionFuture | Contract definitions |
Handles gap recovery—rebuilding order book state after packet loss:
void RecoveryProcessor::operator()() {
while (running) {
// Wait for gap notification from MessageProcessor
{
std::unique_lock<std::mutex> lock(mutex_);
cv_.wait(lock, [this] { return gap_detected_ || !running; });
}
if (gap_detected_) {
// Subscribe to snapshot feed
// Receive full order book state
// Signal completion to MessageProcessor
perform_recovery();
}
}
}Recovery types:
The interface you implement to receive market data:
struct CallBackIF {
// Market-By-Price book update
virtual void MDIncrementalRefreshBook(
uint64_t recv_time, // When we received the packet
uint32_t msgSeqNum, // Sequence number
uint64_t transactTime, // Exchange timestamp
uint64_t sendingTime, // CME sending time
uint32_t securityID, // Instrument identifier
int64_t px_mantissa, // Price (mantissa)
int8_t px_exponent, // Price (exponent)
char side, // '0'=Buy, '1'=Sell
int32_t sz, // Size at this level
int32_t numorders, // Number of orders
uint8_t pxlevel, // Price level (1=best, 2=second, ...)
bool endOfEvent, // Last update in this event
bool recovery // True if from recovery feed
) noexcept = 0;
// Called when a gap is detected - clear your books!
virtual void Clear() noexcept = 0;
// Helper: convert CME price to double
double to_price(int64_t mantissa, int8_t exponent) {
return double(mantissa) * std::pow(10., double(exponent));
}
};Shows aggregated order book levels:
Simpler to process, smaller data volume. Use for:
Shows individual orders with IDs and priorities:
More complex, higher data volume. Use for:
The handler supports both—implement the appropriate callback overload.
Networks lose packets. When this happens:
Detection: MessageProcessor notices sequence number jump
Expected: 1001, Received: 1005 → Gap of 3 packets!Notification: Clear() callback tells your application to invalidate order books
Recovery: RecoveryProcessor subscribes to snapshot feed, receives full order book state
Reconciliation: Apply snapshot, resume incremental updates
Timeline:
─────────────────────────────────────────────────────────────────►
Incremental GAP Recovery Incremental
[1000][1001] ??? [Snapshot] [1010][1011][1012]
↓
Clear() called
Books invalidated
Wait for recoveryKey insight: During recovery, updates have recovery=true flag. Your application must handle the transition from recovery to live data correctly.
#include "CallBackIF.hpp"
#include <iostream>
#include <map>
struct MyOrderBook {
std::map<double, int32_t> bids; // price → size
std::map<double, int32_t> asks;
};
class MyHandler : public m2tech::mdp3::CallBackIF {
std::map<uint32_t, MyOrderBook> books_; // securityID → book
public:
void MDIncrementalRefreshBook(
uint64_t recv_time,
uint32_t msgSeqNum,
uint64_t transactTime,
uint64_t sendingTime,
uint32_t securityID,
int64_t px_mantissa,
int8_t px_exponent,
char side,
int32_t sz,
int32_t numorders,
uint8_t pxlevel,
bool endOfEvent,
bool recovery
) noexcept override {
double price = to_price(px_mantissa, px_exponent);
auto& book = books_[securityID];
if (side == '0') { // Buy
if (sz > 0) book.bids[price] = sz;
else book.bids.erase(price);
} else { // Sell
if (sz > 0) book.asks[price] = sz;
else book.asks.erase(price);
}
if (endOfEvent) {
// Full update received, book is consistent
// ... process the updated book
}
}
void Clear() noexcept override {
std::cout << "Gap detected! Clearing all books.\n";
books_.clear();
}
// ... implement other callbacks
};#include "MessageProcessor.hpp"
#include "RecoveryProcessor.hpp"
#include <thread>
int main() {
MyHandler callback;
// Configure multicast channels (from CME documentation)
MessageProcessor msg_proc(
callback,
"224.0.28.1", 14310, // Channel A
"224.0.28.2", 14310, // Channel B
"eth0" // Network interface
);
RecoveryProcessor rec_proc(
msg_proc,
callback,
"224.0.28.100", 14320, // Instrument Recovery
"224.0.28.101", 14321 // Data Recovery
);
// Start processing threads
std::thread msg_thread(std::ref(msg_proc));
std::thread rec_thread(std::ref(rec_proc));
// Run until shutdown
msg_thread.join();
rec_thread.join();
return 0;
}For offline analysis or testing, compile with PCAP support:
g++ -DUSE_PCAP=true main.cpp -O2 -pthread -lpcap -o mdp3_pcap
./mdp3_pcap captured_data.pcapThis bypasses live multicast and reads from a packet capture file.
Latency: < 1 microsecond from packet receipt to callback invocation
How is this achieved?
MSG_DONTWAIT flag prevents socket blockingWhat's NOT optimized (deliberately):
For most use cases, these trade-offs are acceptable. The handler is production-quality while remaining maintainable.
cd mdp3handler
g++ -std=c++17 -O2 main.cpp -pthread -o mdp3handler
# With PCAP support
g++ -std=c++17 -O2 -DUSE_PCAP=true main.cpp -pthread -lpcap -o mdp3handlerThis handler has passed CME's AutoCert certification for futures. AutoCert validates that your handler correctly:
This open-source CME MDP3.0 handler provides:
Whether you're building a trading system, analyzing market microstructure, or learning about exchange protocols, this handler provides a solid foundation.
GitHub Repository: CME-Market-Data-Handler
Open source under the MIT License.
Copyright 2022 Vincent Maciejewski, Quant Enterprises & M2 Tech