Ways of communications over electronic trading platforms

Electronic trading platforms facilitate the trading of financial securities and commodities. The communication methods on these platforms can vary, but generally, they include:

  1. Electronic Messaging: Traders and brokers communicate via electronic messages using protocols like FIX (Financial Information eXchange). This allows them to exchange information about trades, quotations, and market data.

  2. APIs (Application Programming Interfaces): APIs allow different software applications to communicate with the trading platform, enabling automated trading strategies, order submission, and access to real-time market data.

  3. Direct Market Access (DMA): Professional traders use DMA to place buy and sell orders directly into the order book of an exchange, which requires minimal communication.

  4. Market Data Feeds: Real-time data feeds provide traders with the latest market information, which is crucial for making informed trading decisions.

  5. Trade Confirmation Notifications: After a trade is executed, the platform sends out confirmations via email, SMS, or within the platform itself to inform the involved parties.

  6. Email and Chat Services: Some platforms offer integrated email and live chat options for traders to communicate with each other or with support services.

Each of these methods helps facilitate efficient and timely trading, which is critical in the fast-paced environment of electronic trading platforms.

Market Data Feeds and Binary Protocols

Market Data Feeds are critical infrastructure components for financial markets. They provide real-time and historical data regarding securities transactions and price quotations across various trading venues. This information is indispensable for market participants who rely on timely and accurate data for decision-making processes.

Binary Financial Protocols are the mechanisms through which these data feeds are transmitted. They utilize binary encoding to ensure the efficient and rapid delivery of market data. The use of binary protocols minimizes latency because they require less processing power to encode and decode data, enabling quicker dissemination and processing of information which is a crucial advantage in trading environments where milliseconds can equate to significant financial outcomes.

Types of Protocols

There are different Binary Financial Protocols used over electronic trading platforms to transport business level messages. These protocols can be think of as 2 unique layers (Binary Financial Protocols Overview):

  • low session level network transport-level protocols that manage message sequencing, retransmission, and delivery guarantees.
  • application level business level messages that are delivered using the lower level transport protocol

Different types of application level protocols:

The Financial Information eXchange (FIX) protocol is a widely adopted messaging standard for communicating securities transactions over networks. However, for the high-speed transmission of market data, many exchanges also use custom binary encoded protocols that are optimized for low latency and high throughput.

Different types of network level protocols:

Transmission Control Protocol (TCP) is reliable but relatively slow due to its error checking and correction mechanisms. On the other hand, multicast User Datagram Protocol (UDP) is used for sending the same message to multiple recipients simultaneously, which is ideal for market data dissemination. Playback services are available for simulating market data feeds for testing purposes.

Some protocols define both session level and application level (e.g. FIX) and others define application level with transport session level options (e.g. ITCH with SoupBinTCP, Compressed or MoldUDP64).

List of some major US electronic exchanges with the protocols they use

ExchangeProtocol(s) Used
NasdaqFIX, ITCH, OUCH
NYSEFIX, Pillar
CboeFIX, PITCH, BOE
CTACTS Pillar, CQS Pillar
UTPUQDF, UTDF

For the same binary data protocol, different exchanges use different network protocols to transmit the data. For example, Nasdaq offers the TotalView ITCH data feed in three protocol options:

  • SoupBinTCP: A TCP-based protocol designed for direct feed applications with recovery capabilities.
  • Compressed via SoupBinTCP: Compresses ITCH messages to reduce bandwidth usage.
  • MoldUDP64: A UDP-based protocol designed for low-latency applications.

Advantages of using Multicast UDP Feeds

  • Latency: Multicast UDP allows for the simultaneous delivery of messages to multiple recipients, reducing the time to distribute the same message to all interested parties.
  • Bandwidth: It optimizes network bandwidth consumption as data is sent out only once for all subscribers, unlike TCP which requires a separate stream for each connection.

Design of a Multicast UDP Protocol in General

The design of Binary Multicast UDP protocols in financial markets, such as NASDAQ’s ITCH, NYSE’s Pillar, and CBOE’s PITCH, are designed to provide high-throughput and low-latency market data dissemination. Although each protocol has its specific characteristics, tailored to the needs of the exchange and its participants, they share common design principles that ensure reliability, speed, and efficiency in electronic trading environments. Here are the general design aspects, along with the commonalities and differences:

General Design of Binary Multicast UDP Protocols

  1. Efficient Data Representation: Binary formats are used for their compactness and efficiency, allowing for quicker transmission and less bandwidth usage compared to text-based formats.

  2. Multicast Transmission: To manage bandwidth more efficiently and to allow clients to subscribe to only the relevant subset of market data, symbols are often split across several multicast channels.

  3. Non-Guaranteed Delivery: UDP does not guarantee delivery, so the protocol must handle the possibility of lost or dropped packets.

  4. Packet Sequence Control: Each network packet includes a sequence number to allow recipients to detect missed messages and request retransmissions if necessary.

  5. Session Management: The protocol includes session control messages for login, heartbeats, and logout to manage the state of the connection.

  6. Message Framing: Messages have a clear start and end, often with a length field to facilitate parsing.

  7. Error Checking: Checksums or other error detection codes are included to ensure data integrity.

Common Points

  • Headers: They typically have headers indicating message length, type, and sequence number.
  • Market Data Types: They broadcast similar types of data, like trade summaries, price level updates, and order book changes.
  • Throttling and Congestion Management: Mechanisms are in place to manage data flow and prevent congestion, which is critical in UDP since there is no flow control like in TCP.
  • Fault Tolerance: They provide ways to recover from message loss, such as gap request mechanisms or separate recovery channels.
  • Redundancy: Duplicated data streams and dual sites provide redundancy.

Differences

  • Message Types and Structures: While all protocols serve to disseminate market data, the specific message types and their structures can vary, reflecting the different trading products and market models.
  • Market Data Granularity: Protocols like NASDAQ’s ITCH offer deep granularity with full order book depth, while others might provide a top-of-book view or aggregated updates.
  • Transmission Schemes: Some protocols may use a single multicast channel per instrument, while others might group multiple instruments per channel or use a tiered channel system based on message priority.
  • Recovery Strategies: The method for handling packet loss can differ, with some using retransmission requests within the multicast stream and others offering separate unicast recovery channels.
  • Compression and Encoding: To optimize bandwidth, protocols may implement different compression schemes or encoding methods. Also protocols use different network byte orders (mostly use big-endian)
  • Versioning: The approach to protocol versioning and backward compatibility will vary, with some protocols requiring clients to update to new versions and others supporting multiple versions concurrently.

Packet and Block Structure

The application data is encapsulated in an UDP/IP frame (reference): Block Structure

In each UDP data block, all of the protocols use similar block data structures for delivering messages:

Block Structure

Different multicast protocols in detail

NASDAQ ITCH 5.0

  • Design: ITCH 5.0 is NASDAQ’s flagship data feed protocol, providing a detailed view of Nasdaq.
  • Functionality: Provides a detailed view of the order book with full-depth, order-level market data messages.
  • Data: Order-level data with attribution, trade messages, Net Order Imbalance Data and Administrative messages.

MoldUDP64

  • A networking protocol designed for one-to-many data broadcast contexts.
  • A lightweight protocol layer on top of UDP.

Packet Header: MoldUDP64 Packet Header

Message Block: MoldUDP64 Message Block

Sequence number:

The Sequence Number field of the packet Header indicates the sequence number of the first message in the packet. If there is more than one message contained in a packet, any messages following the first message are implicitly numbered sequentially.

Heartbeats:

Heartbeats are sent periodically by the server so receivers can sense packet loss even during times of low traffic. Typically, these packets are transmitted once per second and contain the next expected Sequence Number. A Heartbeat packet is a MoldUDP64 packet with a Message Count of 0.

Receiver with Recovery Example:

A typical MoldUDP64 receiver client would be configured with the following parameters:

  • The UDP port to listen on and the Multicast group to join
  • A list of one or more Request Servers that are available to answer retransmission requests for this stream. Each server is specified as a host IP address and a UDP port to which to send requests.
  • A session and sequence number of the next expected message if the client is being restarted.

A typical MoldUDP64 receiver client might obey the following flowchart:

  1. Open a UDP socket for the appropriate port and join the desired multicast group.

  2. Examine the first received packet to determine the currently active session.

  3. If the received session does not match the expected session, abort and report the error.

  4. Examine the sequence number of the first recently received packet.

  5. If the sequence number does not match the next expected sequence number, send a Request Packet to the Request Server with the expected packet number. Wait for a new packet and return to step 4.

  6. Process each of the received messages in the packet. If a Downstream Packet with the Message Count set to End of Session is received, handle the End of Session event.

  7. Wait for a new packet and return to step 4.

Documents:

NYSE Pillar

  • Design: NYSE Pillar is an integrated trading technology platform that uses a binary protocol for communication.
  • Functionality: Emphasizes reducing latency and improving the efficiency of the order lifecycle.
  • Data: Streams real-time, top-of-book, and depth-of-book quotations and trade messages.

Packet Header: MoldUDP64 Packet Header

Message Header: MoldUDP64 Message Block

Warning: NYSE Pillar message filed uses little-endian

Sequence Number:

Each message in a given channel is assigned a unique sequence number. Sequence numbers increase monotonically per channel, and can be used to detect publication gaps.

To optimize publication efficiency, the sequence number is not explicitly published in each message. Instead, the Packet Header contains the sequence number of the first message in the packet, along with the number of messages in the packet. Using these fields, the client can easily associate the correct sequence number with each message.

The sequence number combined with the channel ID form a message ID which is unique across the feed

Recovery Strategy:

NYSE Pillar uses a Pillar Request Server to handle the dropped multicast packets.

  • In case of dropped multicast packets, the client can connect to the Pillar Request Server via TCP/IP to request retransmissions of missed messages.
  • In case of client late start or intraday failure, the client can connect to the Pillar Request Server and request snapshot refreshes of the state of the market.

NYSE Request Server

Since multicast is an unreliable protocol, messages can be dropped. For this reason, clients are advised to process both lines in a channel. If a gap occurs on one line, the gap can be filled immediately from the other.

If a gap occurs on both lines simultaneously, the client can send a Retransmission Request Message (Msg Type 10) via TCP to the Request Server. The Retransmission Request contains a unique client ID called a Source ID, along with the Product and Channel IDs and the sequence number range of the missing messages.

Documents:

CBOE Multicast PITCH

  • Design: PITCH is CBOE’s multicast protocol for market data that provides a detailed view of CBOE’s markets, including Cboe BYX Exchange, BZX Exchange, EDGA Exchange, EDGX Exchange, BZX Options Exchange, Cboe Options Exchange (“C1”), C2 Options Exchange, and EDGX Options Exchange platforms.
  • Functionality: It offers a detailed look at CBOE’s market activity
  • Data: real-time depth of book quotations, execution information and auction update information during auctions.

Feed Connectivity:

  • Gig Shaped feeds requires >= 1Gb/s
  • 5 Gig and 8 Gig shaped feeds requires >= 10Gb/s
  • WAN-Shaped feeds requires minimum bandwidth requirements
  • feeds from the secondary datacenter will have additional latency
  • PTICH events are delivered using a range of multicast addresses divided by symbol
  • Dropped messages can be requested by Gap Request Proxy (GRP) servers
  • A spin of all open orders can be requested from a Spin Server
  • Low latency, up to 50% latency reduction vs. TCP PITCH

Multicast PITCH feed message flow:

Cboe Multicast PITCH real-time events are delivered using a published range of multicast addresses divided by symbol range units. Dropped messages can be requested using a TCP/IP connection to one of Cboe’s Gap Request Proxy (GRP) servers with replayed messages being delivered on a separate set of multicast ranges reserved for packet retransmission. Intraday, a spin of all open orders may be requested from a Spin Server. This allows a client to become current without requesting a gap for all messages up to that point in the day.

CBOE MEssage Flow

Protocol:

  • PITCH 2.X protocol over multicast
  • All messages are delivered using Sequenced Unit Header
  • All UDP delivered events will be self-contained. UDP delivered data will not cross frame boundaries and a single Ethernet frame will contain only one Sequenced Unit Header with associated data
  • TCP/IP delivered events from the GRP may cross frames
  • The PITCH data feed is comprised of a series of dynamic length sequenced messages. Each message begins with Length and Message Type fields. Cboe reserves the right to add message types and grow the length of any message without notice.

Sequence numbers:

Message sequence numbers start at one at the beginning of the day and are incremented by one for every sequenced message within a particular symbol unit. When the message sequence number reaches the maximum value of an unsigned 32-bit integer (2^32 – 1, or 4,294,967,295), the message sequence number will rollover to one (not zero which implies un-sequenced). The rollover handling must also be applied to the Gap and Spin servers

The Sequenced Unit Header is used for all Cboe Multicast PITCH messages as well as messages to and from the Gap Request Proxy (GRP) and Spin Servers.

Sequenced messages have implied sequences with the first message having the sequence number contained in the header. Each subsequent message will have an implied sequence one greater than the previous message up to a maximum of count messages. Multiple messages can follow a Sequenced Unit Header, but a combination of sequenced and un-sequenced messages cannot be sent with one header.

The sequence number for the first message in the next frame can be calculated by adding the Hdr Count field to the Hdr Sequence. This logic must account for sequence number rollover 4,294,967,295 to 1. This technique will work for sequenced messages and heartbeats.

Sequenced Unit Header

Heartbeats:

The Sequenced Unit Header with a count field set to “0” will be used for heartbeat messages. During trading hours heartbeat messages will be sent from the GRP and all multicast addresses if no data has been delivered within 1 second.

Outside of trading hours Cboe sends heartbeat messages on all real-time and gap channels with a sequence of “0” to help users validate multicast connectivity. Heartbeat messages may not be sent from 12:00 a.m. – 1:00 a.m. ET or during maintenance windows.

Cboe expects heartbeat messages to be sent to the GRP and Spin Servers on live connections no less than every 5 seconds. Failure to receive two consecutive heartbeat messages will result in the GRP or Spin Servers terminating the client connection.

Recovery Strategy:

  1. Gap server

The Gap Request message is used by a user’s process to request retransmission of a sequenced message (or messages) by one of Cboe’s gap servers.

Gap Server Usage Example:

The member detects a gap, having received sequence 4,294,967,293 and then sequence 3. The member recognizes this as a roll-over and sends a Gap Request to the GRP with Sequence 4,294,967,294 (first sequence in the range) and a Count of 4 (since zero is not included in a roll-over).

The GRP sends a Gap Response with a Sequence of 4,294,967,294 and Count of 4 (to match the request), with a Status of Accepted.

The Gap Server sends the requested gap messages which are sequences: 4,294,967,294 ; 4,294,967,295 ; 1 ; and 2. The member uses these messages to fill the gap.

  1. Spin server

The feed server provides a comprehensive and continuous real-time data feed covering all market events, while the spin server offers a specialized service that provides a snapshot of the current state of all open orders in the market. The data provided by the spin server is a subset of the data available in the feed server and is specifically useful for clients who need to quickly become current with the open order book without processing the entire historical data stream.

Spin Server Usage Example (with rollover):

Spin Server Rollover diagram

Documents:

CTA (Consolidated Tape Association)

  • Design: The CTA oversees the provision of real-time trade and quote information (CQS and CTS) for NYSE-listed securities.
  • Functionality: Provides consolidated market data feeds that include last-sale prices and volume information.
  • Data: Aggregates data across multiple exchanges, offering a comprehensive market view.

General Design of Data Distribution Network:

  1. Real-Time Production Data
  2. Real-Time Retransmission Data
  3. After-Hours Playback Data

Block Header (Both for CQS and CTS): CTA Block Header

Block Sequence Number (Both for CQS and CTS):

The Block Sequence Number denotes the sequence number of the first message in the block. If a Block contains more than one message, any messages following the first message are implicitly numbered sequentially. As such, the Block Sequence Number in the next Block is incremented by the number of messages published in the previous Block.

Block Sequence Number rollover occurs after 4,294,967,295. On a per multicast line basis, the Block Sequence Number on the multicast lines are set to zero at the start of each day, and incremented each time a block is transmitted, with the following exceptions:

  1. The Block Sequence Number in retransmitted blocks contains the Message Sequence Number of the first message in the retransmitted Block
  2. The Block Sequence Number field in the Block Header of a Category C Type L message (Reset Block Sequence Number) Block contains the number to which the Block Sequence Number counter is to be reset. This number is either one (1) in the event the sequence number rolls over from 4,294,967,295 or a number greater than the highest number previously transmitted.
  3. The block containing a Category C Type A message (Start of Day) or Category C Type Z message (End of Day) contains the Block Sequence Number one higher than the last transmitted message block.
  4. The block containing a Category C Type T (Line Integrity) message contains the message Sequence Number of the last block transmitted, which was not a retransmitted block.
  5. Should CTS experience a line failure and recovery, the Block Sequence Number for the recovered line(s) can be reset to a number greater than the last message sequence number transmitted once message transmission is resumed.

Message Header (Both for CQS and CTS): CTA Message Header

Heartbeat (Line Integrity):

Different from other exchanges, whether a message is a heartbeat cannot be decided through the block header. You have to go into the message header and parse Message Category and Message Type (CT) to judge if the message is a heartbeat.

Documents:

Recovery Strategy:

  • retransmission

UTP (Unlisted Trading Privileges)

  • Design: The UTP plan governs the distribution of real-time trade and quote data for NASDAQ-listed securities.
  • Functionality: Like CTA, it provides consolidated market data, essential for participants not using direct exchange feeds.
  • Data: Offers a wide array of market information, including quotations and trades from all UTP Plan participant markets.
  • Integration: UTP feeds are designed to be integrated with other market data systems, providing a holistic view of the market.

Since UTP also uses MoldUDP64 as the network transmission protocol, it uses the same packet structures. block structures and sequence recovery strategy as the one that Nasdaq uses.

Documents: