Core Facts
Provided with Core
Documentation	Reference design manual, Demo instruction manual
Design File Formats	Encrypted netlist file
Instantiation Templates	VHDL
Reference Designs & Application Notes	QuartusII Project, See Reference Design Manual
Additional Items	Demo on Arria 10 GX board
Support
Support provided by Design Gateway Co., Ltd.

Design Gateway Co.,Ltd

E-mail: ip-sales@design-gateway.com

URL: www.design-gateway.com

Features

· TCP/IP stack implementation

· Support IPv4 protocol without IP fragmentation

· Ultra-low latency data transmission, measured from start-of-packet to start-of-packet

· 6.2 ns latency time for transmitting data

· 46.5 ns lateny time for receiving data

· Support one session per one TOE10GLL IP (using multiple TOE10GLL IPs for multi-sessions)

· Support both Server and Client mode (Passive/Active open and close)

· Supported payload data size

· 1-1460 byte per packet for transmitted data (normal frame size, not jumbo frame)

· 1-16000 byte per packet for received data

· Transmit data buffer size: 4.6kByte – 73.7kByte

· User interface: 32-bit data stream interface

· EMAC interface: 32-bit Avalon-stream interface for connecting with DG LL10GEMAC IP or Intel Low Latency Ethernet 10G MAC IP

· Individual clock domain for transmit and receive interface at 322.266/312.5 MHz

· Reference design available on Arria10 GX development board

· Customized service for following features

· Jumbo frame support

· Buffer size extension by Window scaling feature

· Customized user interface

Table 1: Example Implementation Statistics

Family

Example Device

Fmax

(MHz)

ALMs¹

Registers¹

Block Memory bit²

Design

Tools

Arria10GX

10AX115S2F45I1SG

322.266

1705

2600

606,208

QuartusII20.1

Notes:

1) Actual logic resource dependent on percentage of unrelated logic

2) Block memory resources are based on 73.7kByte Tx data buffer size.

Applications

Figure 1: TOE10GLL IP Application

Nowadays the network with low latency access is the core for many real-time applications such as High Frequency Trading (HFT), Data center, and Real-time control system in Industrial, Remote surgery, and Automotive products. Using TCP/IP guarantees all data is received by the receiver. TOE10GLL IP is designed to transfer data via 10Gb Ethernet (BASE-R) with ultra-low latency time by using TCP/IP protocol.

Figure 1 shows the example system of HFT application. A trading venue uses TCP/IP protocol to send Market data. FPGA uses TOE10GLL IP with LL10GEMAC to decode the market data transferred via 10Gb Ethernet. After that, the decoded data is returned to Trading engine for processing the current price of properties. If the price can match with the requirement, the engine will send trading order to another TOE10GLL. Trading order is encapsulated by TOE10GLL IP following TCP/IP protocol and then forwarded to Trading Venue. The confirmation message is returned by Trading Venue after the order is success.

Reference design

Figure 2 Latency time measurement system on reference design

TOE10GLL IP has been launched with the standard demo design on Intel devevelopment board. In the demo, the IP is connected with LL10GEMAC IP and Intel 10GbE PMA (BASE-R) for transferring TCP data with the test application that is run on Test PC. The user interface of TOE10GLL IP is connected to the data generation logic for sending test data and data verification logic for verifying test data. When transmit packet size from user is less than or equal to 44 bytes, the latency time of transmitted data in TOE10GLL IP and LL10GEMAC is 6.2 ns and 18.6 ns respectively. Also, the latency of received data in LL10GEMAC and TOE10GLL IP is 21.7 ns and 46.5 ns respectively.

Note: Latency time is the minimum value when there is no pause time from LL10GEMAC. LL10GEMAC is sometimes not ready to transfer data for processing 64B/66B encoding/decoding logic.

General Description

Figure 3: TOE10GLL IP Block Diagram

TOE10GLL IP implements TCP/IP stack by hardwire logic for transfering 10Gb Ethernet packet with ultra-low latency. User interface of TOE10G IP consists of three interfaces, i.e., Network parameter interface for control signals, Transmit data interface, and Receive data interface.

To create TCP connection, the user needs to setup the network parameters for communicating with target device via 10Gb Ethernet. TOE10GLL IP receives the parameters from Control I/F such as MAC address, IP address, and Port number. Besides, the initialization mode and command are the signals of Control I/F, controlled by the user. After reset is de-asserted, Control block starts IP initialization to get MAC address of the target device, following the initialization mode by using the network parameters. After finishing, the IP is ready for transferring data with the target device.

Following TCP/IP standard, the connection must be created by opening the port as the first step. The user can select to open the port as active mode (Client) or passive mode (Server). Similarly, the connection can be terminated as active mode or passive mode. The data can be transferred when the connection is active. The user can send the data with setting packet size via Transmit data interface of TOE10GLL IP. On the contrary, when TOE10GLL IP receives the packet, the data in the packet is decoded and returned to the user via Receive data interface.

Tx data buffer size can be set as TOE10GLL IP parameters for balancing the memory resource utilization against transfer performance in the system. More details of the hardware inside the IP are described in the next topic.

Functional Description

As shown in Figure 3 TOE10GLL IP consists of three parts, i.e., Control block, Transmit block, and Receive block. Control block and Transmit block are synchronous with TxClk while Receive block is synchronous with RxClk.

Control Block

Control block is the interface module with the user to receive the network parameters and the command via Control I/F. When running transmitting data, Control block decodes the complete transmit data size from received packet. After that, the completed data size is forwarded to the user via Transmit interface to check the progress of data transmission. Status signals and error signals are created by Control block when received packet is transferred in wrong order or the packet is broken.

· NetParam Reg

This module is designed to store the network parameters from the user. Most network parameters such as MAC address and IP address are loaded when RstB is de-asserted from ‘0’ to ‘1’ for starting IP initialization. While port numbers are loaded when the port is created. As a result, the port number can be changed without re-asserting RstB signal. The network parameters are applied to build the header of transmitted packet and validate the header of received packet.

· TCP Stack

Figure 4 TCP Stack in normal condition

TCP Stack controls packet transferring with the target device following TCP/IP protocol. As shown in Figure 4, after finishing reset process, the TCP stack operation has two phases - IP initialization phase and data transferring phase.

During running initialization phase, TCP Stack gets MAC address of the target device by three modes, i.e., Client mode (extracted from ARP reply packet), Server mode (extracted from ARP request packet), and Fixed MAC mode (loaded from the user input), selected by DstMacMode signal. After that, the target MAC address and other network parameters are loaded to the internal registers for creating the transmitted packet and validating the received packet. If there is no error detected in this phase, the stack continues to data transferring phase.

The first step before transferring the data is port creation which can be processed by active mode or passive mode, controlled by TCPCmd signal. After completing port creation, the stack is in Port Active state which is ready state for transferring the data with the target device.

To transfer data, Transmit block and Receive block must be operated for sending the packet and decoding the received packet at the same time. When TOE10GLL IP sends the data, the data packet is created by Transmit block and ACK packet is decoded by Receive block to check the number of data that is completely received. When TOE10GLL IP receives the data, Receive block extracts the data from the valid packet. If the valid data is received, Transmit block creates ACK packet to confirm the number of the data that can be completely received.

If the data lost is found, data recovery process is run. During sending data, if the target device detects the lost packet, it will create duplicate ACK to TOE10GLL IP. After the stack detects duplicate ACK packet during sending process, Transmit block stops current data transmission. Next, Transmit block re-sends the lost packet. On the contrary, if the stack detects the lost data packet during receiving data, the stack will send duplicate ACK to request the lost data from the target device. The skipped data is ignored.

The last step is port termination which can be run in active mode or passive mode. The user sends the command to close the connection when using active close. Otherwise, the passive close can be operated after FIN packet from the target device is received. After the port is terminated, the stack changes to Idle state to wait for the new connection.

During sending data, if TOE10GLL IP detects the error packet from the user such as wrong packet length, TOE10GLL IP will change to Error state. It needs to assert RstB to ‘0’ to restart the IP operation.

Transmit Block

There are four packet types which are created by Transmit block, i.e., ARP request packet, ARP reply packet, TCP packet without TCP data, and TCP packet with TCP data. Tx data buffer and Tx packet buffer are applied for creating TCP packet with TCP data. While other packet types are built by Packet builder. Tx data buffer size is designed as IP parameter, set by HDL code. Using bigger buffer size may increase the transmit performance. The data in Tx data buffer can be flushed after the target device returns ACK packet to confirm that the data is completely received.

· Tx Data Buffer

The depth of Tx data buffer can set by “TxBufBitWidth”, parameter of TOE10GLL IP. The valid value is 10 – 14 for selecting the buffer depth to be 1024 (2¹⁰) – 16384 (2¹⁴). Data width of Tx data buffer is 36 bits, 32-bit data and 4-bit control signals.

Table 2 TxBufBitWidth parameter

Value	Buffer Size
10	1024 x 36-bit (4.6 kByte)
11	2048 x 36-bit (9.2 kByte)
12	4096 x 36-bit (18.4 kByte)
13	8192 x 36-bit (36.9 kByte)
14	16384 x 36-bit (73.7 kByte)

The buffer stores TCP data for transmitting TCP packet with TCP data when the user sends the data to TOE10GLL IP via Transmit interface. According to TCP/IP protocol, the data can be retransmitted if the receiver detects the lost packet and sends the request. Therefore, the data from the user must be stored in Tx data buffer until the target device accepts the data completely, monitored from received ACK packet. The received ACK packet is decoded to check the data position which is completely received. To achieve the best transmit performance, Tx data buffer size must be big to compensate the latency time from processing time of target device and network routing. If the size is too small, all data in the buffer is sent by TOE10GLL IP without receiving ACK from the target. The data transmission must be paused to wait for ACK packet from the target device to clear the buffer.

· Tx Packet Buffer

This buffer is the temporary transmit buffer for storing TCP data from Tx data buffer before forwarding to Packet builder.

· Packet Builder

Packet Builder loads the network parameters from NetParam Reg module to create the packet header. If the packet includes TCP data, the data is read from Tx packet buffer to build the complete packet. Besides, IP and TCP checksum are calculated to be a part of TCP header. After that, the complete packet is transmitted to EMAC.

Receive Block

The received packet from EMAC is validated by Receive block. If the network parameters of the packet header are correct and the packet includes TCP data, only TCP data is extracted and forwarded to the user via Receive interface. If the packet has the error, the packet will be rejected, except two cases. When TCP checksum is error or EMAC shows error CRC in TCP with data packet, TCP data is forwarded to the user logic, but error signal is asserted at the end of user packet.

To achieve the lowest latency time, there is no big buffer inside Receive block. According to TCP/IP protocol, there is a parameter, called Window size, which shows the free space of the receive buffer inside the network device. The window size of TOE10GLL IP is calculated by 65535 – TCPRxBufWrCnt, the input from the user. The best transfer performance in receive path is achieved if TCPRxBufWrCnt value is equal to 0. When TCPRxBufWrCnt is equal to 65535, the target device will pause data transmission. The target device can continue sending the data to IP when the Window size is enough to store the data packet.

· Packet Decoder

Packet decoder decodes the received packet type that is ARP request, ARP reply, or TCP packet. Otherwise, the packet is rejected. After that, the valid packet is forwarded to Header filtering.

· Header Filtering

The header filtering verifies the header of the received packet. The packet is valid and forwarded to the next module when the following conditions are met.

(1) The packet is ARP packet or TCP/IPv4 packet without data fragment flag.

(2) Network parameters are matched to the value in NetParam Reg module, i.e., MAC address, IP address, and Port number.

(3) IP header length and TCP header length are valid (IP header length is equal to 20 bytes and TCP header length is equal to 20 - 60 bytes).

(4) IP checksum is correct.

(5) The data pointer decoded by the sequence number is in valid range.

(6) The acknowledge number is in valid range.

· Data Validation

When the packet is TCP packet with the data, TCP data is extracted from the packet and forwarded to the user. During forwarding TCP data to the user, Data validation calculates TCP checksum of the packet. If calculated TCP checksum is not equal to the value in the packet header, error signal is asserted to the user at the end of packet. Therefore, the user logic needs to reject the received data from TOE10GLL IP if the error is found.

10 Gb Ethernet MAC

To achieve the lowest latency, it is recommended to use LL10GEMAC IP from DesignGateway for connecting with TOE10GLL IP. More details of LL10GEMAC IP are described in following website.

https://dgway.com/Lowlatency-IP_A_E.html

Another solution of 10Gb Ethernet MAC is Low Latency Ethernet 10G MAC, provided by Intel. The user must select the data bus to be 32-bit interface for the low latency solution. More details of the IP are described in following website.

https://www.intel.com/content/www/us/en/programmable/products/intellectual-property/ip/interface-protocols/m-alt-10gbps-ethernet-mac.html

User Logic

Similar to TOE10GLL IP, the user logic must run by using two clock domains. Control interface which is applied to define the network parameters and the command is run on TxClk. The network parameters can be assigned by registers or constant value. Simple state machine can be designed for sending the command with the request and monitoring the status on Control interface.

The transmit interface for sending the data to the target device is run on TxClk by using 32-bit data stream interface. One packet data must be sent continuously. Flow control signals of data stream interface are valid signal and ready signal, similar to Avalon-stream interface. Besides, the user logic sets packet parameters such as packet size at the same clock as sending the first data of each packet. The completed transmit length is returned from TOE10GLL IP when the target device returns ACK to confirm the complete received data position.

The receive interface for receiving the data from TOE10GLL IP is run on RxClk by using 32-bit data stream interface. The data flow is valid signal without ready signal. Therefore, the user logic must be always ready to receive the data. To control the number of received data, TCPRxBufWrCnt is applied to calculate the free size of user buffer. The free size is forwarded to the target device by assigning to be Windows size in TCP header. The target device can send the data to the IP when free space of user buffer is enough.

Core I/O Signals

Descriptions of all parameters and I/O signals are provided in Table 3 - Table 5.

Table 3: Core Parameters

Name	Value	Description
TxBufBitWidth	10-14	Assign Tx Data buffer depth which is equal to (2 ** TxBufBitWidth).

Table 4: User I/O Signals

Signal	Dir	Description
General Interface
IPVersion[31:0]	Out	IP version number
TestPin[31:0]	Out	Reserved to be IP Test point
RstB	In	Synchronous reset signal with TxClk. Asserted to ‘0’ to reset the IP and then de-asserted to ‘1’ for starting IP initialization by using the latest parameters, assigned by the user.
TxClk	In	Clock signal which is synchronous to Tx EMAC interface and Tx user data interface. When running with low-latency 10G EMAC, the frequency is 322.266 MHz.
RxClk	In	Clock signal which is synchronous to Rx EMAC interface and Rx user data interface. When running with low-latency 10G EMAC, the frequency is 322.266 MHz.
Control interface (Synchronous to TxClk)
DstMacMode[1:0]	In	Mode for loading MAC address of the target device “00”: Auto-client mode. MAC address is extracted from ARP reply during IP initialization. “01”: Auto-server mode. MAC address is extracted from ARP request during IP initialization. “1x”: Fixed MAC mode. MAC address is loaded from DstMacAddrIn signal during IP initialization.
InitFinish	Out	IP initialization process is finished. ‘0’: IP is in reset process or initialization process. ‘1’: IP initialization is completed.
SrcMacAddr[47:0]	In	MAC address of TOE10GLL IP. The signal must be valid before RstB is de-asserted from ‘0’ to ‘1’.
SrcIPAddr[31:0]	In	IP address of TOE10GLL IP. The signal must be valid before RstB is de-asserted from ‘0’ to ‘1’.
DstMacAddrIn[47:0]	In	MAC address of the target device, applied when DstMacMode[1]=’1’ (Fixed MAC mode) The signal must be valid before RstB is de-asserted from ‘0’ to ‘1’.
DstIPAddr[31:0]	In	IP address of the target device. The signal must be valid before RstB is de-asserted from ‘0’ to ‘1’.
DstMacAddrOut[47:0]	Out	MAC address of the target device. Valid when InitFinish is asserted to ‘1’.
TimeOutSet[31:0]	In	Timeout value for waiting the packet returned from the target device before starting recovery process. Valid from 1 – 0xFFFFFFFF. Time unit is equal to TxClk period. The signal must be valid before RstB is de-asserted from ‘0’ to ‘1’.
IPInt	Out	IP timeout interrupt. Assert to high for one cycle when timeout is detected. More details of Interrupt status are monitored from IntStatus[19:0].

Signal	Dir	Description
Control interface (Synchronous to TxClk)
IntStatus[31:0]	Out	Interrupt status to show the details of the interrupt. IPInt is asserted to ‘1’ when IntStatus[19:0] is not equal to 0. Other bits are applied for IP monitoring. The signal is valid when IPInt is asserted to ‘1’. [0]-Interrupt when ARP reply packet is not received during IP initialization in Auto-client mode. After that, the IP retries by sending ARP request until ARP reply is received. [1]-Interrupt when SYN\|ACK packet is not received during active open operation. After that, the IP retries by sending SYN packet for 16 times. Finally, FIN packet is sent. [2]-Interrupt when ACK packet is not received during passive open operation. After that, the IP retries by sending SYN\|ACK packet for 16 times. Finally, FIN packet is sent. [3]-Interrupt when FIN\|ACK packet is not received during active close operation. After that, the IP sends RST packet. [4]-Interrupt when ACK packet is not received during passive close operation. After that, the IP retries by sending FIN\|ACK packet for 16 times. Finally, RST packet is sent. [5]-Interrupt when ACK packet is not received during sending data operation. After that, the IP retries by retransmitting the previous data packet. [6]-Interrupt when the IP detects data lost during receiving data operation. After that, the IP generates duplicate ACK to request data retransmission. [7]-Interrupt when the IP pauses sending data because the window size of the target device is less than 2Kbyte. After that, the IP transmits reverse data packet to activate the target device retransmitting the ACK packet with the updated windows size. [8]-Interrupt when the window size of the target device during open connection is too less. After that, the IP sends RST packet. [9]-Interrupt when RST packet is received when connection is active. After that, the connection is terminated. [12]-Interrupt when the transmit packet length set by the user is not equal to total data size of a packet sent via Tx data interface. After that, IPState is equal to STERR (11111b). The user needs to assert RstB signal to reset the IP. [13]-Interrupt when the user de-asserts valid signal during sending a packet to TOE10GLL IP. After that, IPState is equal to STERR (11111b). The user needs to assert RstB signal to reset the IP. [16]-Interrupt when the IP receives data but TCPRxBufWrCnt shows full status. After that, the IP sends duplicate ACK. [21]/[27]-Asserted to ‘1’ when data is lost during receiving data operation. After that, IPInt and IntStatus[6] are asserted ‘1’. [30]-Asserted to ‘1’ when RST packet is received. [31],[29:28],[26:24]-Internal test status
IPState[4:0]	Out	IP state. “00000” – STIPRST : IP Reset state (RstB is asserted to ‘0’) “00001” – STINIT : Initialization state “00010” – STIDLE : Idle state (No connection) “00011” – STAOP : Active open state “00100” – STPOP : Passive open state “00101” – STPACT : Port Active state (Connection is ON and ready for data transmission) “00110” – STACL : Active close state “00111” – STPCL : Passive close state “01000” – STTRST : TCP Reset state (Send or receive RST packet) “11111” – STERR : Error state (Error from user on Transmit interface during sending operation)

Signal	Dir	Description
Control interface (Synchronous to TxClk)
TCPCmd[1:0]	In	Command input. “00” – Active close, “01” – Active open, “1x” – Passive open. The signal must be valid when TCPCmdValid is asserted to ‘1’.
TCPCmdValid	In	Command request. Asserted to ‘1’ for one cycle to send the new command. Note: The user must read IPState before asserting TCPCmdValid to ‘1’, described as follows. (1) Before sending Active or Passive open command, IPState must be equal to STIDLE (00010b). If the connection is created completely, IPState will change to STPACT (00101b). (2) Before sending Active close command, IPState must be equal to STPACT (00101b). If the connection is terminated completely, IPState will change to STIDLE (00010b).
TCPSrcPort[15:0]	in	Port number of TOE10GLL IP. The signal must be valid when TCPCmdValid is asserted to ‘1’ for Active or Passive open command.
TCPDstPort[15:0]	in	Port number of the target device. The signal must be valid when TCPCmdValid is asserted to ‘1’ for Passive open command.
TCPConnOn	Out	Connection Status. ‘1’: connection is active, ‘0’: no connection.
Tx data interface (Synchronous to TxClk)
TCPTxData[31:0]	In	Transmitted TCP data. Valid when TCPTxValid=’1’.
TCPTxValid	In	Asserted to ‘1’ when TCPTxData is valid. To transmit one packet, this signal must be always asserted to ‘1’ until the end of packet to send the data continuously.
TCPTxEOP	In	Asserted to ‘1’ with the last data of the packet on TCPTxData. Valid when TCPTxValid is asserted to ‘1’.
TCPTxPkLen[10:0]	In	Transmit packet size in byte unit. Valid with the first data of each packet.
TCPTxPSH	In	PSH flag asserted in TCP header of this packet. Valid with the first data of each packet.
TCPTxReady	Out	IP ready signal.’0’-IP is not ready to receive data, ‘1’-IP is ready to receive data. When TCPTxReady is de-assserted to ‘0’, TCPTx signals (inputs from user) must hold the same value.
TCPTxBufEmpty	Out	‘1’: All data in Tx data buffer is completely received by the target device ‘0’: Data in Tx data buffer is remained
TCPTxCplLen[15:0]	Out	Show the completed data size which the target is completely received in byte unit. Valid when TCPTxCplValid=‘1’.
TCPTxCplValid	Out	Asserted to ‘1’ for one cycle when TCPTxCplLen is valid.
Rx data interface (Synchronous to RxClk)
TCPRxData[31:0]	Out	Received TCP data. Valid when TCPRxValid=’1’.
TCPRxValid	Out	Asserted to ‘1’ when TCPRxData is valid.
TCPRxByteEn[3:0]	Out	Byte enable of TCPRxData. Valid when TCPRxValid=’1’. During packet transmission, this signal is equal to 1111b for enabling all 32-bit data except the last data of the packet (TCPRxEOP=’1’) which can be equal to four values - 0001b, 0011b, 0111b, or 1111b when TCPRxData[7:0], [15:0], [23:0] or [31:0] is valid respectively.
TCPRxSOP	Out	Start of packet. Asserted to ‘1’ when sending the first data of the packet. Valid when TCPRxValid=’1’.
TCPRxEOP	Out	End of packet. Asserted to ‘1’ when sending the last data of the packet. Valid when TCPRxValid=’1’.
TCPRxError[7:0]	Out	Error status of the received packet. Valid at the end of packet (TCPRxEOP=’1’ and TCPRxValid=’1’). [0]-TCP checksum of the packet is error. [1]-Error detected by EMAC. [7-2]-Reserved
TCPRxBufWrCnt[15:0]	In	Data counter of Rx buffer in user logic to calculate free space size which is equal to (65535 – TCPRxBufWrCnt). The free size is applied to be Window size returned to the target device.

Table 5: EMAC I/O signals

Signal	Dir	Description
Tx MAC interface (Synchronous to TxClk)
MacTxData[31:0]	Out	Transmitted data. Valid when MacTxValid=’1’.
MacTxValid	Out	Asserted to ‘1’ when MacTxData is valid.
MacTxEmpty[1:0]	Out	Specify the number of bytes which are unused of the final word in the frame. The signal is valid when MacTxValid and MacTxEOP are asserted to ‘1’.
MacTxSOP	Out	Asserted to ‘1’ when transmitting the first data of the packet. Valid when MacTxValid=’1’.
MacTxEOP	Out	Asserted to ‘1’ when transmitting the last data of the packet. Valid when MacTxValid=’1’.
MacTxReady	In	Asserted to ‘1’ by EMAC IP when the transmitted data is received correctly. If MacTxReady=’0’, transmitted data and control signals (MacTxData, MacTxValid, MacTxByteEn, MacTxSOP, and MacTxEOP) must hold the same value until MacTxReady is re-asserted to ‘1’.
Rx MAC interface (Synchronous to RxClk)
MacRxData[31:0]	In	Received data. Valid when MacRxValid=’1’.
MacRxValid	In	Asserted to ‘1’ when MacRxData is valid.
MacRxEOP	In	Asserted to ‘1’ when receiving the last data of the packet
MacRxError	In	Asserted to ‘1’ at the end of packet when the packet has error. Valid when MacRxValid=’1’ and MacRxEOP=’1’.

Note: According to Intel 10G Ethernet MAC specification, the data endian of TCPTxData/TCPRxData and MacTxData/MacRxData are swapped. For example, the first data byte of TCPRxData is transferred by bit[7:0], while the first byte of MacRxData is transferred by bit[31:24].

Timing Diagram

IP Reet

When the user asserts RstB signal to ‘0’, all logics inside TOE10GLL IP are reset and IPState shows STIPRST, as shown in Figure 5.

Figure 5: IP Reset process

(1) Before the user de-asserts RstB to ‘1’, the user must set the valid value of initialization mode (DstMacMode), Timeout value (TimeOutSet), and network parameters, i.e., SrcMacAddr, SrcIPAddr, DstMacAddrIn (when using Fixed MAC mode), and DstIPAddr.

(2) After RstB changes from ‘0’ to ‘1’, TOE10GLL loads all input parameters. IPState changes to STINIT during running IP initialization.

The details of each initialization mode are described in more details as follows.

IP Initialization

There are three modes to initailze TOE10GLL IP, set by DstMacMode, i.e., Client mode (DstMacMode=”00”), Server mode (DstMacMode=”01”), and Fixed MAC mode (DstMacMode=”1x”). Figure 6 shows timing diagram when initializing in Client mode.

Figure 6: IP Initialization in Client mode

(1) User sets DstMacMode to “00” and then de-asserts RstB to ‘1’ to start IP initialization in Client mode.

(2) IPState changes status from STIPRST to STINIT. After that, TOE10GLL IP sends ARP request packet to the target device.

(3) After the target device detects ARP request, it generates ARP reply to return MAC address.

(4) TOE10GLL IP receives the ARP reply packet and then extracts MAC address of the target device. After finishing the initialization process, IPState changes to STIDLE and InitFinish is asserted to ‘1. DstMacAddrOut is also valid to show the MAC address of the target which is extracted from ARP reply packet.

Next, timing diagram of TOE10GLL IP when running the initialization in Server mode is displayed in Figure 7

Figure 7: IP Initialization in Server mode

(1) User sets DstMacMode to “01” and then de-asserts RstB to ‘1’ to start IP initialization in Server mode.

(2) IPState changes status from STIPRST to STINIT. After that, TOE10GLL IP waits until ARP request packet is received from the target device.

(3) TOE10GLL IP extracts MAC address of the target device from ARP request and then returns ARP reply.

(4) After that, the IP finishes the initialization process. IPState changes to STIDLE and InitFinish is asserted to ‘1. DstMacAddrOut is also valid to show the MAC address of the target which is extracted from ARP request packet.

The last initialization mode is Fixed MAC mode. The target MAC address is defined by DstMacAddrIn signal. Timing diagram when running this mode is dispayed in Figure 8.

Figure 8: Initialization in Fixed MAC mode

(1) User sets DstMacMode to “1X” and then de-asserts RstB to ‘1’ to start IP initialization in Fixed MAC mode.

(2) IPState changes status from STIPRST to STINIT. After that, TOE10GLL IP starts the initialization process.

(3) After finishing the initialization process, IPState changes to STIDLE and InitFinish is asserted to ‘1. DstMacAddrOut is equal to DstMacAddrIn.

Open Connection

According to TCP/IP standard, the connection must be opened as the first step before starting data transmission. There are two modes to open the port by TOE10GLL IP, i.e., Active open by setting TCPCmd=01b and Passive open by setting TCPCmd=1Xb. Before sending open connection command, IPState must be equal to STIDLE. More details of open connection in Active and Passive mode are described as follows.

Figure 9: Active open connection

(1) User sets TCPCmd=01b with the valid TCPSrcPort (TOE10GLL Port) and TCPDstPort (Target device Port) and asserts TCPCmdValid to ‘1’ for one cycle to send Active open command.

(2) Within 20 clock cycles after receiving the user request, IPState changes from STIDLE to STAOP. After that, TOE10GLL IP sends SYN packet to the target device for creating the new connection.

(3) The target device accepts the new connection request and then returns SYN|ACK packet.

(4) TOE10GLL IP detects the acknowledge from the target and then returns ACK packet to complete the new connection establishment.

(5) IPState changes to STPACT and TCPConnOn is asserted to ‘1’ to show connection creating is completed. After that, the IP is ready to transfer data with the target device in both directions. TCPTxReady is asserted to ‘1’ in the next clock to allow data transmission with the user.

Note: If the target device does not return SYN|ACK packet until timeout, TOE10GLL IP re-transmits SYN packet with asserting IPInt signal for 16 times. After the last retry time, the IP sends FIN packet to cancel the operation and IPState changes to STIDLE.

Figure 10: Passive open connection

(1) User sets TCPCmd=1Xb with the valid TCPSrcPort (TOE10GLL Port) and then asserts TCPCmdValid to ‘1’ for one cycle to send Passive open command.

(2) Within 4 clock cycles after receiving the user request, IPState changes from STIDLE to STPOP. In this state, TOE10GLL IP waits until the new connection is requested on TCPSrcPort number.

(3) SYN packet is sent by the target device. The IP continues the next step if the parameters in the packet is matched. Otherwise, the request is ignored.

(4) TOE10GLL IP sends SYN|ACK packet to accept the new connection request.

(5) ACK packet is received to complete the connection establishment process.

(6) IPState changes to STPACT and TCPConnOn is asserted to ‘1’. In the next clock, TCPTxReady is asserted to ‘1’ to allow data transmission with the user.

Note: If the target device does not return ACK packet in step (5) until timeout, TOE10GLL IP re-transmits SYN|ACK packet with asserting IPInt signal for 16 times. After the last retry time, the IP sends FIN packet to cancel the operation and IPState changes to STIDLE.

Close Connection

Similar to Open connection process, when there is no data transmission between two devices, the connection should be terminated. There are two modes for closing the port by TOE10GLL IP, i.e., Active close by setting TCPCmd=11b and Passive close which is requested by the target device. The details of close connection in each mode are described as follows.

Figure 11: Active close connection

(1) Before sending Active close command, user needs to confirm that there is no data transmission remained in TOE10GLL IP by checking if TCPTxBufEmpty=’1’. After that, user asserts TCPCmdValid to ‘1’ and TCPCmd=00b to send Active close command.

(2) Within 4 clock cycles after receiving the user request, IPState changes from STPACT to STACL. After that, TCPTxReady is de-asserted to ‘0’ to stop data transmission from the user.

(3) TOE10GLL IP sends FIN|ACK packet to the target device for terminating the connection.

(4) The target device accepts to terminate the connection and returns FIN|ACK packet.

(5) TOE10GLL IP returns ACK packet to complete the connection termination.

(6) IPState changes to STIDLE and TCPConnOn is de-asserted to ‘0’. The user can open the new connection by using the different port number, defined by TCPSrcPort and TCPDstPort.

Note: If the target device does not return FIN|ACK packet in step (4) until timeout, TOE10GLL IP sends RST packet to close the connection with asserting IPInt signal once. After that, IPState changes to STIDLE.

Figure 12: Passive close connection

(1) The passive close operation begins when TOE10GLL IP receives FIN|ACK packet from the target device in STPACT state. After that, IPState changes to STPCL to start passive close operation.

(2) TCPTxReady is de-asserted to ‘0’ to stop data transmission from the user. Typically, the data should not be remained in TOE10GLL IP and TCPTxBufEmpty is asserted to ‘1’. However, this example shows the example when some data is remained in Tx data buffer.

(3) TOE10GLL IP sends FIN|ACK packet to accept the request for terminating the connection.

(4) The target device returns ACK packet to complete the connection termination.

(5) After receiving ACK packet, IPState changes to STIDLE and TCPConnOn is de-asserted to ‘0’. At the same time, TCPTxBufEmpty is asserted to ‘1’ because all data in Tx data buffer is flushed. The user can open the new connection by using the different port number, defined by TCPSrcPort and TCPDstPort.

Note: If the target device does not return ACK packet in step (4) until timeout, TOE10GLL IP re-transmits FIN|ACK packet with asserting IPInt signal for 16 times. After the last retry time, the IP sends RST packet to close the connection and IPState changes to STIDLE.

When the data transmission by TCP/IP protocol has critical problem and the recovery process cannot solve the problem, RST packet may be sent to terminate the connection. If TOE10GLL IP receives RST packet while IPState is equal to STPACT, the connection will be closed as shown in Figure 13

Figure 13: Connection closed by RST packet

(1) When RST packet is received from the target device and IPState is equal to STPACT, TOE10GLL IP starts the reset connection process which is similar to closing the connection process.

(2) There is no packet transmitted by TOE10GLL IP during running reset connection. After finishing, all signals change to no connection status, as follows.

- IPState changes to STIDLE.

- TCPConnOn is de-asserted to ‘0’ to show there is no active connection.

- TCPTxBufEmpty is asserted to ‘1’ after all data inside Tx data buffer is flushed.

- TCPTxReady is de-asserted to ‘0’ to stop data transmission from the user.

Tx data Interface

When the connection is active and TOE10GIP LL is ready to receive data, TCPTxReady is asserted to ‘1’. When sending the first data of a packet, user logic needs to check if TCPTxReady is asserted to ‘1’. Figure 14 shows the example for sending two packets to TOE10GLL IP, 12-byte packet and 18-byte packet respectively.

Tx user data input

Figure 14: Send a packet data on Tx data interface

(1) The first data of the packet is sent on TCPTxData with asserting TCPTxValid to ‘1’. At the same time, user must set the packet length on TCPTxPkLen and PSH flag on TCPTxPSH. When TOE10GLL IP is ready, TCPTxReady is asserted to ‘1’ to accept the request and load all inputs in this clock. After that, TCPTxReady is asserted to ‘1’ until the end of packet for receiving a packet data continuously.

(2) User sends the next data (D1) and the remained data in the packet continuously. Therefore, TCPTxValid is asserted to ‘1’ until the end of packet.

(3) The last data of the packet is sent on TCPTxData with asserting TCPTxEOP to ‘1’. Total numbers of data must be matched to the value set on TCPTxPkLen at the start of packet. For example, when packet length is equal to 12 bytes, three 32-bit data (D0-D2) are transferred on TCPTxData.

(4) After finishing the packet transmission, TCPTxReady is de-asserted to ‘0’ at least two clock cycles to run packet post-processing. TCPTxReady will be de-asserted more than two clock cycles by three conditions.

(a) EMAC is not ready to receive the next packet.

(b) Tx data buffer is full because the target device does not return ACK packet.

(c) The packet length set by user is not aligned to 4-byte and it needs to wait until the target device returns ACK packet to confirm all data is received completely.

(5) If user sends the new packet but IP is not ready (TCPTxReady is de-asserted to ‘0’), all inputs (TCPTxPkLen, TCPTxPSH, TCPTxValid, and TCPTxData) must hold the same value until TCPTxReady is re-asserted to ‘1’ to accept the next data packet.

(6) When the packet size is not aligned to 4-byte and the last data on TCPTxData is valid only the lower-byte, the remaining data on the upper byte must be filled by zero value. For example, when packet length is 18 bytes, the last data (D7) is valid only bit[15:0]. Bit[31:16] of the last data (D7) must be filled by 0000h.

(7) When the packet is not aligned to 4-byte, TOE10GLL IP must wait until the target device returns ACK packet to confirm that all data is received completely. After that, TCPTxBufEmpty is asserted to ‘1’. In the next clock, TCPTxReady is re-asserted to ‘1’ to accept the next packet from the user.

Note:

(1) PSH flag is a part of TCP flag inside TCP/IP header which is applied to push the data for starting processing without waiting the additional data.

(2) To get the best performance for transferring data, it is recommended to use the packet size which is aligned to 4-byte. TCPTxReady can be re-asserted to ‘1’ to accept the new packet without waiting for TCPTxBufEmpty asserted to ‘1’.

(3) Tx data latency time is related to the packet size which shows more details in Figure 16 - Figure 17.

TCPTxCpl Interface

The data completion of Tx data interface is monitored by two ways. First, TCPTxBufEmpty is applied to check the completed status of all data. Second, TCPTxCpl interface is designed to check the progress of completed data. The simple logic is designed when using TCPTxBufEmpty. TCPTxBufEmpty changes from ‘0’ to ‘1’ when all data is completely received by the target device. While TCPTxCpl interface shows the number of the completed length to show the latest progress. More details are shown in Figure 15.

Figure 15: TCPTxCplLen interface

(1) Assume that the user sends three data packets which all are aligned to 4-byte continuously, i.e., 8-byte packet, 12-byte packet, and 16-byte packet.

(2) After the target device returns ACK packet to confirm the data position which is completely received, TOE10GLL IP calculates the completed byte length. After that, the IP returns the completion length to the user by asserting TCPTxCplValid to ‘1’ with the complete received byte on TCPTxCplLen.

(3) ACK packet from the target may not match to the transmit length, set by the user on TCPTxPkLen. When TCP delayed ACKs feature is implemented on the target, ACK packet may be returned after receiving many data packets. As shown in Figure 15, the complete length is equal to 20 bytes and 16 bytes respectively. Finally, total sum of TCPTxPkLen (8 + 12 + 16 = 36 bytes) must be equal to total sum of TCPTxCplLen (20 + 16 = 36 bytes).

(4) After the last complete length is sent on TCPTxCpl interface, TCPTxBufEmpty is asserted to ‘1’ to show all data in Tx data buffer is transmitted completely.

Tx data latency time

Tx data latency time depends on many issues such as packet length, EMAC status, and Tx data buffer status. The lowest latency time on Tx data interface is achieved when running in following conditions.

- Packet length (TCPPkLen) is less than or equal to 44 bytes.

- EMAC is ready for receiving the data (MacTxReady=’1’).

- TOE10GLL IP is ready for receiving data from the user (TCPTxReady=’1’) and there is no data remained in Tx data buffer (TCPTxBufEmpty=’1’).

Figure 16 shows timing diagram when all above conditions are matched. Tx data latency time, measured by start-of-packet of user interface and EMAC interface, is equal to 2 clock cycles or 6.2 ns @ 322.265625 MHz.

Figure 16: Tx latency time when packet size is less than or equal 44 bytes

(1) When TOE10GLL IP is Idle (TCPTxBufEmpty=’1’) and EMAC is ready to receive the data (MacTxReady=’1’), the user sets small packet length which is less than or equal to 44 bytes to TOE10GLL IP with transmitting the first data of a packet (D0) by asserting TCPTxValid to ‘1’.

(2) After two clock cycles, the first data of TCP packet (TCP0) is transmitted to EMAC by asserting MacTxSOP and MacTxValid to ‘1’.

(3) During data transmission, if MacTxReady is de-asserted to ‘0’, MacTx output signals from TOE10GLL will latch the same value until MacTxReady is re-asserted to ‘1’.

When the packet size is more than 44 bytes, tx data latency time depends on the packet size which can be calculated by following equation.

Tx data latency time (clock cycle) = Roundup (TCPTxPkLen / 4) – 9

For example, when TCPTxPkLen is equal to 45 – 48 bytes, Tx data latency time = 3 clock cycles.

Figure 17: Tx latency time when packet size is more than 44 bytes

Note:

(1) Latency time in Figure 16 and Figure 17 are the minimum value when both TOE10GLL IP and EMAC are ready for transferring data.

(2) If EMAC is not ready by de-asserting MacTxReady to ‘0’, TOE10GLL IP waits until it is re-asserted to ‘1’ for starting transferring the data. EMAC can be de-asserted to ‘0’ when synchronous gearbox is applied in Physical layer. Therefore, latency time can be increased to wait for EMAC ready.

(3) If TOE10GLL IP has the data inside Tx data buffer, it needs to wait until the IP transmits all data to the target device. Therefore, latency time is increased to wait for the IP transmitting all data.

Rx data Interface

When the packet is received from EMAC, TOE10GLL IP compares the network parameters of a packet with the set value by the user. If the parameters are correct, only TCP data of a packet is forwarded to the user logic. Latency time of Rx data interface is equal to 15 clock cycles or 46.5 ns @ 322.265625 MHz as minimum value. Latency time is increased if EMAC pauses data transmission by de-asserting MacRxValid to ‘0’. Figure 18 shows timing diagram of Rx data interface when receiving the packet from EMAC.

Figure 18: Rx data interface

(1) EMAC sends the first data of a packet (TCP0) to TOE10GLL by asserting MacRxValid to ‘1’ with the valid data on MacRxData.

(2) TOE10GLL IP verifies the network parameters in the header which must be matched to the set value from the user. If the packet is valid, TCP data is extracted and forwarded to the user with 15-clock cycle latency time as minimum value.

(3) When EMAC pauses data transmission by de-asserting MacRxValid to ‘0’, TOE10GLL IP also asserts TCPRxValid to ‘0’ to pause data transmission to the user logic after two clock cycles.

(4) EMAC sends the last data of a packet by asserting MacRxValid and MacRxEOP to ‘1’.

(5) After completing to verify the header and data in the packet, TOE10GLL IP sends the last data to user by asserting TCPRxValid and TCPRxEOP to ‘1’. At the same clock, Error status is valid on TCPRxError and TCPRxByteEn shows the last byte valid which may be not equal to all 1111b. If some errors are detected (TCPRxError is not equal to 00h), the user logic must ignore this packet and then waits the next packet which is re-transferred from data recovery process.

(6) According to TCP/IP standard, the sender must check the free space size of the receiver’s buffer before sending the data. TCPRxBufWrCnt, input from the user, is applied to calculate Window size value returned to the target device for data flow control. Window size is the free spae size which is equal to 65565 – TCPRxBufWrCnt. TCPRxBufWrCnt is not applied during transferring packet to user. After finishing a packet to user, TCPRxBufWrCnt must be valid within two clock cycles, as shown in Figure 18. If the user logic is always ready to receive data, user can set TCPRxBufWrCnt to be 0 as a constant value.

Note: Latency time in Figure 18 is the minimum value when EMAC sends data continuously by asserting MacRxValid to ‘1’. Latency time is increased if EMAC pauses data transmission.

Error

The examples of error situation when running the IP core are shown in this topic. For transmit operation, the error is found when the user sends error inputs to the IP such as wrong packet size or pausing data transmission before end of packet. As a result, IPInt is asserted to ‘1’ and IPState changes to STERR. The user needs to assert RstB to solve the error.

Tx Error from incorrect packet size

Figure 19 shows timing diagram of error when transmit packet size is set to 20 bytes, but the user sends only 16-byte data.

Figure 19: Error when packet size does not match to total transmit size

(1) At the same clock as transmitting the first data of packet, the user sets TCPTxPkLen to be equal to 20 bytes or 5x32-bit data transmission.

(2) User sends 16-byte data or 4x32-bit data (D0-D3) and then asserts TCPTxEOP to ‘1’ to finish the packet transmission.

(3) TOE10GLL IP detects the error condition of the user inputs because data size from the user is less than the set value on TCPTxPkLen. IPState changes to STERR with asserting IPInt and bit[12] of IPStatus to ‘1’. Also, TCPTxReady is de-asserted to ‘0’ to stop data transmission.

(4) The user asserts RstB to ‘0’ to clear all error signals. TOE10GLL IP changes to reset state and it needs to rerun the initialization process.

Tx Error from wrong TCPTxValid

Figure 20 shows timing diagram of error when data transmission is paused by the user before end of packet.

Figure 20: Error when the user pauses transmitting the data before end of packet

(1) During a packet transmission, the user de-asserts TCPTxValid to ‘0’ to pause transmitting data to TOE10GLL IP before the end of packet.

(2) TOE10GLL IP detects the error condition because data transmission is paused. After that, IPState changes to STERR with asserting IPInt and bit[13] of IPStatus to ‘1’. Also, TCPTxReady is de-asserted to ‘0’ to stop data transmission.

(3) The user asserts RstB to ‘0’ to clear all error signals. TOE10GLL IP changes to reset state and it needs to rerun the initialization process.

Rx Error

When the error is found in received packet such as EMAC error or TCP checksum error, TCPRxError (output from TOE10GLL IP) shows error value which is not equal to 0. After the error packet is sent to the user, the user needs to ignore the packet and then waits until the next packet is re-transmitted by TOE10GLL IP. The same data packet is received from TCP recovery process. Figure 21 shows timing diagram of Rx data interface when error status is found.

Figure 21: Packet retransmission in Received interface

(1) If TOE10GLL IP detects the error from EMAC or the error from TCP checksum in the packet, TCPRxError which is sent at the end of packet (TCPRxValid=’1’ and TCPRxEOP=’1’) will be not equal to 0. TCPRxError[0] is asserted to ‘1’ when TCP checksum is error. While TCPRxError[1] is asserted to ‘1’ when EMAC detects CRC error in the packet. The user logic must ignore the error packet and wait for the next packet which is re-transmitted packet.

(2) After that, TOE10GLL IP sends the request to the target device for retransmitting the error packet. After that, TOE10GLL IP resends the same packet, resent by the target device, to user.

Verification Methods

The TOE10GLL IP Core functionality was verified by simulation and also proved on real board design by using Arria10 GX development board.

Recommended Design Experience

User must be familiar with HDL design methodology to integrate this IP into their design.

Ordering Information

This product is available directly from Design Gateway Co., Ltd. Please contact Design Gateway Co., Ltd. For pricing and additional information about this product using the contact information on the front page of this datasheet.

Revision History

Revision	Date	Description
1.0	26-Apr-21	New release
1.1	30-Apr-21	Coorect Figure1