Axtrinet™ APG Ethernet Packet Generators offer affordable 40Gbps & 10Gbps full wire-speed Ethernet load generation, capture and analysis capabilities for R&D, manufacturing, sales and support teams developing and selling products with high speed Ethernet interfaces.

An intuitive Graphical Control Interface or TCL-scripted interface can be used to configure and control the packet generation, capture and analysis capabilities of the unit via a Linux or Windows PC, managing it locally over USB or remotely over Ethernet LAN.

This application note describes an example of how the packet capture and analysis capabilities were used to diagnose network equipment performance issues.

Network Performance Issues

A customer observed that packet retransmissions were causing poor performance on their test network. The problem was traced to a network element occasionally dropping packets.

Further investigations into the network element behaviour determined that it would perform correctly with Layer 2 and Layer 3 Ethernet traffic, but started dropping occasional Layer 4 packets.

Axtrinet™ APG Packet & Timing Capabilities

Axtrinet™ APG Ethernet Packet Generators offer a dynamically allocated 1GByte ‘deep’ packet capture buffer, and a fixed 64KByte capture buffer per port.

Four user-definable filters* per port allow the amount of captured data to be reduced so that only the critical data is captured, for example to filter on a defined MAC Address, VLAN ID or IPV4 Source Address.

Each transmitted packet can be optionally marked with a 32-bit port signature; a 32-bit sequence number and a 64-bit transmit timestamp, to an accuracy of ±8ns.

All packets captured by the Axtrinet™ APG Ethernet Packet Generator are timestamped with the arrival time at the port.

The captured data can be downloaded to the Control Interface GUI and TCL scripting environment for further analysis, or stored externally as a PCAP file for offline analysis in a third-party tool, such as Wireshark™.

The Control Interface displays:

• the captured packet data
• packet lengths
• inter-packet gap (IPG), derived from the receive timestamps, and
• latency, derived from the transmit and receive timestamps for each packet.

Diagnosis

Using breakout taps either side of the failing network element enabled full access to the data before and after the device in both directions. A breakout tap connects Ports A & B, whilst replicating received traffic on Port A to Port C, and Port B to Port D. Data flowing A to B through the faulty element was monitored on tap ports C, and data flowing B to A was monitored on tap ports D.

Connecting the tap ports to an Axtrinet™ APG208 (2x 40Gbps QSFP+ ports and 8x 10Gbps SFP+ ports) enabled packet capture and analysis of all of the data flowing through the faulty module in both directions.

Deep packet capture is enabled on APG208 Ports 1-4, allowing simultaneous capture of all data received on each port into the shared 1Gb capture buffer. Capture stops automatically when the buffer is full. The capture buffer in the APG208 is then downloaded to the Control Interface (or TCL) for viewing.

Analysis & Resolution

Visual comparison of large quantities of captured data is difficult using the Control Interface packet viewing tool, so the data was exported as a PCAP file and manipulated externally.

The data captured and saved for each port was analysed for differences. There was no identifiable pattern in the dropped packets from different MAC addresses, different IP flows, and of different lengths.  By making use of the time-stamping capability, analysis of the inter-packet gaps (IPG) suggested the cause of the problem.

Immediately before the dropped packets there was a burst of packets separated by short IPGs.  The port was running at full line rate for a short period, a significantly higher rate than the normal network load.  Immediately before the line-rate burst was a jumbo frame.

This behaviour is indicative of egress port buffering problems within the network element.  This knowledge allowed the customer to work with the manufacturer of the network element to identify and resolve a buffer configuration issue.

Once resolved, an updated network element was retested and accepted by the customer.