Tcpreplay

home

How To...

Overview

This section is designed to get you up and running quickly and easily by supplying step-by-step examples of various scenarios.

The following how-to examples use one of provided sample captures. Two identical machines are used, each with i7 processors and multi-port Intel 82599 10GigE adapters. The 10GigE adapters are connected back-to-back with patch cables. LAN connectivity with SSH is via one of the GigE interfaces. GigE interfaces are numbered eth0 through eth5. The 10GigE interfaces are eth6 and eth7.

Both systems are running Ubuntu 10.04 with a Linux version 3.1.10 kernel.

There are several screencast videos available that are related this section. The entire play list of screencasts are available here, or you can watch all screencasts below. If you are more interested in detailed descriptions, skip the screencasts and proceed to the next section.

How to do a Performance Test for an IP Flow Appliance

Use this example if you need to test the performance of a device that captures and reports IP Flow/NetFlow statistics. In this example we are sending to a device under test (DUT) running nprobe. We will use the nprobe logging features to measure how many packets, bytes and flows per second are captured.

The ability to generate wire-rate traffic and extremely high flows-per-second is new in Tcpreplay v4.0.0. The physical setup is very simple. In our experience Tcpreplay will greatly outperform the receiving station. Therefore it is important to track the amount of traffic sent and received, and determine the amount of data lost.

# nprobe -T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %INPUT_SNMP \
%OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT \
%L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %SRC_AS %DST_AS %IPV4_SRC_MASK \
%IPV4_DST_MASK" -i eth7
03/Jan/2014 20:50:24 [nprobe.c:2822] Welcome to nprobe v.6.7.3 for x86_64
03/Jan/2014 20:50:24 [plugin.c:143] No plugins found in ./plugins
03/Jan/2014 20:50:24 [plugin.c:143] No plugins found in /usr/local/lib/nprobe/plugins
03/Jan/2014 20:50:24 [nprobe.c:4004] Welcome to nprobe v.6.7.3 for x86_64
03/Jan/2014 20:50:24 [plugin.c:665] 0 plugin(s) enabled
03/Jan/2014 20:50:24 [nprobe.c:3145] Using packet capture length 128
03/Jan/2014 20:50:24 [nprobe.c:4222] Flows ASs will not be computed 
03/Jan/2014 20:50:24 [nprobe.c:4298] Capturing packets from interface eth7
03/Jan/2014 20:50:24 [util.c:3262] nProbe changed user to 'nobody'
tcpreplay -i eth7 -tK --loop 50 --unique-ip bigFlows.pcap  
File Cache is enabled
Actual: 39580750 packets (17770889200 bytes) sent in 20.02 seconds.
Rated: 876866137.5 Bps, 7014.92 Mbps, 1953026.60 pps
Flows: 2034300 flows, 100378.13 fps, 39558950 flow packets, 21800 non-flow
Statistics for network device: eth7
	Attempted packets:         39580750
	Successful packets:        39580750
	Failed packets:            0
	Truncated packets:         0
	Retried packets (ENOBUFS): 0
	Retried packets (EAGAIN):  0

03/Jan/2014 20:52:30 [nprobe.c:1557] ---------------------------------
03/Jan/2014 20:52:30 [nprobe.c:1558] Average traffic: [1.870 M pps][6 Gb/sec]
03/Jan/2014 20:52:30 [nprobe.c:1563] Current traffic: [569.089 K pps][2 Gb/sec]
03/Jan/2014 20:52:30 [nprobe.c:1568] Current flow export rate: [0.0 flows/sec]
03/Jan/2014 20:52:30 [nprobe.c:1571] Flow drops: [export queue too long=0]
[too many flows=0]
03/Jan/2014 20:52:30 [nprobe.c:1575] Export Queue: 0/512000 [0.0 %]
03/Jan/2014 20:52:30 [nprobe.c:1582] Flow Buckets: [active=749448][allocated=749448]
[toBeExported=0][frags=0]
03/Jan/2014 20:52:30 [nprobe.c:1465] Processed packets: 13089188 (max bucket search: 162)
03/Jan/2014 20:52:30 [nprobe.c:1468] Flow export stats: [0 bytes][0 pkts][0 flows]
03/Jan/2014 20:52:30 [nprobe.c:1477] Flow drop stats:   [0 bytes][0 pkts][0 flows]
03/Jan/2014 20:52:30 [nprobe.c:1482] Total flow stats:  [0 bytes][0 pkts][0 flows]
03/Jan/2014 20:52:30 [nprobe.c:235] Packet stats (pcap): 13100072/234355 pkts 
rcvd/dropped [1.8%] [Last 13334427/234355 pkts rcvd/dropped]
03/Jan/2014 20:53:00 [nprobe.c:1557] ---------------------------------
03/Jan/2014 20:53:00 [nprobe.c:1558] Average traffic: [626.567 K pps][2 Gb/sec]
03/Jan/2014 20:53:00 [nprobe.c:1563] Current traffic: [336.465 K pps][1 Gb/sec]
03/Jan/2014 20:53:00 [nprobe.c:1568] Current flow export rate: [0.0 flows/sec]
03/Jan/2014 20:53:00 [nprobe.c:1571] Flow drops: [export queue too long=0]
[too many flows=0]
03/Jan/2014 20:53:00 [nprobe.c:1575] Export Queue: 0/512000 [0.0 %]
03/Jan/2014 20:53:00 [nprobe.c:1582] Flow Buckets: [active=1542197][allocated=1542197]
[toBeExported=0][frags=0]
03/Jan/2014 20:53:00 [nprobe.c:1465] Processed packets: 23183007 (max bucket search: 336)
03/Jan/2014 20:53:00 [nprobe.c:1468] Flow export stats: [0 bytes][0 pkts][0 flows]
03/Jan/2014 20:53:00 [nprobe.c:1477] Flow drop stats:   [0 bytes][0 pkts][0 flows]
03/Jan/2014 20:53:00 [nprobe.c:1482] Total flow stats:  [0 bytes][0 pkts][0 flows]
03/Jan/2014 20:53:00 [nprobe.c:235] Packet stats (pcap): 23183007/11039913 pkts 
rcvd/dropped [32.3%] [Last 20888493/10805558 pkts rcvd/dropped]

Reducing Flows Per Second

To reduce the flow rate to 50Kfps you adjust the --mbps option …

tcpreplay -i eth7 -K --mbps 3500 --loop 50 --unique-ip bigFlows.pcap 
File Cache is enabled
Actual: 39580750 packets (17770889200 bytes) sent in 40.06 seconds.
Rated: 437499777.1 Bps, 3499.99 Mbps, 974434.59 pps
Flows: 2034300 flows, 50082.23 fps, 39558950 flow packets, 21800 non-flow
Statistics for network device: eth7
	Attempted packets:         39580750
	Successful packets:        39580750
	Failed packets:            0
	Truncated packets:         0
	Retried packets (ENOBUFS): 0
	Retried packets (EAGAIN):  0

Increasing Flows Per Second with AF_XDP

As of version 4.5.1, you can achieve line speed transmission on newer Linux kernels by using the --xdp options. No kernel modifications are required.

Increasing Flows Per Second with netmap

We have already shown that the maximum rate that we can send is just over 7 Gbps. To increase flow rate to maximum possible with selected capture file you must install netmap, which will allow Tcpreplay to write directly to the network adapters TX buffers.

Note: Using netmap is invasive and is not for the faint of heart. You should be somewhat familiar with compiling kernel modules before proceeding. Also, be aware that the network stack will be disconnected from the selected network adapter for the duration of the test. If you use SSH, do not connect via the same interface that you are testing on.

./configure

 ...

##########################################################################
             TCPREPLAY Suite Configuration Results (4.0.0)
##########################################################################
libpcap:                    /usr (>= 0.9.6)
libdnet:                    no ()
autogen:                    /usr/local/bin/autogen (5.16.2)
Use libopts tearoff:        yes
64bit counter support:      yes
tcpdump binary path:        /usr/sbin/tcpdump
fragroute support:          no
tcpbridge support:          yes
tcpliveplay support:        yes

Supported Packet Injection Methods (*):
Linux TX_RING:              no
Linux PF_PACKET:            yes
BSD BPF:                    no
libdnet:                    no
pcap_inject:                yes
pcap_sendpacket:            yes **
Linux/BSD netmap:           yes /usr/src/netmap-release
./configure --with-netmap=/home/fklassen/git/netmap/

 ...

##########################################################################
             TCPREPLAY Suite Configuration Results (4.0.0)
##########################################################################
libpcap:                    /usr (>= 0.9.6)
libdnet:                    no ()
autogen:                    /usr/local/bin/autogen (5.16.2)
Use libopts tearoff:        yes
64bit counter support:      yes
tcpdump binary path:        /usr/sbin/tcpdump
fragroute support:          no
tcpbridge support:          yes
tcpliveplay support:        yes

Supported Packet Injection Methods (*):
Linux TX_RING:              no
Linux PF_PACKET:            yes
BSD BPF:                    no
libdnet:                    no
pcap_inject:                yes
pcap_sendpacket:            yes **
Linux/BSD netmap:           yes /home/fklassen/git/netmap/
#!/bin/sh
ifdown eth6
ifdown eth7
rmmod ixgbe
rmmod netmap_lin
insmod ./netmap_lin.ko
mknod /dev/netmap c 10 59
modprobe mdio
insmod ./ixgbe.ko
ifup eth6
ifup eth7
tcpreplay -i eth7 -tK --loop 50 --unique-ip --netmap bigFlows.pcap 
Switching network driver for eth7 to netmap bypass mode... done!
File Cache is enabled
Actual: 39580750 packets (17770889200 bytes) sent in 15.00 seconds.
Rated: 1182219090.4 Bps, 9457.75 Mbps, 2633133.19 pps
Flows: 2034300 flows, 135333.03 fps, 39558950 flow packets, 21800 non-flow
Statistics for network device: eth7
	Attempted packets:         39580750
	Successful packets:        39580750
	Failed packets:            0
	Truncated packets:         0
	Retried packets (ENOBUFS): 0
	Retried packets (EAGAIN):  0
Switching network driver for eth7 to normal mode... done!
tcpreplay -i eth7 -tK --loop 50 --unique-ip --netmap smallFlows.pcap 
Switching network driver for eth7 to netmap bypass mode... done!
File Cache is enabled
Actual: 713050 packets (460826550 bytes) sent in 0.385312 seconds.
Rated: 1195982865.8 Bps, 9567.86 Mbps, 1850578.23 pps
Flows: 60450 flows, 156885.84 fps, 712150 flow packets, 900 non-flow
Statistics for network device: eth7
	Attempted packets:         713050
	Successful packets:        713050
	Failed packets:            0
	Truncated packets:         0
	Retried packets (ENOBUFS): 0
	Retried packets (EAGAIN):  0
Switching network driver for eth7 to normal mode... done!

How to do a Performance Test for a Packet Capture Device

When testing to a packet capture device or a capture application such as tcpdump, the setup is similar to the IP Traffic Flow Appliance example above. Instead of nprobe run your packet capture application.