IPng Networks - Loadtesting at Coloclue (2024)

Author: Pim van Pelt <pim@ipng.nl>
Reviewers: Coloclue Network Committee <routers@coloclue.net>
Status: Draft - Review - Published

Introduction

Coloclue AS8283 operates several Linux routers running Bird. Over the years, the performance of their previous hardware platform (Dell R610) has deteriorated, and they were up for renewal. At the same time, network latency/jitter has been very high, and variability may be caused by the Linux router hardware, their used software, the intra-datacenter links, or any combination of these. One specific example of why this is important is that Coloclue runs BFD on their inter-datacenter links, which are VLANs provided to us by Atom86 and Fusix networks. On these links Colclue regularly sees ping times of 300-400ms, with huge outliers in the 1000ms range, which triggers BFD timeouts causing iBGP reconvergence events and overall horrible performance. Before we open up a discussion with these (excellent!) L2 providers, we should first establish if it’s not more likely that Coloclue’s router hardware and/or software should be improved instead.

By means of example, let’s take a look at a Smokeping graph that shows these latency spikes, jitter and loss quite well. It’s taken from a machine at True (in EUNetworks) to a machine at Coloclue (in NorthC); this is the first graph. The same machine at True to a machine at BIT (in Ede) does not exhibit this behavior; this is the second graph.

Images: Smokeping graph from True to Coloclue (left), and True to BIT (right). There is quite a difference.

Summary

I performed three separate loadtests. First, I did a loopback loadtest on the T-Rex machine, proving that it can send 1.488Mpps in both directions simultaneously. Then, I did a loadtest of the Atom86 link by sending the traffic through the Arista in NorthC, over the Atom86 link, to the Arista in EUNetworks, looping two ethernet ports, and sending the traffic back to NorthC. Due to VLAN tagging, this yielded 1.42Mpps throughput, exactly as predicted. Finally, I performed a stateful loadtest that saturated the Atom86 link, while injecting SCTP packets at 1KHz, measuring the latency observed over the Atom86 link.

All three tests passed.

Loadtest Setup

After deploying the new NorthC routers (Supermicro Super Server/X11SCW-F with Intel Xeon E-2286G processors), I decided to rule out hardware issues, leaving link and software issues. To get a bit more insight on software or inter-datacenter links, I created the following two loadtest setups.

1. Baseline

Machine dcg-2, carrying an Intel 82576 quad Gigabit NIC, looped from the first two ports (port0 to port1). The point of this loopback test is to ensure that the machine itself is capable of sending and receiving the correct traffic patterns. Usually, one does an “imix” and a “64b” loadtest for this, and it is expected that the loadtester itself passes all traffic out on one port back into the other port, without any loss. The thing I am testing is called the DUT or Device Under Test and in this case, it is a UTP cable from NIC-NIC.

The expected packet rate is: 672 bits for the ethernet frame is 10^9 / 672 == 1488095 packets per second in each direction and traversing the link once. You will often see 1.488Mpps as “the theoretical maximum”, and this is why.

2. Atom86

In this test, Tijn from Coloclue plugged dcg-2 port0 into the core switch (an Arista) port e17, and he configured that switchport as an access port for VLAN A; which is put on the Atom86 trunk to EUNetworks. The second port1 is plugged into the core switch port e18, and assigned a different VLAN B, which is also put on the Atom86 link to EUNetworks.

At EUNetworks then, he exposed that same VLAN A on port e17 and VLAN B on port e18. And Tijn used DAC cable to connect e17 <-> e18. Thus, the path the packets travel now becomes the Device Under Test (DUT):

Detailed Analysis

This section goes into details, but it is roughly broken down into:

Prepare machine (install T-Rex, needed kernel headers, and some packages)
Configure T-Rex (bind NIC from PCI bus into DPDK)
Run T-Rex interactively
Run T-Rex programmatically

Step 1 - Prepare machine

Download T-Rex from Cisco website and unpack (I used version 2.88) in some directory that is readable by ‘nobody’. I used /tmp/loadtest/ for this. Install some additional tools:

sudo apt install linux-headers-`uname -r` build-essential python3-distutils

Step 2 - Bind NICs to DPDK

First I had to find which NICs that can be used, these NICs have to be supported in DPDK, but luckily most Intel NICs are. I had a few ethernet NICs to choose from:

root@dcg-2:/tmp/loadtest/v2.88# lspci | grep -i Ether01:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)01:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)01:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)01:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)0c:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)0d:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

But the ones that have no link are a good starting point:

root@dcg-2:~/loadtest/v2.88# ip link | grep -v UP | grep enp76: enp7s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 10007: enp7s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

This is PCI bus 7, slot 0, function 0 and 1, so the configuration file for T-Rex becomes:

root@dcg-2:/tmp/loadtest/v2.88# cat /etc/trex_cfg.yaml- version : 2 interfaces : ["07:00.0","07:00.1"] port_limit : 2 port_info : - dest_mac : [0x0,0x0,0x0,0x1,0x0,0x00] # port 0 src_mac : [0x0,0x0,0x0,0x2,0x0,0x00] - dest_mac : [0x0,0x0,0x0,0x2,0x0,0x00] # port 1 src_mac : [0x0,0x0,0x0,0x1,0x0,0x00]

Step 3 - Run T-Rex Interactively

Start the loadtester, this is easiest if you use two terminals, one to run t-rex itself and one to run the console:

root@dcg-2:/tmp/loadtest/v2.88# ./t-rex-64 -iroot@dcg-2:/tmp/loadtest/v2.88# ./trex-console

The loadtester starts with -i (interactive) and optionally -c (number of cores to use, in this case only 1 CPU core is used). I will be doing a loadtest with gigabit speeds only, so no significant CPU is needed. I will demonstrate below that one CPU core of this machine can generate (sink and source) approximately 72Gbit/s of traffic. The loadtest starts a controlport on :4501 which the client connects to. You can now program the loadtester (programmatically via an API, or via the commandline / CLI tool provided. I’ll demonstrate both).

In trex-console, I first enter ‘TUI’ mode – this stands for the Traffic UI. Here, I can load a profile into the loadtester, and while you can write your own profiles, there are many standard ones to choose from. There’s further two types of loadtest, stateful and stateless. I started with a simpler ‘stateless’ one first, take a look at stl/imix.py which is self explanatory, but in particular, the mix consists of:

 self.ip_range = {'src': {'start': "16.0.0.1", 'end': "16.0.0.254"}, 'dst': {'start': "48.0.0.1", 'end': "48.0.0.254"}} # default IMIX properties self.imix_table = [ {'size': 60, 'pps': 28, 'isg':0 }, {'size': 590, 'pps': 16, 'isg':0.1 }, {'size': 1514, 'pps': 4, 'isg':0.2 } ]

Above one can see that there will be traffic flowing from 16.0.0.1-254 to 48.0.0.1-254, and there will be three streams generated at a certain ratio, 28 small 60 byte packets, 16 medium sized 590b packets, and 4 large 1514b packets. This is typically what a residential user would see (a SIP telephone call, perhaps a Jitsi video stream; some download of data with large MTU-filling packets; and some DNS requests and other smaller stuff). Executing this profile can be done with:

tui> start -f stl/imix.py -m 1kpps

.. which will start a 1kpps load of that packet stream. The traffic load can be changed by either specifying an absolute packet rate, or a percentage of line rate, and you can pause and resume as well:

tui> update -m 10kppstui> update -m 10%tui> update -m 50%tui> pause# do something, there will be no traffictui> resumetui> update -m 100%

After this last command, T-Rex will be emitting line rate packets out of port0 and out of port1, and it will be expecting to see the packets that it sent back on port1 and port0 respectively. If the machine is powerful enough, it can saturate traffic up to the line rate in both directions. One can see if things are successfully passing through the device under test (in this case, for now simply a UTP cable from port0-port1). The ‘ibytes’ should match the ‘obytes’, and of course ‘ipackets’ should match the ‘opackets’ in both directions. Typically, a loss rate of 0.01% is considered acceptable. And, typically, a loss rate of a few packets in the beginning of the loadtest is also acceptable (more on that later).

Screenshot of port0-port1 loopback test with L2:

Global Statistitcsconnection : localhost, Port 4501 total_tx_L2 : 1.51 Gbpsversion : STL @ v2.88 total_tx_L1 : 1.98 Gbpscpu_util. : 5.63% @ 1 cores (1 per dual port) total_rx : 1.51 Gbpsrx_cpu_util. : 0.0% / 0 pps total_pps : 2.95 Mppsasync_util. : 0% / 104.03 bps drop_rate : 0 bpstotal_cps. : 0 cps queue_full : 0 pktsPort Statistics port | 0 | 1 | total-----------+-------------------+-------------------+------------------owner | root | root |link | UP | UP |state | TRANSMITTING | TRANSMITTING |speed | 1 Gb/s | 1 Gb/s |CPU util. | 5.63% | 5.63% |-- | | |Tx bps L2 | 755.21 Mbps | 755.21 Mbps | 1.51 GbpsTx bps L1 | 991.21 Mbps | 991.21 Mbps | 1.98 GbpsTx pps | 1.48 Mpps | 1.48 Mpps | 2.95 MppsLine Util. | 99.12 % | 99.12 % |--- | | |Rx bps | 755.21 Mbps | 755.21 Mbps | 1.51 GbpsRx pps | 1.48 Mpps | 1.48 Mpps | 2.95 Mpps---- | | |opackets | 355108111 | 355111209 | 710219320ipackets | 355111078 | 355108226 | 710219304obytes | 22761267356 | 22761466414 | 45522733770ibytes | 22761457966 | 22761274908 | 45522732874tx-pkts | 355.11 Mpkts | 355.11 Mpkts | 710.22 Mpktsrx-pkts | 355.11 Mpkts | 355.11 Mpkts | 710.22 Mpktstx-bytes | 22.76 GB | 22.76 GB | 45.52 GBrx-bytes | 22.76 GB | 22.76 GB | 45.52 GB----- | | |oerrors | 0 | 0 | 0ierrors | 0 | 0 | 0

Instead of stl/imix.py as a profile, one can also consider stl/udp_1pkt_simple_bdir.py as a profile. This will send UDP packets of 0 bytes payload from a single host 16.0.0.1 to a single host 48.0.0.1 and back. Running the 1pkt UDP profile in both directions at gigabit link speeds will allow for 1.488Mpps in both directions (a minimum ethernet frame carrying IPv4 packet will be 672 bits in length – see wikipedia for details).

Above, one can see the system is in a healthy state - it has saturated the network bandwidth in both directions (991Mps L1 rate, so this is the full 672 bits per ethernet frame, including the header, interpacket gap, etc), at 1.48Mpps. All packets sent by port0 (the opackets, obytes) should have been received by port1 (the ipackets, ibytes), and they are.

One can also learn that T-Rex is utilizing approximately 5.6% of one CPU core sourcing and sinking this load on the two gigabit ports (that’s 2 gigabit out, 2 gigabit in), so for a DPDK application, one CPU core is capable of 71Gbps and 53Mpps, an interesting observation.

Step 4 - Run T-Rex programmatically

I wrote a tool previously that allows to run a specific ramp-up profile from 1kpps warmup through to line rate, in order to find the maximum allowable throughput before a DUT exhibits too much loss, usage:

usage: trex-loadtest.py [-h] [-s SERVER] [-p PROFILE_FILE] [-o OUTPUT_FILE] [-wm WARMUP_MULT] [-wd WARMUP_DURATION] [-rt RAMPUP_TARGET] [-rd RAMPUP_DURATION] [-hd HOLD_DURATION]T-Rex Stateless Loadtester -- pim@ipng.nloptional arguments: -h, --help show this help message and exit -s SERVER, --server SERVER Remote trex address (default: 127.0.0.1) -p PROFILE_FILE, --profile PROFILE_FILE STL profile file to replay (default: imix.py) -o OUTPUT_FILE, --output OUTPUT_FILE File to write results into, use "-" for stdout (default: -) -wm WARMUP_MULT, --warmup_mult WARMUP_MULT During warmup, send this "mult" (default: 1kpps) -wd WARMUP_DURATION, --warmup_duration WARMUP_DURATION Duration of warmup, in seconds (default: 30) -rt RAMPUP_TARGET, --rampup_target RAMPUP_TARGET Target percentage of line rate to ramp up to (default: 100) -rd RAMPUP_DURATION, --rampup_duration RAMPUP_DURATION Time to take to ramp up to target percentage of line rate, in seconds (default: 600) -hd HOLD_DURATION, --hold_duration HOLD_DURATION Time to hold the loadtest at target percentage, in seconds (default: 30)

Here, the loadtester will load a profile (imix.py for example), warmup for 30s at 1kpps, then ramp up linearly to 100% of line rate in 600s, and hold at line rate for 30s. The loadtest passes if during this entire time, the DUT had less than 0.01% packet loss. I must note that in this loadtest, I cannot ramp up to line rate (because the Atom86 link is used twice!), and I’ll also note I cannot ramp up to 50% of line rate (because the loadtester is sending untagged traffic, but the Arista is adding tags onto the Atom86 link!), so I expect to see 711Kpps which is just about 47% of line rate.

The loadtester will emit a JSON file with all of its runtime stats, which can be later analyzed and used to plot graphs. First, let’s look at an imix loadtest:

root@dcg-2:/tmp/loadtest# trex-loadtest.py -o ~/imix.json -p imix.py -rt 50Running against 127.0.0.1 profile imix.py, warmup 1kpps for 30s, rampup target 50%of linerate in 600s, hold for 30s output goes to /root/imix.jsonMapped ports to sides [0] <--> [1]Warming up [0] <--> [1] at rate of 1kpps for 30 secondsSetting load [0] <--> [1] to 1% of lineratestats: 4.20 Kpps 2.82 Mbps (0.28% of linerate)…stats: 321.30 Kpps 988.14 Mbps (98.81% of linerate)Loadtest finished, stoppingTest has passed :-)Writing output to /root/imix.json

And then let’s step up our game with a 64b loadtest:

root@dcg-2:/tmp/loadtest# trex-loadtest.py -o ~/64b.json -p udp_1pkt_simple_bdir.py -rt 50Running against 127.0.0.1 profile udp_1pkt_simple_bdir.py, warmup 1kpps for 30s, rampup target 50%of linerate in 600s, hold for 30s output goes to /root/64b.jsonMapped ports to sides [0] <--> [1]Warming up [0] <--> [1] at rate of 1kpps for 30 secondsSetting load [0] <--> [1] to 1% of lineratestats: 4.20 Kpps 2.82 Mbps (0.28% of linerate)…stats: 1.42 Mpps 956.41 Mbps (95.64% of linerate)stats: 1.42 Mpps 952.44 Mbps (95.24% of linerate)WARNING: DUT packetloss too highstats: 1.42 Mpps 955.19 Mbps (95.52% of linerate)

As an interesting note, this value 1.42Mpps is exactly what I calculated and expected (see above for a full explanation). The math works out at 10^9 / 704 bits/packet == 1.42Mpps, just short of 1.488M line rate that I would have found had Coloclue not used VLAN tags.

Step 5 - Run T-Rex ASTF, measure latency/jitter

In this mode, T-Rex simulates many stateful flows using a profile, which replays actual PCAP data (as can be obtained with tcpdump), by spacing out the requests and rewriting the source/destination addresses, thereby simulating hundreds or even millions of active sessions - I used astf/http_simple.py as a canonical example. In parallel to the test, I let T-Rex run a latency check, by sending SCTP packets at a rate of 1KHz from each interface. By doing this, latency profile and jitter can be accurately measured under partial or full line load.

Bandwidth/Packet rate

Let’s first take a look at the bandwidth and packet rates:

Global Statistitcsconnection : localhost, Port 4501 total_tx_L2 : 955.96 Mbpsversion : ASTF @ v2.88 total_tx_L1 : 972.91 Mbpscpu_util. : 5.64% @ 1 cores (1 per dual port) total_rx : 955.93 Mbpsrx_cpu_util. : 0.06% / 2 Kpps total_pps : 105.92 Kppsasync_util. : 0% / 63.14 bps drop_rate : 0 bpstotal_cps. : 3.46 Kcps queue_full : 143,837 pktsPort Statistics port | 0 | 1 | total-----------+-------------------+-------------------+------------------owner | root | root |link | UP | UP |state | TRANSMITTING | TRANSMITTING |speed | 1 Gb/s | 1 Gb/s |CPU util. | 5.64% | 5.64% |-- | | |Tx bps L2 | 17.35 Mbps | 938.61 Mbps | 955.96 MbpsTx bps L1 | 20.28 Mbps | 952.62 Mbps | 972.91 MbpsTx pps | 18.32 Kpps | 87.6 Kpps | 105.92 KppsLine Util. | 2.03 % | 95.26 % |--- | | |Rx bps | 938.58 Mbps | 17.35 Mbps | 955.93 MbpsRx pps | 87.59 Kpps | 18.32 Kpps | 105.91 Kpps---- | | |opackets | 8276689 | 39485094 | 47761783ipackets | 39484516 | 8275603 | 47760119obytes | 978676133 | 52863444478 | 53842120611ibytes | 52862853894 | 978555807 | 53841409701tx-pkts | 8.28 Mpkts | 39.49 Mpkts | 47.76 Mpktsrx-pkts | 39.48 Mpkts | 8.28 Mpkts | 47.76 Mpktstx-bytes | 978.68 MB | 52.86 GB | 53.84 GBrx-bytes | 52.86 GB | 978.56 MB | 53.84 GB----- | | |oerrors | 0 | 0 | 0ierrors | 0 | 0 | 0

In the above screen capture, one can see the traffic out of port0 is 20Mbps at 18.3Kpps, while the traffic out of port1 is 952Mbps at 87.6Kpps - this is because the clients are sourcing from port0, while the servers are simulated behind port1. Note the asymmetric traffic flow, T-Rex is using 972Mbps of total bandwidth over this 1Gbps VLAN, and a tiny bit more than that on the Atom86 link, because the Aristas are inserting VLAN tags in transit, to be exact, 18.32+87.6 = 105.92Kpps worth of 4 byte tags, thus 3.389Mbit extra traffic.

Latency Injection

Now, let’s look at the latency in both directions, depicted in microseconds, at a throughput of 106Kpps (975Mbps):

Global Statistitcsconnection : localhost, Port 4501 total_tx_L2 : 958.07 Mbpsversion : ASTF @ v2.88 total_tx_L1 : 975.05 Mbpscpu_util. : 4.86% @ 1 cores (1 per dual port) total_rx : 958.06 Mbpsrx_cpu_util. : 0.05% / 2 Kpps total_pps : 106.15 Kppsasync_util. : 0% / 63.14 bps drop_rate : 0 bpstotal_cps. : 3.47 Kcps queue_full : 143,837 pktsLatency Statistics Port ID: | 0 | 1-------------+-----------------+----------------TX pkts | 244068 | 242961RX pkts | 242954 | 244063Max latency | 23983 | 23872Avg latency | 815 | 702-- Window -- | |Last max | 966 | 867Last-1 | 948 | 923Last-2 | 945 | 856Last-3 | 974 | 880Last-4 | 963 | 851Last-5 | 985 | 862Last-6 | 986 | 870Last-7 | 946 | 869Last-8 | 976 | 879Last-9 | 964 | 867Last-10 | 964 | 837Last-11 | 970 | 867Last-12 | 1019 | 897Last-13 | 1009 | 908Last-14 | 1006 | 897Last-15 | 1022 | 903Last-16 | 1015 | 890--- | |Jitter | 42 | 45---- | |Errors | 3 | 2

In the capture above, one can see the total latency measurement packets sent, and the latency measurements in microseconds. One can see that from port0->port1 the measured latency was 0.815ms, while the latency in the other direction was 0.702ms. The discrepancy can be explained by the HTTP traffic being asymmetric (clients on port0 have to send their SCTP packets into a much busier port1), which creates queuing latency on the wire and NIC. The Last-* lines under it are the values of the last 16 seconds of measurements. The maximum observed latency was 23.9ms in one direction and 23.8ms in the other direction. I have to conclude therefore that the Atom86 line, even under stringent load, does not suffer from outliers in the entire 300s duration of my loadtest.

Jitter is defined as a variation in the delay of received packets. At the sending side, packets are sent in a continuous stream with the packets spaced evenly apart. Due to network congestion, improper queuing, or configuration errors, this steady stream can become lumpy, or the delay between each packet can vary instead of remaining constant. There was virtually no jitter: 42 microseconds in one direction, 45us in the other.

Latency Distribution

While performing this test at 106Kpps (975Mbps), it’s also useful to look at the latency distribution as a histogram:

Global Statistitcsconnection : localhost, Port 4501 total_tx_L2 : 958.07 Mbpsversion : ASTF @ v2.88 total_tx_L1 : 975.05 Mbpscpu_util. : 4.86% @ 1 cores (1 per dual port) total_rx : 958.06 Mbpsrx_cpu_util. : 0.05% / 2 Kpps total_pps : 106.15 Kppsasync_util. : 0% / 63.14 bps drop_rate : 0 bpstotal_cps. : 3.47 Kcps queue_full : 143,837 pktsLatency Histogram Port ID: | 0 | 1-------------+-----------------+----------------20000 | 2545 | 249510000 | 5889 | 61009000 | 456 | 4218000 | 874 | 8547000 | 692 | 7576000 | 619 | 6375000 | 985 | 9944000 | 579 | 6203000 | 547 | 5462000 | 381 | 4051000 | 798 | 697900 | 27451 | 346800 | 163717 | 22924700 | 102194 | 154021600 | 24623 | 171087500 | 24882 | 40586400 | 18329 |300 | 26820 |

In the capture above, one can see the number of packets observed between certain ranges; from port0 to port1, 102K SCTP latency probe packets were in transit some time between 700-799us, 163K probes were between 800-899us. In the other direction, 171K probes were between 600-699us and 154K probes were between 700-799us. This is corroborated by the mean latency I saw above (815us from port0->port1 and 702us from port1->port0).