Linux Fu: A Little Bit of (Network) History Repeating Itself

These days, embedded systems often have networks and that can make them significantly more complex. Networks are usually pretty nondeterministic and there are a variety of oddball conditions. For example, when your public-access pick and place machine gets written up on Hackaday and you suddenly get a 50X surge in traffic, how does your network stack handle it? While there’s no silver bullet for network testing, there are some tricks that can make it easier and one of those is the tcpreplay utilities that allow you to record complex network traffic and then play it back in a variety of ways. This has many benefits, especially if you manage to capture that one thing that triggers bad behavior sporadically. Being able to play it back on demand can speed up diagnostics considerably.

General Idea

You probably know that tcpdump allows you to grab packet captures from a network interface and save them to a file. If you prefer a GUI, you probably use Wireshark, which uses the same underlying library (libpcap) to grab the data. In fact, you can capture data using tcpdump and look at it with Wireshark, although there are other tools like tcptrace or Ngrep that can work with the output, also.

While the output of the command can be a little cryptic without tool support, a program called tcpreplay can take that data and feed it back in a variety of ways. Of course, you can modify the file first — there are tools to make that easier and — if you need to — you can craft your own network traffic by hand or using one of a variety of tools. This process is often called “packet crafting.”

Getting the Data

Sometimes using tcpdump is an example of how you can have too much of a good thing. You’ll get huge data files if you just grab everything off a particular network device:

tcpdump -i eth0

Usually, you’ll want to limit the data in different ways. For example:

tcpdump src 192.168.1.111   # data from .111
tcpdump src 192.168.1.111   # data to .111
tcpdump host 192.168.1.111  # data to/from .111

Other common filters include “net” for a particular subnet or “port” or even a protocol like “arp” or “ip6.” You can combine these so:

tcpdump port 8088      # traffic on port 8088
tcpdump dst port 8080  # traffic to port 8088

Ports can also have ranges (80-89) and there are a host of other filters including packet size limits. You can join conditions with both “or” and “and.” There’s a lot to learn, and you can read more on the tcpdump man page.

The exact output of tcpdump depends on the protocol in use. You can add the -X or -XX options to get a hex dump, as well.

So now you have a file with some interesting network data in it. How do you repeat it for debugging purposes?

The Big Playback

That’s where tcpreplay comes into play. Simplistically, it is easy to use:

tcpreplay -i eth0 traffic.pcap

However, you may need more control. For example, by default, the replay is at the same speed it originally occurred. However, you can add the “–mbps” option to force a certain flow rate. You can even use “–mbps=0” to have no delay at all between packets.

There are other performance options, too. The “-K” option reads the entire capture file into memory to improve performance, if possible. If you are using the “–loop” option to repeat the playback, this can have a big performance advantage.

In some extreme cases, you may want to really saturate the network interface for testing. There are special drivers that tcpreplay can use that allow it to directly write to network hardware, although doing so will disable normal network operations while you are playing back.

Editing

Sometimes you want to make subtle changes to the traffic in the capture file without actually changing the file. The basic program has a few options to help. For example, the “–unique-ip” option will cause the program to alter packets to be unique each time through a loop.

However, sometimes you need more control. The tcpreplay-edit program can do lots of changes. For example, it can remap TCP or UDP ports. It can also randomize IP addresses, remove broadcasts, and make other changes on the fly.

As a practical example, you might record a session between a client and a server machine. To reproduce the server’s behavior, you would want to strip the server’s responses from the file and rewrite the MAC addresses so the client is now the machine running tcpreplay.

These tools are very powerful and — like many powerful tools — can be used for good or ill purposes. There are so many options, but you’ll find if you just start out trying a few “toy” cases and spend a few minutes with each man page that you’ll be able to find ways to monitor and replay your network traffic to help solve that next intractable network problem.