Introducing the Cisco Nexus NIC HPT
The Cisco Nexus NIC HPT has a timestamp granulatiry of 250ps (0.25ns). This makes the device capable of taking very high precision measurements. The high precision of the device makes it more sensitive to measurement and methodological errors than previous devices. To understand how to make the best use of the NIC HPT, we have produced this guide.
Cisco Nexus NIC HPT Example Applications
The ExaNIC software package ships with two example NIC HPT utilities which demonstrate its capabilities: 1. exanic-measure.c -- This tool shows how to measure network device latency with high precision. 2. exanic-hpt-fiber-len.c -- This tool shows how to measure fiber/DAC cable lengths to within a few centimetres (inches) accuracy using a NIC HPT.
exanic-measure.c
Exanic-measure is an example of a measurement application written primarily for use with NIC HPT (though it can be used with other devices such as the Nexus SmartNIC devices). To measure latency performance with high precision, we suggest that you undertake the following two step procedure:
-
Measure the average latency for a device of with known constant latency, ideally as close to zero as possible. This will give a calibration estimate of the "offset" including cabling delays and internal NIC delays.
A fiber optical coupler is a good example of known constant latency "network" device. The latency of an optical coupler is effectively zero. -
Replace the known latency with the device that has an unknown latency (using the same cables as in step 1). Apply the -O offset parameter from step 1 to the software and repeat the measurement. Your result will now be high precision, compensated for cabling delays and NIC internal delays.
For example, to perform step 1, on exanic0 with transmit on port 0, and receive on port 1, run the following:
./exanic-measure -d exanic0 -t 0 -r 1 -c 100000
Average: 1.51
Percentile 100.00 = 1.72 ns
Percentile 99.00 = 1.72 ns
Percentile 95.00 = 1.47 ns
Percentile 90.00 = 1.47 ns
Percentile 75.00 = 1.47 ns
Percentile 50.00 = 1.47 ns
Percentile 25.00 = 1.47 ns
Percentile 10.00 = 1.22 ns
Percentile 5.00 = 1.22 ns
Percentile 1.00 = 1.22 ns
Percentile 0.00 = 1.22 ns
The result of step one is that all constant features of the test can be taken into account. In this case, the test harness adds an average of 1.51ns of overhead to the measurement. It also shows the variance that can be expected in any test. In this case, it is all values are no more than +/- 0.25ns from the median.
To perform step 2, remove the coupler device and insert the device under test. In this example, we use a layer 1 patch from a Nexus 3550-F (betwen ports B11/B12 ) You can automatically account for the overhead using the -O (offset) parameter as follows:
./exanic-measure -d exanic0 -t 0 -r 1 -c 100000 -O 1.51
Average: 4.81
Percentile 100.00 = 5.02 ns
Percentile 99.00 = 5.02 ns
Percentile 95.00 = 5.02 ns
Percentile 90.00 = 5.02 ns
Percentile 75.00 = 4.77 ns
Percentile 50.00 = 4.77 ns
Percentile 25.00 = 4.77 ns
Percentile 10.00 = 4.77 ns
Percentile 5.00 = 4.77 ns
Percentile 1.00 = 4.52 ns
Percentile 0.00 = 4.52 ns
The above measurement reports a median value of 4.77ns +/- 0.25ns, which is exactly as expected (for ports B11/B12).
The exanic-measure tool is capable of writing all values out to a csv file which can then be used for further analysis (e.g. generating timeline or distribution plots).
To do so, pass the -w filename option to the application.
exanic-hpt-fiber-len.c
This tool attempts to estimate the length of a loopback cable attached to the NIC HPT. It does this by sending a small packet and measuring the time that the packet takes to return (and taking into account internal delays).
Despite the name, exanic-hpt-fiber-len.c can be used to operate on DAC/Twinax cables as well as fibers. To estimate the length of a cable/fiber, the utility needs to know the type of media (fibre, direct attach AWG24 or direct attach AWG30 cable) and which physical interfaces will be used to use for sending and receiving. The following example uses ports 0 and 1 for TX/RX respectively (on exanic0) to estimate the length of a loopback fibre:
$ ./exanic-hpt-fiber-len -d exanic0 -t 0 -r 1 -m -c 100000
Percentile 99.00 = 1.72 ns
Percentile 95.00 = 1.47 ns
Percentile 90.00 = 1.47 ns
Percentile 75.00 = 1.47 ns
Percentile 50.00 = 1.47 ns
Percentile 25.00 = 1.47 ns
Percentile 10.00 = 1.22 ns
Percentile 5.00 = 1.22 ns
Percentile 1.00 = 1.22 ns
Fiber length estimated to be 0.30m +0.05m/-0.05m
Programming the Cisco Nexus NIC HPT
The NIC HPT supports exactly the same programming API as existing Nexus SmartNIC network cards. Starting from version 1.9.0, libexanichas been modified to include picosecond timestamping capabilities. Following is a description of the relevant parts of this API for NIC HPT operation.
typedef uint64_t exanic_cycles_t;
typedef uint32_t exanic_cycles32_t;
A timestamp value is represented by either a 32 bit exanic_cycles32_t value or a 64 bit exanic_cycles_t value. An exanic_cycles_t value holds the number cycles of the hardware clock since the UNIX epoch. An exanic_cycles32_t value contains the lower 32 bits of the exanic_cycles_t value.
Most of the libexanic API functions that send or receive frames return an exanic_cycles32_t value. The two most common examples are exanic_get_tx_timestamp() and exanic_receive_frame().
ssize_t exanic_receive_frame(exanic_rx_t *rx, char *rx_buf, size_t rx_buf_size,
exanic_cycles32_t *timestamp);
exanic_cycles32_t exanic_get_tx_timestamp(exanic_tx_t *tx)
The exanic_receive_frame() function returns the timestamp corresponding to the moment when a frame was received. Conversely, the exanic_get_tx_timestamp() function returns a timestamp corresponding to the moment when the most recently transmitted frame was transmitted.
As described above, exanic_cycles32_t values represent only the lower 32 bits of a full 64 bit timestamp. To make use of the timestamp returned by exanic_receive_frame() or exanic_get_tx_timestamp(), the exanic_cycles32_t value must be 'expanded' into an exanic_cycles_t (64bit) value.
The exanic_expand_timestamp() function performs this expansion.
There is a time limit on how long you can wait before performing the expansion. The high clock speeds of NIC HPT devices means that 32 bit values must be expanded quickly. A NIC HPT has an timestamping clock speed of 4GHz (0.25ns), which means that the value can overflow in a little over 1 second. If you do not expand a 32bit timestamp within about 1 second of it being captured, the result is undefined.
exanic_cycles_t exanic_expand_timestamp(exanic_t *exanic,
exanic_cycles32_t timestamp)
The output of exanic_expand_timestamp is an exanic_cycles_t value. As the name suggests, this value a cycles count (since the Unix epoch). For speed sensitive applications we recommend performing timestamp related calculations in cycles space.
However, it is often useful to represent the final result as nano/picoseconds. The libexanicAPI contains a variety of functions for converting to useful formats. Nanoseconds values can be converted directly into a 64 bit nanoseconds since the epoch value, or into a familiar UNIX struct timespec value.
void exanic_cycles_to_timespec(exanic_t *exanic, exanic_cycles_t cycles,
struct timespec *ts);
uint64_t exanic_cycles_to_ns(exanic_t *exanic, exanic_cycles_t cycles);
Picosecond values since the UNIX epoch can be contained without overflow in an exanic_timespcps_t structure. Meanwhile smaller values (such as those coming from a difference between timestamps) can be converted directly into a 64 bit picosecond value.
struct exanic_timespecps
{
uint64_t tv_sec; /* seconds since UNIX epoch */
uint64_t tv_psec; /* picosecond portion */
};
void exanic_cycles_to_timespecps(exanic_t *exanic, exanic_cycles_t cycles,
struct exanic_timespecps *tsps);
uint64_t exanic_cycles_to_ps(exanic_t *exanic, exanic_cycles_t cycles,
bool *overflow);
Feedback