Re: [chrony-dev] SW/HW timestamping on Linux

[Denny Page <dennypage@xxxxxx>]

Miroslav,

I found the problem. The hardware receive timestamps are incorrect. It doesn’t matter if there is a switch involved or not.

NTP depends upon round trip latency being symmetrical for it’s accuracy, both in local network and in remote networks. To help ensure this, NTP uses preamble timestamps for transmit and trailer timestamps for receive. The following article makes for good background reading:

      https://www.eecis.udel.edu/~mills/stamp.html

From that article:

	• A preamble timestamp is struck as near to the start of the packet as possible. The preferred point follows the last bit of the preamble and start-of-frame (STF) octet and before the first octet of the data.
	• A trailer timestamp is struck as near to the end of the packet as possible. On transmit this follows the last octet of the data and before the frame check sequence (FCS); on receive this follows the last octet of the FCS.

In the current Chrony implementation, we have the following timestamp types available:

	• Daemon timestamps. Time stamps generated in application space.
	• Software timestamps. Timestamps created by the kernel / driver using SOF_TIMESTAMPING_TX_SOFTWARE and SOF_TIMESTAMPING_RX_SOFTWARE. This is called “driver timestamping” in the article.
	• Hardware timestamps. Timestamps created by the network interface using SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_RX_HARDWARE.

With the current implementation, software timestamps are the most accurate. Depending on network speed, processor speed and load, daemon timestamps are second, and hardware timestamps third.


Looking at the transmit side, these are the steps involved:

	send() invoked
	… variance from time in kernel ...
	kernel invokes driver
	… variance from time in driver ...
	driver writes packet to network controller
	… variance from time in network controller (packet scheduling/buffering, inter-packet gap, preamble collisions, etc.) ...
	network controller sends SFD (start frame delimiter, called STF in the article)
	network controller sends data bytes
	network controller sends FCS (frame check sequence)

With daemon timestamps, Chrony generates the timestamp right before invoking send(). We suffer variance from time in kernel, network driver, and network controller. Software timestamping improves this by moving timestamp generation to the point immediately prior to the network driver writing the packet to the network controller, which removes the variance from the kernel and driver. Hardware timestamps improve this even further by moving timestamp generation to the point immediately following the SFD and prior to the data being sent, removing the last of the variance. The perfect preamble timestamp.


Looking at the receive side, these are the steps involved:

	network controller receives SFD
	network controller receives data bytes
	network controller receives FCS
	network controller generates interrupt
	… variance from time waiting to process interrupt ...
	driver invoked
	… variance from time in driver ...
	driver hands packet to kernel
	… variance from time in kernel ...
	select() returns
	recv() invoked

With daemon timestamps, Chrony generates the timestamp immediately after select returns. We suffer variance from time in the network driver and kernel. Software timestamping improves this by moving timestamp generation to the point immediately after the driver hands the packet to the kernel, which removes the vast majority of the variance from time spent in the kernel. Now we come to hardware timestamps. What we would desire from hardware timestamping is that it be generated immediately following the FCS, which would remove the rest of the variance and give us the perfect trailer timestamp. However that’s not how the network controller generates hardware receive timestamps. With network controllers, the hardware receive timestamps are generated immediately following the SFD, just as they are for transmit. See Figure 7-27 (Time Stamp Point) on page 454 here: http://www.intel.com/content/www/us/en/embedded/products/networking/ethernet-controller-i350-datasheet.html.

What this means for NTP is that hardware timestamps are off by a minimum of 752 transmission bits with IPv4. This assumes no VLAN, and no IP option headers. In a 100Mb network, this means a guaranteed minimum timestamp error of 7.52 microseconds.

In order to generate a correct receive timestamp from the Ethernet hardware timestamp, one needs to need to have the FCS timestamp, the current interface speed, and the length of the packet at the Ethernet level. This is doable, but requires use of raw sockets and is quite a bit of work. A simpler (and safer) approach would be to use a combination of hardware timestamping for send (SOF_TIMESTAMPING_TX_HARDWARE), and software timestamping for receive (SOF_TIMESTAMPING_RX_SOFTWARE). This is probably the best available option.

Denny


--
To unsubscribe email chrony-dev-request@xxxxxxxxxxxxxxxxxxxx with "unsubscribe" in the subject.
For help email chrony-dev-request@xxxxxxxxxxxxxxxxxxxx with "help" in the subject.
Trouble?  Email listmaster@xxxxxxxxxxxxxxxxxxxx.