Hm, the fix helped with the spikes I was seeking. Did we rule out the
possibility that in your case the spikes are due to the other issue with
out-of-order HW timestamps? Could you try it with this patch to make
sure only measurements with HW timestamps are used?
--- a/ntp_core.c
+++ b/ntp_core.c
@@ -1434,6 +1434,9 @@ receive_packet(NCR_Instance inst, NTP_Local_Address *local_addr,
       prevent a synchronisation loop */
    testD = message->stratum <= 1 || REF_GetMode() != REF_ModeNormal ||
            pkt_refid != UTI_IPToRefid(&local_addr->ip_addr);
+
+    if (inst->local_tx.source != NTP_TS_HARDWARE || rx_ts->source != NTP_TS_HARDWARE)
+      testB = 0;
  } else {
    offset = delay = dispersion = 0.0;
    sample_time = rx_ts->ts;