[chrony-users] chrony-top

[Marc Compere <comperem@xxxxxxxxx>]

Hello,

I've accidentally made a chrony-top like application. It is a simple python script that uses successive output of `chronyc -n clients` to detect which IP addresses are (a) found in two successive reports and (b) have an increasing NTP request count.

I made this to detect the offending IP address during sporadic periods of unusually high NTP request activity. It turns out it detects the very high users but also updates with typical user's NTP requests over a period of 5 to 10 seconds.

It updates the console screen every dt seconds with IPs from the most recent highest NTP requests.

Can someone try this out and let me know if it works? It works on Ubuntu 20.04.2 quite well and reports something similar to the attached screenshot:

It must be run as sudo or root, so make sure you read the script before executing. It's simple and short. It makes a folder in /tmp to save output from `chronyc -nc clients`.

MDC

#!/usr/bin/env python3
#
# use output between two successive 'chronyc clients' results to detect which
# specific IP addresses are requesting a lot of NTP traffic over the last dt seconds
#
# this is meant to be run when many NTP packets are being requested and served by chrony
#
# use chronyc to list all clients, repeatedly, then find the IP with increasing packet count
#
# must be run sudo (or as root) because 'chronyc -n clients' must be run as root or sudo
#
# Marc Compere, comperem@xxxxxxxxx
# created : 09 Nov 2020
# modified: 27 May 2021

import os
import csv
import sys
import time
from datetime import datetime
import pandas as pd # https://www.usepandas.com/csv/sort-csv-data-by-column
import subprocess  # for clear_screen()
import platform    # for clear_screen()

# -----------------------------------------------------------------
#               create a directory to place files
# -----------------------------------------------------------------
baseDir='/tmp' # this must exist
dt=datetime.now()
newDir=dt.strftime("chronyc_clients_%Y_%m_%d__%H_%M_%S") # 'chronyc_clients_2021_05_17__21_51_30'
path = os.path.join(baseDir, newDir) # '/tmp/chronyc_clients_2021_05_17__21_51_30'

try:
    os.mkdir(path)
    print('just created: [ %s ]'%path)
except:
    print('could not create [ %s ]'%path)
    print('exiting.')
    sys.exit(-1)


file1=os.path.join(path,'file1.csv')
file2=os.path.join(path,'file2.csv')

# from the chronyc man page on clients: sudo chronyc -nc clients > chronyc_clients.txt
# The intervals are displayed as a power of 2 in seconds.
# The columns are as follows: 
#    1.The hostname of the client.
#    2.The number of NTP packets received from the client.
#    3.The number of NTP packets dropped to limit the response rate.
#    4.The average interval between NTP packets.
#    5.Time since the last NTP packet was received
#    6.The average interval between NTP packets after limiting the response rate.
#    7.The number of command packets received from the client.
#    8.The number of command packets dropped to limit the response rate.
#    9.The average interval between command packets.
#    10.Time since the last command packet was received.
#               1       2           3              4            5             6                     7      8    9      10
fieldnames = ['host','ntp_cnt','ntp_cnt_dropped','ntp_interval','ntp_t_last','ntp_interval_limited','cmd_cnt','cmd_dropped','cmd_interval','cmd_t_last']

cutoff_cnt = 5 # threshold above which to include in high-ntp-count list (make this smaller to show more IPs over the last dt seconds)

def get_high_ntp_cnt_list():
    fd1=open(file1,'w')
    retVal = subprocess.run(["/usr/bin/chronyc", "-nc", "clients"], stdout=fd1, stderr=subprocess.STDOUT ) # must be root for chronyc
    fd1.close()
    
    # bring into pandas dataframe
    df = pd.read_csv(file1)
    df.columns=fieldnames # assign column names
    
    # compute average ntp_cnt and standard deviation
    ntp_cnt_avg=df.ntp_cnt.mean()
    ntp_cnt_std=df.ntp_cnt.std()
    
    # sort, then isolate the top 6*sigma of high ntp_cnt clients
    sorted_df = df.sort_values(by=["ntp_cnt"], ascending=False)
    
    #cutoff_cnt = ntp_cnt_avg + 6*ntp_cnt_std
    #cutoff_cnt = 5
    high_ntp_cnt_clients_df = sorted_df[ sorted_df['ntp_cnt'] > cutoff_cnt ]
    
    return high_ntp_cnt_clients_df


def clear_screen():
    " clear terminal screen for win/mac/linux"
    # from: https://stackoverflow.com/a/42877403/7621907
    command = "cls" if platform.system().lower()=="windows" else "clear"
    return subprocess.call(command) == 0


cntMax=5
cnt=0
continous=1 # (0/1) run continuously?
dt=5 #10 # (s) delay between chronyc updates

while (cnt<=cntMax) or (continous==1):
    
    try:
        list1 = get_high_ntp_cnt_list()
        time.sleep(dt)
        list2 = get_high_ntp_cnt_list()
        
        # identify all client IPs in list2 that are also in list1
        # .isin() method from: https://datascience.stackexchange.com/a/19249/117876
        list2['in_both'] = list2.host.isin(list1.host)
        
        # these are the IPs that have ntp_cnt different from the two chronyc clients lists just created
        list3 = list2[ list2['in_both'] == False ]
        
    except KeyboardInterrupt:
        exit(0)
    
    
    #print('\n\n---------------------------------------------')
    clear_screen()
    now=datetime.now()
    print("{0}, dt={1}(sec), total chronyc hosts: {2}".format( now.ctime(),dt,len(list1) ))
    print("--------------------------------------------------------------------")
    print("recent hosts with NTP request count above {0} over the last {1} seconds:".format(cutoff_cnt,dt))
    print(list3)
    
    cnt+=1