Re: [chrony-users] Monitoring Chrony

- Ben Kochie

On Fri, Feb 12, 2016 at 10:05 AM, Miroslav Lichvar <mlichvar@xxxxxxxxxx> wrote:

On Thu, Feb 11, 2016 at 02:12:36PM +0100, Ben Kochie wrote:
> On Thu, Feb 11, 2016 at 1:58 PM, Miroslav Lichvar <mlichvar@xxxxxxxxxx>
> wrote:
>
> > I looked at the page describing the archicture, but it's not clear to
> > me how would a support in chrony look like. Would chronyd or something
> > using the chronyc protocol be listening on a port for requests? Or
> > would it periodically push data over socket somewhere? The page
> > listing client libraries does't include a C library.
> >
>
> Typically we do this one of a few ways.

> #2 - We run a side-car exporter. We do this quite a lot for existing open
> source software, like mysql, that would never listen on http, but can
> provide metrics with their own protocol.

This one seems most reasonable to me. A separate service that uses the
chronyc protocol to read the metrics from chronyd.

> #3 - The way we collect metrics for ntpd, is we have a loop script, or cron
> script, that parse output and put that output in prometheus format into a
> text file. Then we access these metrics via the node_exporter's textfile
> reader.

This is probably the easiest way :).

> #4 - We use something like mtail[0] and parse log files. This is what I do
> for things like apache[1] that have minimal useful internal metrics.

The chrony logs are good in showing when exactly has the state
changed, but if you are interested in metrics like root dispersion,
which are constantly changing (in a deterministic way), you would have
to calculate their current value.

> > > * Is there a mode for the various metrics outputs to be more machine
> > > readable? (json?)
> >
> > No, not yet. I'd like to add a raw mode to chronyc that would print
> > the values in something easily parseable. I'm not sure about json, I'd
> > probably prefer something usable even from shell using just sed or
> > awk.
> >
>
> One idea I had would be to add a "metrics" command to chronyc. Then you
> could run a loop/cron job that would be basically "chronyc metrics >
> chrony_metrics.prom"

Which metrics it would print? With the "clients" command for instance
there can megabytes of data, which in most cases probably wouldn't be
useful to collect, but in some cases I think it might, e.g. monitoring
if clients are alive from the server in a small network.

> The output format would be sed/awk friendly as you always get one metric
> key and value per line.

If there was just one key/value per line, wouldn't it be more
difficult for a simple sed/awk parser to group data by source, as in
sourcestats?

I was considering something like CSV, which can be parsed in shell
with a single "read" command and can be easily converted to more
verbose formats like json.

$ chronyc -r tracking
#refid,address,stratum,...
10.16.255.1,10.16.255.1,2,...

$ chronyc -r sources | grep -v '^#' | while IFS=, read mode state ...
do
echo $mode $state ...
done

--
Miroslav Lichvar

--
To unsubscribe email chrony-users-request@xxxxxxxxxxxxxxxxxxxx
with "unsubscribe" in the subject.
For help email chrony-users-request@xxxxxxxxxxxxxxxxxxxx
with "help" in the subject.
Trouble? Email listmaster@xxxxxxxxxxxxxxxxxxxx.