Re: [chrony-users] Monitoring Chrony

On Thu, Feb 11, 2016 at 1:58 PM, Miroslav Lichvar <mlichvar@xxxxxxxxxx> wrote:

On Thu, Feb 11, 2016 at 10:54:49AM +0100, Ben Kochie wrote:
> So far, I haven't been able to find a good programmatic way to extract
> stats with chronyc. There are a bunch of annoying parsing issues with
> things like the sourcestats command. The offset includes a precision, so I
> have to parse the precision and convert that to be all in one precision.

Yeah, I've struggled with that too. I like the human readable format
when inspecting the chrony state, but it does complicate parsing quite
a bit.

> A couple of specific questions.
> * Would chrony be interested in supporting the Prometheus metrics format?

I looked at the page describing the archicture, but it's not clear to
me how would a support in chrony look like. Would chronyd or something
using the chronyc protocol be listening on a port for requests? Or
would it periodically push data over socket somewhere? The page
listing client libraries does't include a C library.

Typically we do this one of a few ways.

#1 - The application listens on a port for http requests, the default is /metrics. It then can respond with plain/text in the format I posted above. Or it will content negotiate and use grpc, a nice compact protobuf format. The grpc format is the most efficient, but we've had few problems collecting text metrics at scale.

#2 - We run a side-car exporter. We do this quite a lot for existing open source software, like mysql, that would never listen on http, but can provide metrics with their own protocol.

#3 - The way we collect metrics for ntpd, is we have a loop script, or cron script, that parse output and put that output in prometheus format into a text file. Then we access these metrics via the node_exporter's textfile reader.

#4 - We use something like mtail[0] and parse log files. This is what I do for things like apache[1] that have minimal useful internal metrics.

[0]: https://github.com/google/mtail

[1]: https://github.com/google/mtail/blob/master/examples/apache_metrics.mtail

> * Is there a mode for the various metrics outputs to be more machine
> readable? (json?)

No, not yet. I'd like to add a raw mode to chronyc that would print
the values in something easily parseable. I'm not sure about json, I'd
probably prefer something usable even from shell using just sed or
awk.

One idea I had would be to add a "metrics" command to chronyc. Then you could run a loop/cron job that would be basically "chronyc metrics > chrony_metrics..prom"

The output format would be sed/awk friendly as you always get one metric key and value per line.

> * Is there documentation for the chronyc protocol outside the code?

No, unfortunately not. FWIW, the protocol is quite simple, almost all
information you would need to implement a new client is contained in
candm.h.

Ok, I will take a look.

> * Are there any non-C chronyc client implementations? (python/ruby/whatever)

Probably not, at least I've not seen anything. At some point I'd like
to split chronyc into a library and a client application. Bindings for
other languages could then be easily created.

This would be pretty nice.

--
Miroslav Lichvar

--
To unsubscribe email chrony-users-request@xxxxxxxxxxxxxxxxxxxx
with "unsubscribe" in the subject.
For help email chrony-users-request@xxxxxxxxxxxxxxxxxxxx
with "help" in the subject.
Trouble? Email listmaster@xxxxxxxxxxxxxxxxxxxx.