Re: [chrony-dev] [GIT] chrony/chrony.git branch, master, updated. 1.25-pre1-18-g20a4340

[Ed W <lists@xxxxxxxxxxxxxx>]

On 14/04/2011 03:55, Bill Unruh wrote:
> Ok I think I have figured out what is happening. The way in which the
> weights
> are used in the program is as if a set of w_i measurements were made at the
> point x_i, with each of them yielding the value y_i. But the estimate of
> the
> standard deviation goes as 1/sqrt(N) where N is the total number of
> measurements. Ie, as the w_i grow, the variance decreases as 1/W=1/sum W_i
> 
> As you say this is unstable. I think at present the best thing would be to
> revert to the old 1+(distance_i-Min_dist)/Min_dist. Alternativey you could
> use something like
> 
> M= (Min_dist+sd)
> This essentially makes M the larger of the Min_dist and the standard
> deviation and then divide by that instead of by Min_dist. This means
> that if
> the stadard deviation is smaller than the min distance, then one uses the
> min distance, but is the standard deviation is larger, then you use that.
> 
> Alternatively you could divide the weights by the mean weight to normalise
> the weights. This should prevent the runaway, but it still leaves open
> why you
> would divide by sd rather than by the min_dist

I don't understand the context of the calculation, but from a pure
number theoretic point of view it's dangerous to follow Gaussian
distributions into the tail region when modelling "real" probabilities.
 Real situations tend to look like a gaussian with a large "tail" event
that occurs infrequently, but with relatively massive magnitude (I come
from a finance background, so to me I think of things behaving like a
gaussian with the odd crash every so often - in ntp terms I guess we
mean steady state and sudden massive network bottlenecks)

The way to deal with this obviously varies depending on the domain, but
I'm a fan of bounding the confidence level at some lower level.  In the
calculation Bill describes, we seem to be using sqrt(N) as a kind of
proxy for "confidence", this (or any other calculation) intuitively
requires a lower bound to keep the calculations sane.  Bill describes
several suggestions, but I'm sure there are others also.

Seems like if you find a performance advantage in weighting things, then
this is worth preserving, however, as you already stated, all weights
need a bound to avoid getting into the tail regions of what is basically
a "confidence" calculation.



However, getting back to release schedules - we have git - it's
WONDERFUL for branching... Can I please suggest a feature freeze on the
current code, revert anything which could be controversial and stabilise
(urgently?) for a release.  Create a new branch for the next version and
hack away on new confidence calculations in there.  Remember that if you
checkin a bug fix it's trivial to "pull" that to all release branches
(ie git makes it easy to maintain 1.1.x and 1.2.x and 1.3.x, where bug
fixes get easily pulled into all relevant branches in one hit!)

I really appreciate your work on this, but for right or wrong I think
most users do not test git code points, only "numbered releases" and so
it's important to get a "number" on the current code and get it out?

Please also give serious consideration to a new numbering convention now
we switched to git?  Git branching now makes it much easier to develop
in multiple code bases at the same time, so releases should become
slightly more frequent, with each release being a "feature", and bug
fixes/non dangerous improvements are easily backported to previous
releases.  This seems to work for both developers and users in that you
can now pick a branch with the features you need and stabilise on it

Thanks for all your work on this - hope the above doesn't teach you to
suck eggs and appears constructive?

Ed W

---
To unsubscribe email chrony-dev-request@xxxxxxxxxxxxxxxxxxxx with "unsubscribe" in the subject.
For help email chrony-dev-request@xxxxxxxxxxxxxxxxxxxx with "help" in the subject.
Trouble?  Email listmaster@xxxxxxxxxxxxxxxxxxxx.