Re: [chrony-dev] [GIT] chrony/chrony.git branch, master, updated. 1.25-pre1-18-g20a4340

[Bill Unruh <unruh@xxxxxxxxxxxxxx>]

On Thu, 14 Apr 2011, Miroslav Lichvar wrote:

> On Wed, Apr 13, 2011 at 02:28:40PM -0700, Bill Unruh wrote:
> > On Wed, 13 Apr 2011, Miroslav Lichvar wrote:
> > >Why? A significant part of the distance is static delay corresponding
> > >to the length of the path which the signal has travel through. I just
> > >wanted to remove that from the weight calculation.
> > 
> > 
> > But subtracting off the min_distance surely removes that already (the
> > min_distance is calculated from the string of samples for that particular
> > source).
>
> I don't think it does.
>
> Assume we have two sources with the same jitter, but different static
> delay. Four samples for each source. Let's calculate the weights in
> both ways with the same SD_TO_DIST_RATIO 1.0.
>
> source1:
> peer distances (ms):    11.0    13.0    12.0    14.0
> min dist (ms):          11.0
> last sd (ms):           3.0
> weights1:               1.000   1.397   1.190   1.620
> weights2:               1.000   2.778   1.778   4.000
>
> source2:
> peer distances (ms):    1001.0  1003.0  1002.0  1004.0
> min dist (ms):          1001.0
> last sd (ms):           2.0
> weights1:               1.000   1.004   1.002   1.006
> weights2:               1.000   2.778   1.778   4.000
>
> weights2 are same for both sources, but weights1 are very different.

Agreed, but then I at least would have a lot less confidence in 2 rather than
1 because of the longer delay.

As I said, the problem is with the calculation of the stddev and the use of
weights. The calculation assumes that a weight Wi is treated as if it
 indicates that that item was measured Wi with the same value each time. The
more times you measure a random variable, the smaller the estimated standard
deviation becomes. But that is clearly wrong. If I give each item a weight of
1000, the variance drops by a factor of 1000 but just because I gave them
those weights does not mean that their variance has decreased. Ie, another way
around it would be to use normalised weights-- force the weights to sum up to
1. This would make no difference to the calculated slope and intercept-- those
are homogeneous of degree 0 in the weights-- the weights naturally scale out,
but the variance is not. It scales inversely with the weights which is why you
are getting your runaway. For a small sd, the weights go up, which makes sd
still smaller, which shoots the weights up even more, etc. That is simply
wrong. If anything the sd should go up if you have large weights, but
certainly not down.

Also, the small sd gets reported and used in the hi/lo calculations and the
skew calculation, and may other things, and it is simply wrong to use that
small sd. In fact, with weights, even the formula for the uncertainty is
slopes is wrong. The usual one assumes that each measurement point has the
same variance, but the use of weights is precisely to argue that the variance
of some of the points is larger than others. (If you get an anomolous delay,
you expect that delay to also deliver a higher variance to that point--
undortunately we have no way from the data to estimate that)

In short, the current calculation of sd ( or the variance of the slope or of
the intercept) in the way it handles weights is simply wrong, and that
wrongness is leading to the instability.

> > >The selection algorithm takes care of that.
> > 
> > I was a bit confused. The weights are calculated separately for each source
> > as far as I understand it. So the problem becomes what happens if that source
> > suddenly has a large delay. Do you trust it? If all of the items from the
> > source suffer the same delay, then subtracting off min_distance takes care of
> > that. Surely you would trust a measurement whose delay suddenly became large
> > less.
>
> I think we still do that even by dividing by sd.
>
> --
> Miroslav Lichvar
>
> ---
> To unsubscribe email chrony-dev-request@xxxxxxxxxxxxxxxxxxxx with "unsubscribe" in the subject.
> For help email chrony-dev-request@xxxxxxxxxxxxxxxxxxxx with "help" in the subject.
> Trouble?  Email listmaster@xxxxxxxxxxxxxxxxxxxx.

--
William G. Unruh   |  Canadian Institute for|     Tel: +1(604)822-3273
Physics&Astronomy  |     Advanced Research  |     Fax: +1(604)822-5324
UBC, Vancouver,BC  |   Program in Cosmology |     unruh@xxxxxxxxxxxxxx
Canada V6T 1Z1     |      and Gravity       |  www.theory.physics.ubc.ca/

---
To unsubscribe email chrony-dev-request@xxxxxxxxxxxxxxxxxxxx with "unsubscribe" in the subject.
For help email chrony-dev-request@xxxxxxxxxxxxxxxxxxxx with "help" in the subject.
Trouble?  Email listmaster@xxxxxxxxxxxxxxxxxxxx.