Anyway, it should not be switching sources unless the deviation of the
selected source exceeds the variance of the alternative (or unless the source
has disappeared for a suitable number of poll intervals, probably related to
how long one would expect to wait for the drift rate variance to make the
system clock deviate by more than the second source's variance. Ie, you are
far better off letting a clock drift unconstrained for a while than to jump to
source which has a huge (factors of a 1000) worse variance.
The selection algorithm prefers sources with shortest distance (with
refclock that's the measured dispersion + configured delay). If there
are more sources with similar distance they will be combined together.
If a source disappears for 8 polling intervals, chronyd will select
another source even if it's much worse. I agree that could be
improved. With NMEA sources it's usually better to use the noselect
option or don't configure it at all.