[chrony-users] Restarting the chrony service brakes chrony

I have a strange problem: When chrony is started after boot all works fine.

chronyc --version
chronyc (chrony) version 4.1 (-READLINE +SECHASH +IPV6 -DEBUG)

After 5 min:
chronyc sources

MS Name/IP address Stratum Poll Reach LastRx Last sample

===============================================================================

^- excalibur.prolixium.com 2 6 377 13 +5513us[+4097us] +/- 98ms

^+ gowest.hojmark.net 2 6 371 13 -4181us[-5596us] +/- 59ms

^* ntp0.bgwlan.nl 1 6 327 12 -138us[-1553us] +/- 26ms

^? cloud.discloud.biz 2 6 3 13 +2145us[ +729us] +/- 26ms

Now I restart chrony
sudo service chrony restart
and I get after 1 min:
MS Name/IP address Stratum Poll Reach LastRx Last sample

===============================================================================

^? ntp18.kashra-server.com 0 6 0 - +0ns[ +0ns] +/- 0ns

^? dns02.wsrs.net 0 6 0 - +0ns[ +0ns] +/- 0ns

^? time.cloudflare.com 0 6 0 - +0ns[ +0ns] +/- 0ns

^? ntp1.trans-ix..nl 0 6 0 - +0ns[ +0ns] +/- 0ns

And after 2min

[bmterra@armxl ~]$ chronyc sources

MS Name/IP address Stratum Poll Reach LastRx Last sample

===============================================================================

^? nts1.time.nl 0 6 0 - +0ns[ +0ns] +/- 0ns

^? ntp18.kashra-server.com 0 6 0 - +0ns[ +0ns] +/- 0ns

^? dns02.wsrs.net 0 6 0 - +0ns[ +0ns] +/- 0ns

^? time.cloudflare.com 0 6 0 - +0ns[ +0ns] +/- 0ns

^? docker01..rondie.nl 0 6 0 - +0ns[ +0ns] +/- 0ns

^? ntp1.trans-ix.nl 0 6 0 - +0ns[ +0ns] +/- 0ns

^? aardbei.vanderzwet.net 0 6 0 - +0ns[ +0ns] +/- 0ns

I mean, I can ping the time servers:
ping nts1.time.nl

PING nts1.time.nl(nts1.time.nl (2a00:d78:0:712:94:198:159:11)) 56 data bytes

64 bytes from nts1.time.nl (2a00:d78:0:712:94:198:159:11): icmp_seq=1 ttl=56 time=53.5 ms

64 bytes from nts1.time.nl (2a00:d78:0:712:94:198:159:11): icmp_seq=2 ttl=56 time=36.6 ms

64 bytes from nts1.time.nl (2a00:d78:0:712:94:198:159:11): icmp_seq=3 ttl=56 time=34.8 ms

--- nts1.time.nl ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 5ms

rtt min/avg/max/mdev = 34.818/41.657/53.519/8.423 ms

ping ntp18.kashra-server.com

PING ntp18.kashra-server.com (188.165.11.86) 56(84) bytes of data.

64 bytes from ntp18.kashra-server.com (188.165.11.86): icmp_seq=1 ttl=51 time=91..1 ms

64 bytes from ntp18.kashra-server.com (188.165.11.86): icmp_seq=2 ttl=51 time=76..7 ms

--- ntp18.kashra-server.com ping statistics ---

3 packets transmitted, 2 received, 33.3333% packet loss, time 5ms

rtt min/avg/max/mdev = 76.662/83.883/91.104/7.221 ms

Any idea why chrony stops to work after a restart of the service?

Or how to debug that further?

Uwe