Ok, I think I understand, now, the noselect suggestion earlier and to focus in on the offsets. I found a link that discusses comparing to NTP sources and then calculating the offsets and making adjustments (
https://robrobinette.com/pi_GPS_PPS_Time_Server.htm#GPS_&_PPS_Offset_Tuning_With_Chrony)
I've started doing that and I'm getting better results with measured and corrected offsets. I'll continue to do that. I'm down to only 3-5 losses of majority in any given hour which is much better than every 2 minutes.
I still have the outstanding question of:
If I'm on GPS, social distincing myself in the woods, is there an option for a 3rd reference clock for "majority?" I could do two GPS units though that's just a bit overkill I think as I am ok with a bit of wobble. I could do an RTC but I know that'll drift and just be tied to the GPS anyway.
Thanks for the guidance from everyone so far. I've already learned more about chrony and ntp stuff in a week than for the last 20 years of just throwing up an ntp server to make sure all the clocks in my work networks are accurate(ish).