I'm generally talking about UK roads where the biggest problem is on a 2 lane road that sightlines are very limited due to bends and hedges or farm walls. Any overtaking opportunities are limited and there is greater risk the further and longer one has to spend in the oncoming lane. One's own lane is not risk free either due to oncoming trucks, vans RVs and SUV/BMW drivers afraid what such walls and hedges might do to their paintwork.
The opposing view is that both you and the cyclists recognise all you stated above.
But the cyclists have a better idea of how passable the road is in front of them than you do, simply because they're further forward along the sightline. Therefore riding single file is an invitation for you to pass whereas 2-3 abreast is an intentional statement of 'don't do this, not here, not now'.
That half of the argument applies to groups of similarly well seasoned, experienced, skilled cyclists - and only such.
The other part of my response has to do with group rides of mixed skill, tempo and equipment. Lateral drift is a fact of life in such groups because otherwise they accordion into each other with every tempo change, missed shift, or inability to hold line in a corner. Lateral drift is what they do for safety with regard to the other cyclists, and blocking your passing opportunity is
very far down their list of awareness priorities. They're just trying to keep themselves safe from the surging/lagging/darting squirrel riders in their immediate vicinity.
Oh, and - most riders who think they're 'safe' are actually squirrels. Unless they spend
a lot of time in pacelines.
Next time you see a group of 5-10 in a lane with riders abreast of each other or overlapping wheels, please recognise that ~4 of them are probably squirrels already endangering everyone.