I never said it would be easy
I've seen this done somewhere else but at the moment I cannot recall, i'll get back to you sometime when I remember it.
The easier way is probably to not calculate where to start it, but simply start connecting ships dot to dot. you could just do this for turrets on wormholes if too intensive, or add it to the options for those of us with grunty machines.
When the dot-to-dot cannot connect to anything of its same type and is greater than 15 units in number, use render-to-texture to place alpha channel circles(over each unit) on a new texture, which is then applied to the either the original unit, or average center and stretches over the area. Apply your favorite color!
A loop check of 4000-8000 units in should be within cpu restrictions. It's possible you might have to parse it every 60 frames, or calculating 100 ships a frame and only at farzoom etc
Its also quite possible this will be uneven, but will provide FAR more information per second than mess of ships at farzoom.
Ship types inside the circle should just be a single color-outlined icon with number next to it, I don't think thats clutter compared to a 50 turrets dotted everywhere.
This is mostly theory, and i'm not saying to implement this.. I would just like to see a slightly sexier implementation of far zoom battlefield information.