If they are in a group, then anything goes -- they will position themselves in such a way that they all fit in the area that would be optimum range if there were only one of them. There's nothing we can really do about that, as the position-checking stuff is already one of the most hugely complex parts of the game, and doing checks against range would just cause a computational explosion. The code itself would be easy to write, but with so many ships it would result in game speed degrading exponentially during battles in particular. Writing code like that which wouldn't be a CPU-killer is quite nontrivial.
In terms of a single ship that is ordered to go kill something, it will approach until it sees that it is in range to attack the target (actually, the same is true of multiple ships, too, but when they stop if they are on top of other ships they have to move somewhere nearby to shoot, and that might be too far in or out, causing them to get shot or move continually).
