Oh, they can build up to pretty good quantities by the mid game (try scouting a pretty well connected, pretty "central" planet in the mid game or later, you will see a surprising number of ships flying around). The problem is that for most games, they tend to gather in areas that are far removed from human presence, where they don't contribute much to anything. The explanation for why is a little hard to explain (it has to do with the randomness they pick their target waypoint planets, how eligible waypoint planets tends to shift away from human presence as humans gain more ground (though not due to any intentional logic, it is just a result of the AI planet layout changing due to less AI planets), and because for many map layouts, central, "well connected" worlds are often not interesting or important worlds)
I've peddled my ideas about making special forces a bit smarter plenty of places elsewhere, so I won't do it here.
