I think one of the big problems is that the AI gets alerted if you so much as sneeze in the direction of one of it's worlds, but it doesn't particularly care if you dump 100 or 1000 ships on to one of it's planets. You can't really place a decent raiding force on their planet without them starting to spam ships like crazy, and if you actually start taking worlds, the AI is all over you garrisoning tons of ships at every planet near the one you took.
What if the AI was made less sensitive to your incursions, at least at first? So you could send in some bombers and ninja raid the command station or important target, or maybe actually use raid starships to destroy something
So you'd have:
* "Unalerted" (no reinforcements, perhaps extra ships over a certain amount are funneled away to other worlds),
* "Low alert" (receives few reinforcements, doesn't alert nearby worlds), and
* "High alert" (receives a lot of reinforcements and bumps adjacent worlds to low alert).
Low alert would be common to have when you attack with raiding forces, but it would only stay that way while ships are there. If you destroy an important structure (like a guard post or ion cannon or something) on the planet, then it will enter high alert for a short period of time.
If you mass a major fleet or build a large structure within 1 hop of an AI planet (what constitutes major and large structure is TBD), then that planet is placed on high alert until the fleet or structure goes away.
All guard posts become "Warp Gate (REINFORCEMENT ONLY)" structures, and if the command station is destroyed the planet is put on high alert until no more warp gate structures remain.
Transports offer a lower profile for transporting and storing ships, and allow you to keep more ships in one sot than would be expected.
All of this is aimed to give players an active disincentive to using large fleets to do one thing at once (for the most part, homeworlds still need a ton of ships I bet). Maybe it's just the style of maps i've been choosing though.