Ah! That's why I stopped using forts! They come with huge amounts of firepower. When a fleet is huge they can generate border buildup, but once the fleet leaves the buildup is relieved. BUT! With a fort, that buildup does not go anywhere, until the buildup is enough to smash the fort on its own. That does not happen with turrets, and even once a in a blue moon it does, the turrets are easily replaced. The fort does not.
This doesn't seem like a legitimate complaint to me. If you let an enemy force build up over several hours outside of your Fortress' wormhole, you deserve to die when they finally become strong enough to overpower you.
This isn't tower defense, it's AI War. If the enemy is massing an army right outside your door, the appropriate response is to not just pretend like it isn't happening. There's plenty of mechanics in the game that let you deal with these kinds of build-ups, such as local missiles, EMP missiles, Spire Martyrs, Autobombs, melee ships, Neinzul ships, Riot Control Starships, Parasite Starships, Tractor Beam Platforms, and many other options to deal with this kind of threat.
I think the true power of the Fortress COMES from its ability to intimidate your enemy and buy yourself time to deal with it. If you don't like playing this way, the solution is simple: Don't build Fortresses. But plenty of us think they're fine as-is.