Tackle Drone launchers and Blade spawners are the most annoying for me. Mostly because there isn't any way to kill them quickly or mitigate their damage. Maws, Stealth Battleships, Tractor Platforms, etc... I can all live with.
Btw, I have played through games with TDLs, but I find that they just force the playstyle of attacking exclusively with starships/superweapons and leaving all my fleet ships on defense. Which is annoying. Because when I'm attacking a defended AI planet, moving fleet ships away from my ball and keeping them away just makes them useless on the attack and eventually dead.