Generally my answer would involve Riots
But mainly: remember it takes time for the special forces to get somewhere. If you know where they're basing, particularly (they just huddle up somewhere when not going after you), you can figure out how long you've got to take something out. The more distance they have to cover, the more spread out they'll be due to speed differences within the special forces fleet, which lets you defeat them in detail.
Also going with that: if you hit them in one place with a decoy fleet (but strong enough to draw their attention) farther away from your actual target, it will take them that much longer to respond.
Finally: they don't defend every planet; it's gotta have a Broken Golem, core world, home world, or CSG (if you have CSGs off, then an AdvFact or ARS) for them to bother.
Of course, you may also be seeing the threatfleet if you have any threat, and they follow different rules.