Yeah, they give the AI a place to put their reinforcements. They actually used to be completely invisible and invulnerable, but people kept a) seeing them when an EMP would go off and b) getting annoyed that there was no way to prevent reinforcements from spawning at the wormhole. They also don't count towards installations that count guard posts.
I generally never bothered clearing them out unless I *really* needed to pass through an AI controlled system. Here's the thing though, if you really need to regularly pass through a system, just blow up the command station and eat the 20 AIP.