Are you sure they are swarmer hulls? They are meant to be close-combat hulls, which you attack with Fighters, not Missile Frigates.
Combining several of the below will help:
- Unlock Fighter III, not Missile Frigate III.
You should have a much easier time already with them.
- In fact, you may as well send you whole triangle-ship blob if possible. Chameleons have no bonuses against them.
- Snipe at the main blob with a squad of plasma seige-Is (immediately available). You want to disable their engines (15 ships each shot) and leave those stragglers for the clean-up duty afterwards.
- Get the Grav turret-Is, and cover your flaks and tractors with one FF. Hardened-FF will last a bit longer.