Depending on your difficulty setting, you can sometimes entice enemy ships to chase you back through your wormhole a few at a time, which is good if you have a deathtrap waiting. A bunch of tractor beams to hold them in place, lightning turrets, and other turrets along with your fleet can make short work of even mark V ships if they come through in smallish doses. If you have heavy beam cannons, especially the mark III and IV, these help immensely against the higher mark ships.
For "bait" typically I use a few empty transports, which really seems to tick the computer off. On higher difficulty settings this doesn't work, though, and you'll have to actually go blow something up. Sneaking siege or raid starships in via a transport and having them blow up a guardpost of some sort seems to work sometimes. This tactic can backfire if you don't have enough defense and the computer just decides to roflstomp you with all 1200 units, so proceed with caution. You also need to watch for guardians because some of them can utterly chew up turrets.
I don't know if this will work, but if you have maws or etherjets, you can attempt to whittle down their army by physically dragging some of them through your wormhole into 'badly outnumbered' territory. To do this effectively you need to send something with a lot of HP through to take the first hits. Transports work well for this, as do starships.
Yet another (rather fun) tactic is to get mobile repair stations and leech starships and steal their ships. This doesn't work nearly as well as it used to though, since leech starships got hit pretty hard upside the head with the nerfbat in terms of reclamation. Mercenary micro parasites might work better for that, though. It's also been a while since I tried it but I'm not sure if core ships can be reclaimed either. Even if you don't reclaim mark V ships though, you can go to a different neighbor and steal some of their ships to use against the core world.
Another tactic that sometimes works is to move in (or build) a bunch of mark II/III force fields along with a mobile repair station and 40 engineers (and army too), then start building sniper/spider turrets and such in a distant corner of their planet, behind the force fields. You need a really big stock of resources for that, though, and have to be willing to micro quite a bit while under pressure. While you're in there you can knowledge raid, too.
Hope some of that helps give you ideas!