Reserves cannot go through wormholes. You can provoke them to spawn on core worlds, but not past a certain minimum remaining % that's held in reserve for HW defense.
There's no normal way to know for sure which ships are reserves, but you can:
1) Bait everything else off the planet, process of elimination (in at least two senses).
Or
2) Get scout coverage, attack until reserves spawn, leave, and watch which ships suddenly disappear all at once within the following minute or so as the reserves "reabsorb" them.
Or
3) Press F3 to toggle debug info on and mouseover a ship; if they have a special behavior type (special forces, strategic reserves, threatfleet, or the rarer-and-not-particularly-important planetary-roamer) it will show in the little tooltip next to the mouse cursor.