Kzzaman -- Kasnavada and Chthon gave some good tips, I'll just tell you more about hacking then and Threat.
Hacking
Hacking response spawns are indeed classed as Threat, that's why you're seeing the Threat tally climb even when you're blowing stuff up. If you keep your fleet on the hacking planet cloaked so they don't see a target immediately when the spawn, you'll see them promptly run off to go stalk a planet of yours (or attack it, if they find one they think they can take out).
The AI has a number of things it can do in response to hacking (I won't spoil them all, the surprise is fun >D), and one of the options it sounds like you might have seen is to send waves, either to that planet or to nearby ones. You'll get a warning popup with a timer like a regular wave, though it will probably read "sending to ? ? ? ?" as it will likely go to an AI-held planet, which you won't have warp detection on unless you have a ship with that ability there (Sentinel Frigates, a few others).
The "expect AI retaliation" is the standard hacking response warning -- as you are hacking, the game spawns ships every so many seconds in a pulse. That tells you the total strength of the ships that are about to spawn in that next pulse, so you have some warning of how bad things are going to get in a few seconds. When the hack ends, it does one final LARGE pulse before it stops trying to kick you in the face. This is the one that has the more complex warning of "Vigorous Response" or the like.
And yeah, the ships frantically streaming around are likely the hacking response, especially if they are popping out of random places on the map as well as the command station and guardposts, rather than coming through wormholes.
If you see ships coming out of the wormholes, especially if they are accompanied with Riot Control Starships, then you have probably gotten the attention of the Special Forces, presumably by hacking on a planet that the AI has designated as especially important. CSGs, fabs, factories, starship constructors, etc will get the planet tagged this way.
BTW, if you mouse over the HAP tally at the top of the screen, you'll see a note about the expected AI hacking response for future hacks. The more you hack, the nastier it gets. Just how nasty "Very Low", "Moderate", "Forest Fire", etc actually is is affected by the difficulty level. The ships that spawn tend to be Mk.III, but if you hack on a Mk.IV planet I believe I have seen Mk.IV ships spawn instead. I forget what came out the last time I hacked on an AI Homeworld, I was too busy trying not to explode at the time.
Threat
When ships enter Threat mode, they'll pick a planet of yours (how smartly they choose depends on difficulty) and group up on an adjacent AI planet not too far from the wormhole to enter it. If they accumulate enough strength within 30 minutes, or if enough of your strength is tied up in combat/more than 3 planets away, they'll attack. So it's not uncommon to see a wave hit a planet, and the nearby Threat piggyback on the attack. Difficulty level factors in -- I think it's at 7 where they will attack if they are about equal in strength to their target planet. Below 7 they will attack even if they are weaker, making it less likely they actually build up enough to trash your planet. Over 7, they wait until they are *stronger* by increasing amounts.
If they hit the 30 minute mark and haven't met the condition to attack, they switch to Threatfleet mode. They will move back away from the front lines by 3-4 hops and park. At that point, they are actively scanning for a new target, measuring the overall Threatfleet strength against different possible targets. When they see something they like relative to their own strength, and see an opportunity (your fleet is away, etc), they'll move up to the front lines again, next to their chosen target, and begin staging the attack. Since Threatfleet doesn't necessarily park on the same planet and ships don't all move at the same speed, it can take a little while for the whole army to arrive and the attack to start. This is why it's good to keep a scout on all AI planets adjacent to yours so you can see when this happens.
A trick that I use to know when Threat/Threatfleet is on the move is to keep an eye on the Threat tally at the top of the screen. If you see it start changing rapidly, but you're not doing anything that would cause ships to spawn or be destroyed, then that means that Threat is on the move. There is a slight delay where the ships despawn and respawn as they pass through a wormhole, so if you are alert you'll see the number briefly dip as the ship stops existing in the world for a split second, then snap back up as it reappears on its new planet. If you see the number "flickering" by a small amount, it's usually nothing to worry about (unless you missed the initial mass move to a staging point). If you see the number bouncing around wildly and for a longer period of time, it's almost certainly a sign that a large number of Threat ships are on the move, and that usually means they're preparing for an attack.
One last useful UI thing -- if you mouseover the Threat tally, it'll show you the number of ships and their strength, as well as what planet they are on. You only get that info for ships that are sitting on a planet you have scouting coverage on though, so the number of ships you see in the detailed notes may not match up with the total Threat tally.
Good luck!
