Check out the wiki for all the really specific rules on what causes great when. I think it's under the ai section, but I'm typing from my phone.
In general:
1. No, all ships in a system are not threat. Threat is only ships that can attack you a any time. Most enemy ships are guards, special forces, or planetary roamers, none of which count as threat.
2. Waves where ai ships retreat are one big source of threat, sure. But also as you engage the enemy on their worlds, a lot of those guards, roamers, etc, will get alerted to you and become threat. Most of them you will kill, so it amounts to nothing. Some will run away and come back later, but you might miss that in the heat of battle. Other times you will lose a battle, and then those forces that drove you back will in turn go on the offensive.
3. There are other sources of threat such as cross planet attacks, border aggression, and similar. The wiki has the goods on them. Also some more specialized things, like alarm posts, cause threat. When you destroy a command station of the ai, that also has a 50% chance of making all it's ships on that planet into free/threat mode.
4. Bear in mind that most ships in threat mode won't attack you most of the time. They see they are outnumbered, so they stack up around your borders, usually in a loose pattern around the wormholes to your planets. This is "stalking." When they see they outnumber you, they will attack. If you were trying to trick them into attacking by temporarily withdrawing your forces, they will retreat again when you arrive. Some might try to slip past further into got territory, though so be careful. Most losses come from that.
Hope that helps as a starting point!