I don't know them all, but here are some conditions that will cause a ship to go 'free':
1. Their Guard Post is attacked*
2. Their Guard Post is destroyed
3. The Command Center is attacked (affects ALL Guard Posts on the planet)*
4. The Command Center is destroyed (again, affects ALL Guard Posts on the planet)
5. Certain AI Units will free ships under various conditions (eg an Alarm Post has a 50% chance to trigger if it is destroyed, 100% if a CC within 1 wormhole is destroyed)
6. You bring in much more firepower than the AI's defenders (I think at least double)--all AI units go live and enter fight-or-flight mode
7. Your attackers start firing on/get too near to the AI's ships. This depends on the range of the defending vessel. Typically, infinite range forces will attack you as soon as you enter. However, they generally will not free other ships at the guard post. Teleporting units may also be freed...I don't remember.
I'm sure that there are other conditions, but I'll let a more experienced player comment on that. Or correct me if I'm wrong on the above.
Note that defenders under a shield are much more difficult to get to move than others. Guess they like their little shooting post. Also, if the AIs' ships are already free on a planet, they will join in planetary defense if you attack. Or walk right past you and jump your planet. As a rule, you should expect a counterattack any time you enter an AI's planet with military units (anything that isn't a Scout--INCLUDING TRANSPORTS!!!).
Some players take advantage of this and just blob the planet, counting on a good economy and Space Docks on infinite loops to offset losses. This is definitely the faster approach, which makes it great for quickly clearing an important planet. Others, like myself, use things like engine damage and tractors (when available) to spread out defenders and keep losses down. Just because the guards go free does not mean you cannot fight a 'battlefield within a battlefield'...you just might have to create the arena yourself
Hope that helps.
*I think that all of these are 50% chance to free the guards, but I am not sure