Hey Verily,
Glad you're enjoying the game so much!
For planets that are on alert, if I recall it is usually around 100 of your military ships that have to be directly at the planet (not in transports) to put it and adjacent planets on alert. So very tiny raids will not alert the planet, but anything substantial will. As soon as your fleet leaves, the AI will go off of alert back to the way it was, though.
What is irreversible about alert is when you take a planet. The AI views any planets that it does not control as being hostile, so that includes your planets and neutral planets. There's no way to get it off of alert once it does not control a planet, as it never takes planets back or anything like that.
Hope that helps!