For each remote signal there's a hardcoded desired-depth-within-AI-territory (2 for the first couple, 3 for the third, 4 for the second-fifth city shard, and I think 6 for the last one, though I had considered just having it always spawn on an AI homeworld).
It then gets a list of all non-AI planets, this is "List 0"
It then gets a list of all planets adjacent to one of those that's not in List 0, this is "List 1" (and is all AI planets)
It then gets a list of all planets adjacent to a planet in List 1 that isn't already in lists 0 or 1.
etc, until it winds up with a list of zero planets because it's already filed them all in some list or another.
I should note that for the first couple signals spawns it refuses to add a mkIV AI world to any list. And for all signals except the very last one refuses to add a core AI world or AI homeworld to any list (this doesn't mean that your shortest route to a signal won't take you through a planet that wasn't on the list, it just pathed to it a different way).
It then picks the list corresponding to the desired-depth-within-AI-territory. If there actually are none it checks the next higher list, repeat. If all the higher lists have zero, it starts at desired-depth minus 1 and checks, going down from there. It will actually use a planet of depth 0 (non-AI territory) if there are literally no AI planets eligible, like when you're bottlenecked by a mkIV world for one of the first two signals (as Lance discovered).
Anyway, once it has a list, it checks each planet in that list for distance-from-a-human-homeworld, and only considers those with the smallest such distance. So if the list has 4 planets, with distance-to-human-homeworlds of 4, 4, 7, 12, it will always pick one of the "4"s, randomly between them. This also means that if somehow List 0 is chosen above, it will always pick a human homeworld.
I should add that the "don't go through a mkIV world on the first 2" rule, the "don't go through an AI core or AI home world on any but the last" rule and the "pick as close to the human homeworlds as you can within the eligible list" rules were all not part of the original design and were added later based on player feedback/suggestions
Before anyone asks, yes, this will wind up in the wiki
mavorspam, with regard to your question just now, there is very little if any documentation on the wiki of the Fallen-Spire stuff since I was preserving a bit of anti-spoiler for a while, but now since everything is done aside from balancing it (and hopefully that's come along some way) I'll be trying to work on that today.