Personally I don't think the current system really works very well. The city building doesn't really improve anything as you are not really making any truly meaningful decisions in where you choose to place your buildings. Now I would like to suggest their are many routes forward to improve this and really it is still in its infancy.
1) Change the buildings so they still receive bonus from terrain but are negatively affected by distance from the settlement. However the problem with this is the fact that due to rng, you could simply end up with an ineffective city, thus I would suggest that a new guardian power could be created, quite rare, which emulates the effects of placing said building in its (correct) place.
2) Alternatively you could just simply abandon the bonuses, although they do make aesthetics sense the distance away from the settlement not being included means that even if these bonuses are included they don't actually make logical sense.
3) A third option is that placing buildings could actually draw the overlord attention towards you, leading to invasions and such as suggested by :
http://www.arcengames.com/mantisbt/view.php?id=8880Thus you can build the buildings anywhere but the further you build the buildings away from your settlement the more attention you draw from the overlords and the more responses they will send. It is similar to the idea of the current ai threat levels in ai war. You can either farm inefficiently but keep the overlords attention away from ya or ya could farm efficiently but draw his attention.