Crime makes it so that one of your buildings per turn in that town turns to a bandit yellow building every turn. It's not as obvious yet partly because the end of turn reports are not in, to be honest. Though on the description of I believe the military buildings and maybe town center in the sidebar window, I think it talks about what crime does.
As for building destruction, aside from trojan horses or token powers, no they don't get destroyed. When they hit 0hp they go to a white flag and can't be used. The idle message shouldn't be showing, it should say something better than that. Minor UI thing. After 10 turns of being white, the building returns to the control of the player who holds the town. Or if the whole town changes hands, then all the buildings that are white immediately spring to full health on the side of the new owners.
