Oh, you better believe this is something we've had endless internal discussions about; we're not ones to do... anything... terribly casually. The leveling serves a really large number of purposes, but among them:
1. As you noted, you get new equipment, spells, buildings, types of areas to explore, types of enemies to explore, and so on. This basically is what gives the game a lot of its progression.
2. This also provides a reason for questing and a way to have an infinite progression in a single world. In other words, let's say you play AI War: you start out with very little, then you gain stuff, then you win or lose as you face the AIs. Then you start over and play again. We wanted that same sort of flow in AVWW, but having it many times over inside the same world so that your history gets kept, etc. Since there is inherently a finite amount of content that is genuinely new in any given game, you're either going to have to start over at some point if you want to keep playing, OR you're going to have to have your stuff become obsolete. In AVWW there's a 5-level cycle of equipment obsolescence, which means that players are encouraged to experiment and upgrade their existing equipment, as well as finding new stuff.
3. This also lets us introduce new content that is ostensibly available to lower-level players, but which is also new and exciting and interesting to players who are at a very high civ level. If we introduce a new spell that becomes available at level 6, but you are already level 60, you didn't miss that content -- you'll just have the first version of that spell be a much higher tier than most players, and it will still be something you can enjoy no matter how long you've been playing a single world.
4. This is also how we're able to have a large amount of choose-your-own-difficulty in the game: being able to play up or down region levels relative to your civ level is the most organic way to choose your level of challenge as you go.
5. There's other stuff too, but I gotta run to dinner and those are certainly the high points.