Quite the post you have there. A very thought provoking one as well. As a fellow lover of 4X games (yet also see that they all fall below the mark) I have to discuss this with you. Many of my points will simply be echos of yours, but I feel I need to make them anyways. You have a bunch of great ideas I have never considered before.
1) Several levels of abstraction of scale both in terms of fleets and planet management.
Yes please. I think 4X fans will all be on board with this provided it is done right. The genre for a while has had a problem: It needs to simulate every ship, yet scale to thousands. You need to be able to control the first border skirmishes with single ships, all the way up to fleets as big as the entire production of the map. You need an AI that can handle the Economy just as well as the player, so he never feels he needs to micromanage every little thing to get the best performance.
Without constantly shifting abstraction layers, and with that an automatic removal of all Multiplayer functions
Annnnnnnnd you have destroyed any chance of the game you want ever being made in your second sentence. Back in the days before the Internet was always-on it was totally cool to make SP only games. These days, if you can't play with your friends, people don't buy your game. Human opponents are always the most challenging ones anyways, as well as the most fun, so why deny that to a game?
(because shifts in scale would not work in MP unless someone comes up with a superb explanation how a fleet of 60000 super planet destroyer ships loses against a single star-base...)...
Shifts in scale don't necessarily mean the computer has to abstract battles out to just numbers and dice rolls. If you are playing a turn based game there is a boatload of processing power at your disposal when it's time to compute turns. Use that, and actually simulate battles. That should result in no weird results.
or how someone playing on the "build 1 ship at a time" layer has any chance against an empire that builds 5 thousand ships at a time.
Well, they don't. They fell too far behind the economic curve and lost. I don't see the problem here.
2a) Complex SP campaign structure that incorporates said shifts in abstraction layers with a proper dynamic story. Complex long-term consequences of choices (particularly with alien species that you meet)
Yes please. This would go hand in hand with proper diplomacy. Peace treaties and war declarations are not simple matters, and should be detailed and fun to work through.
2b) Hero Character that represents the player. Levels up over games. Vital because.. nothing would be more epic than standing on the flagship as YOU in hard-core mode (dead is dead) in the final battle.
Heck yes. This would go hand in hand with the story.
3) Complex combat system, somehow abstracted of course, that takes into account admirals with many personal perks and traits (Total war style) for large fleets sub-commanders and for super large fleets a uber-commander. Maybe even the emperor (or Minister of Warfare) himself. (with minor chance to lose them all ,p). See -> http://www.youtube.com/watch?v=JaPl4DSFUQI for how to do it. ENTIRE Fleets are commanded in largest abstraction layers, multiple fleets = multiple units. Facing, weapons, special attacks or tactics....
Yesssss. I want this.
Also I want that game. Wish I knew Japanese.
4) Proper border system
I assume this is for it's interaction with civilians and trade and pirates and the like?
planet invasion/defense
Indeed this needs some focus. Taking a planet is no small task in well-written Scifi.
So many games skip the whole ground fighting thing and just Glass planets (and often easily recolonize them). Glassing is not a good thing. Even in the Formic War (Enders Game) they only glassed (well, worse) one planet. The rest they fought tooth and nail for, because we wanted to colonize them. With mortal beings, usable planets are a limited resource in the universe, and not to be wasted easily.
Conquering a planet of say, 10 million civilians doesn't require 20 million troops. It does probably require a couple hundred thousand though, and will require continuing troop presence for a long while.
It's far, far cheaper than having to replace all those people and infrastructure though. Incentivising this is the key I think.
5) Intelligent research system that simulates that development of technology is not linear.
Not sure exactly how such a research system would be shaped. Some elaboration would be cool. I wonder if your idea would fit with the story idea I'll put below.
--
So, you want your 4X to have story? Heres the first concept off the top of my head that could easily kick off a 4X.
Earth, 20whatever. World War IV (III was the Cold War) has concluded. Luckily most of the world had put up missile defense systems by this point, so it was a conventional war with only a couple nuke events. The countries of the world are all beat up pretty badly. All the continents saw ground fighting, and a lot of infrastructure has been destroyed. There is a lot of rebuilding to do.
Luckily the good guys won though, as congratulations player: you are Prime Minister of the new planetary government.
Day one of your rein? Not exciting.
Day two? Exciting. Aliens drop by and give you a nice book as congratulations for uniting your world.
Oh wait, did I say book? I meant Library. As in the whole thing.
Earth now has access to the entire Galactic Library. Where do your scientists get started though? It will take millions of man hours to go through the thing.
And as all technology goes, actually
implementing knowledge is the hard part, and the Library is distinctly lacking in Blueprints and Patent diagrams.
Except for a few things. There DO seem to be diagrams for efficient interplanetary engines, and there is a whole section on FTL tech that the scientists already seem to be drooling over. They seem to think they can have us at Alpha Centauri within a year or two.
And go there we must, as it turns out our Solar System is short on certain heavy elements needed for most of the stuff in this Library.
Space will never know what hit it.
--
Exploring and controlling access to the Galactic Library would make the tech tree something unique I think.
Thoughts?
