Generally speaking, what we do is manage things as objects -- custom classes -- in our own code, for the sake of clarity and readability. Arrange this in proper OOP fashion so that it is easy to maintain and work with in code. Ignore saving considerations for that.
Now have a "serialization" function on each object that you need to save. This basically looks at the relevant fields that it wants to save from that object (only the fields that are persistent; there may be 300% more working variables that do not need to be saved). And then it saves those variables as simple types, no problem. Then there needs to be a deserialization method, and it just creates the object and sets the fields from that.
There are two basic approaches to serialization, each with pros and cons:
String serialization:
Convert everything into big strings, with field names and then a delimiter and then the value, and then newlines (or whatever) to separate fields. And something else to separate objects, etc. There are a variety of ways to organize this in terms of how it flows in and out through serialization. This approach is flexible and easy to manage old and new versions. When you are parsing this data in deserialization, you just look at the field name flags and do whatever you want. If you want to stop saving a field in a future version, you just have it ignore those field tags. If you want to have it load an old version that is missing some field tags, that will work just fine and use default values, no problem.
We use this approach for our settings files in our games. They are small and don't need to be efficient. But they do need to be hand-editable, which is another benefit of this approach. You can just crack open a settings file and make changes, which is really nice.
Binary serialization:
There are varying degrees of complexity you can do here. But basically the gist is that order of operations is IMPERATIVE. You know that you start with X operation (like saving out or loading in the levels). The first item is perhaps a count of the number of levels. It then calls the deserialize-level method X number of times, where X is that first integer you read in. Then it does the same thing for your other data types, etc.
This is more brittle, but smaller and quite powerful in general. You can also make it backward-compatible by including an "overall serialization version" as actually the first thing you serialize and deserialize. And then every time you make a serialization schema change, you increase that serialization version. That way you can have deserialization branches and conditionals based on the serialization version that it reads in, which retains your flexibility.
This type of serialization requires a lot of diligence, but it's what we use on all our main data for the Valley games, Skyward, and Bionic. We might use it for AI War now, I can't recall.
Bottom line
In-engine object organization and the serialization/deserialization of those objects are two separate considerations. Treat them as such, and you'll get the best of both worlds when you do each specific sort of action.
And I can sympathize on being frustrated with your earlier designs. It takes a while to find something you love. In 2002 I did one game, then got to a point where it was just interminable, and I trashed the engine and started over in 2003. Then that became the basis of Alden Ridge, but then I trashed that three times in 2008 and made major overhauls. Then that became the basis of AI War, which then evolved a number of times and trashed a lot of older concepts. Then Tidalis did a lot of the same, and those ideas got retrofitted into AI War. Then the same with Valley 1, and those also got retrofitted into AI War (not Tidalis though). And so on.
Engine work is always a work in progress, I think, heh.
