Ha. Master of Disaster, perhaps.
And yea, if these objects are all entities for which you want robust physics then Unity objects is probably the way to go with them. At least for now, until you've accomplished your bigger goals. Later on you may prefer doing your own physics, so you have more control over them. But for now it would probably just bog down your project.
That said, if there's any significant chunk of these objects which are not really physical entities or otherwise can be handled with a very simple homebrew physics model (or none at all), then you perhaps could pull them under the umbrella of the camera object's Update function.
As for doing it, it's pretty easy, basically you just define a method on each "non-physics" object for "DoSimStep(float EffectiveDeltaTime)" or something like that, and then in the camera's Update logic you make sure all of those get called. Often it makes sense to nest things inside each other, so you might call World.DoSimStep(effectiveDeltaTime), which then loops over all MetaMapNode's and calls node.DoSimStep(effectiveDeltaTime) and then loops over all RaceState's and calls race.DoSimStep(effectiveDeltaTime), etc. To adapt some examples from what I'm working on currently.