Author Topic: What I'm up to (unity upgrade!)  (Read 2744 times)

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
What I'm up to (unity upgrade!)
« on: April 24, 2019, 04:26:12 pm »
Some release notes-in-progress:

* Upgraded the game to Unity 2019.1.0f2!  The prior version was on 2018.2.0 or something along those lines, so this is a big jump.
** The really big improvement is time-sliced garbage collection: https://blogs.unity3d.com/2018/11/26/feature-preview-incremental-garbage-collection/
*** Or here: https://www.youtube.com/watch?v=XhfzzLyG7_w
** This also has numerous bugfixes, and lets us upgrade from a VERY old version of TextMeshPro to the newest version, which in turn fixes a variety of bugs relating to text, blurry text, textboxes, etc.
*** That said, as part of this, we knew this was going to happen, but it breaks ALL the references to any sort of text-based thing inside of any of our unity projects.
*** But we kind of needed to go ahead and have this done before we started in o na bunch more UI updates or else it was just going to be creating even more work for ourselves in the future.
** There are also a variety of stability improvements to the unity engine in general, and a few performance improvements.  I think that Linux machines will see the biggest boost.
*** There are some new capabilities that we can in theory use in the future, too, and nested prefabs are an amazing thing for sure.  But we're not really needing those bits in this specific project right at the moment.
** Oh and this does upgrade us to the latest version of the post processing stack v2, as well, which means probably better performance and some edge bugfixes for some folks in that area, too.

* Okay, so we've been using the mcs.exe compiler for the last two years, and it turns out that doesn't support the newer versions of C#.  It would have been nice if that had been more clear.
** Instead, when we were updating our TextMeshPro component or a few other things -- or trying to, anyway -- we would instead get mysterious "System.MissingMethodException: Method not found: "Void System.Array.Reverse(!!0[])" compilation errors.  Good luck googling that, almost nobody has that.
** The solution is to STOP USING MCS.EXE, and instead use csc.exe.  That's the new Roslyn compiler that Micrsoft created and unity is using for the newer C# 6.0+ features.
*** The !fun! part about that is that all the syntax is different, and also it requires references to... a whole bunch of dlls that are the "reference dlls" instead of the "runtime ones."
*** If you ever find yourself referencing a dll called System.Private.CoreLib.dll, or anything like that, STOP WHAT YOU ARE DOING and try a different dll.
**** Even though the compiler just told you that what you need is in that dll, it's actually lying.
**** Actually what you need is in... good luck figuring that out!... but probably it's in the REFERENCE version of System.Core.dll or similar.  It has nothing to do with "netcoreapp" or "netstandard," unlike the few non-unity-related references to this on the internet would have you believe.  That's unrelated to unity, and is just a fun thing on some web servers and various other software.
**** What the heck is a "reference dll" in terms of what the compiler needs?  Well, it's a fuller version than is used at runtime, I suppose.  I haven't explicitly seen that said anywhere, but that seems to be the general feeling I'm getting.  Why they needed this split I'm not sure, but I'm guessing different compiler flags or perhaps the ability to have different per-platform versions.  Something like that.
**** Where do you find those reference dlls??  Ha!  Good luck with that.  Some of them seem to be in the Roslyn compiler folder... except not really, some of them have way too small of sizes.  Others are almost certainly in your [UnityInstallLocation]\Editor\Data\Managed\UnityEngine folder, but there again it's not perfect.
***** Generally you're going to find yourself searching around inside the unity install folder, and you might even find yourself (if you're on windows) searching around inside C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\ or whatever your equivalent is.  Nicely packaged for you?  NO.
** So that explains one of the big mysteries that Keith and I have been scratching our heads on for a couple of years now.
*** But even with that "solved," I couldn't get all of the aspects of LINQ working, and some of our third-party networking code uses LINQ, so I moved that into an external dll that will get loaded at runtime.  That's more efficient anyway, but annoyingly indirect.
*** All of the network code has been open-sourced by the original authors anyhow, so that actually will work out extra well for us when we get to that point.  But in the short term it's another pain in the rear we didn't need.

* One bonus of all this upgrading of unity-stuff is that the editor is no longer complaining about our external dlls being something that will crash the editor and so something they refuse to load.  That's... always a plus.  And is related to the whole compiler saga.

* Okay, nested prefabs from the latest versions of unity are actually indeed insanely useful and we're starting to use those more than we otherwise might have.

* ALL of the fonts in the game have been reworked (in terms of how they are imported into TextMeshPro and thus how they look in general).
** And a bunch of new fonts have been added to the game as options for us to use in various contexts.
** And we've completely redone what fonts we are using in almost all contexts as it is.
** There was a lot of complaining about the fonts not feeling properly militaristic, or otherwise being non-thematic, and hopefully this helps.

* The background story is again its own window and scrolls rather than is paginated, but this time is a lot more condensed, uses an improved version of the more-recent rewritten version, and uses strategic coloration to make it easier to read quickly.
BackgroundStory  600x400




------------------

But I still have a fair bit to go through.  This is a HUGE leap forward for us in terms of being able to not have to do this sort of major rework again anytime soon, though.  The unity upgrade had been hindered by the compiler thing that Keith and I couldn't figure out last year.  The textmeshpro upgrade was then further hindered by the fact that I'd have to go in and manually fix every freaking UI.  Good thing you guys hated the fonts, some more than others, so I could hit both with one stone.

In the future, a unity upgrade will still be using the same compiler for the foreseeable future (that was the case for the prior 10 years or so), so those aren't such a big deal.  This game has already had something like 5 unity version upgrades, anyway.

In the future, a textmeshpro upgrade won't be such a big deal because of the new way we're compiling that in general, and the fact that all the GUIDs will match.  So it will just get us the new bugfixes and features without all this other mess.

Anyway, so that's where I'm at right now. ;)
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline RocketAssistedPuffin

  • Arcen Volunteer
  • Sr. Member
  • *****
  • Posts: 260
Re: What I'm up to (unity upgrade!)
« Reply #1 on: April 24, 2019, 06:03:00 pm »
Well I'm prepared on my end...I think.

Reading stuff like this makes me amazed anyone understands it, let alone make multiple games.

Very curious on these new fonts, and the future icons.
Autistic, so apologies for any communication difficulties!

Offline Vinco

  • Jr. Member Mark II
  • **
  • Posts: 80
Re: What I'm up to (unity upgrade!)
« Reply #2 on: April 24, 2019, 09:53:49 pm »
It's fascinating getting a peek behind the curtain like this.  Thanks for sharing!

Offline WeaponMaster

  • Newbie Mark III
  • *
  • Posts: 46
Re: What I'm up to (unity upgrade!)
« Reply #3 on: April 25, 2019, 12:26:41 am »
What would this mean for me, someone who strictly makes changes to the external code dll? I already don't know how to properly compile the game in unity, so I just build universal and core from it and then work from Visual Studio and the Steam copy.

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: What I'm up to (unity upgrade!)
« Reply #4 on: April 25, 2019, 08:10:10 am »
WeaponMaster: building should be exactly the same for you! I'm still even using visual studio 2015 just because I prefer the feel.

There are a bunch of new dlls and files and whatnot, but all of that is already set up for folks like you and Puffin. Hopefully I can do a push today, if not then tomorrow. Today I do have an interruption for most of the day being a chaperone for a field trip for my son, so we'll see how that cuts into it.
« Last Edit: April 25, 2019, 01:03:24 pm by BadgerBadger »
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: What I'm up to (unity upgrade!)
« Reply #5 on: April 26, 2019, 04:44:54 pm »
Today can be summed up so far as:

1. Lots of stuff went really well, although there was a longer list of things-wrong than I expected.

2. I literally had one copy-paste error that broke all networking (including loopback networking which is needed for the game to function at all), and there went 2 hours of my time on that alone.

3. I forgot I made a bunch of additions to dropdowns of TextMeshPro that had to be adapted over.  That was hard.  No wait, easy.  No wait, it was easy but now dropdowns don't work at all (still).  Dang.

4. Man some of these fonts are near impossible to read at certain resolutions.  Doing this on two different monitors was a good idea.

5. Man this stuff is faster to update than ever, now, but there are a bunch of tiny errors that I'm picking out of my fur, so to speak.



We shall see what this means for release timing.  I'm going to see Endgame tonight with my girlfriend (no spoilers!) and I'm super looking forward to that.  Tomorrow we have stuff with kids, but Sunday I'll be working probably half the day.  Yes, girlfriend.  Life has improved a lot.  But divorce after 18 years still has a lot of scars, and it's never a clean break when there are kids involved.  But in the romantic department I landed back on my feet... surprisingly quickly.  It's still early yet, but I get to be a dad almost every day of the week now, which is a pretty darn big deal to me.

Having some of that stuff sorted out a bit has helped my productivity enormously.  Now let's see if I can finish up this current batch of junk.
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Draco18s

  • Resident Velociraptor
  • Core Member Mark V
  • *****
  • Posts: 4,251
Re: What I'm up to (unity upgrade!)
« Reply #6 on: April 26, 2019, 09:38:56 pm »
but there are a bunch of tiny errors that I'm picking out of my fur, so to speak.

Those are burs. ;)

Offline TechSY730

  • Core Member Mark V
  • *****
  • Posts: 4,570
Re: What I'm up to (unity upgrade!)
« Reply #7 on: May 03, 2019, 05:14:07 pm »
Hooray! Glad to see that Unity is finally making actual steps to improve their terrible garbage collector after...way too many years. (How many years has it been?)

Still not as good as a full generational garbage collector, but a definite step in the right direction.

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: What I'm up to (unity upgrade!)
« Reply #8 on: May 03, 2019, 06:08:00 pm »
Still not as good as a full generational garbage collector

They claim it is, which is odd to me, but whatever.
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Draco18s

  • Resident Velociraptor
  • Core Member Mark V
  • *****
  • Posts: 4,251
Re: What I'm up to (unity upgrade!)
« Reply #9 on: May 04, 2019, 12:20:08 pm »
Frankly, not having to think about memory management in general (even if the GC isn't very good) saves me, as a developer, a lot of headache.

Sure, it still rears its ugly head now and again, but I like not having to think about it when I'm programming. Unlike C/C++ where you are required to clean up after yourself every time all the time.

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: What I'm up to (unity upgrade!)
« Reply #10 on: May 04, 2019, 12:33:28 pm »
You'd actually be shocked (or maybe not) how much we have to think about memory management in games like AI War 2, though.  Even in C#.  It's a different sort of thinking, though: not "if I don't think about this, then we have a memory leak," but instead "if I don't think about this, then we have a lag spike more frequently and potentially more severely."

So it still makes things vastly better, but there's a lot of overhead in allocating certain kinds of objects and getting rid of them, and some of them have to be manually disposed anyway.  Which all means that effective pooling (reuse of existing objects which are just invisibly held when not in use) is the optimal strategy, by far, in either style of environment.

Don't get me wrong, I'll still defend C# over C++/C until the day I die, but the benefits of garbage collection are more of a safety net than a complete abdication of responsibility to manage memory well.  If the garbage collector was better, and/or memory was allocated faster, then that might not be true anymore.  So the future of C# is a lot brighter, even.  But that's where I feel things are right now.
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Draco18s

  • Resident Velociraptor
  • Core Member Mark V
  • *****
  • Posts: 4,251
Re: What I'm up to (unity upgrade!)
« Reply #11 on: May 04, 2019, 06:36:49 pm »
I was speaking in the general case. Absolutely there will be times when you need to think about it, but it's more of an "offload this to a good time" rather than having to decide, in the exact moment that an object is deconstructed, how to shift things around in memory to avoid fragmentation.

Offline robdon

  • Newbie
  • *
  • Posts: 1
Re: What I'm up to (unity upgrade!)
« Reply #12 on: May 15, 2019, 09:28:41 am »
Regarding the TextMeshPro problems Chris, we also had problems upgrading a couple of times with it and GUIDs.

Have you seen this post for info with upgrading TMP? https://forum.unity.com/threads/migrating-from-legacy-asset-store-to-package-manager-releases.511748/

That mostly worked for us, but it did miss upgrading some prefabs and I talked about it here: https://forum.unity.com/threads/migrating-from-legacy-asset-store-to-package-manager-releases.511748/page-8#post-3980113

I had to scan through the scenes text '.unity' file and do a replace on some of them that were incorrect, wasn't too bad once I found the problem, but it took a while to understand what was wrong.

That was when we went to version 2081.3.0.f2, soon we're upgrading to 2019.1, so will see how that goes :)

HTHs, Rob.

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: What I'm up to (unity upgrade!)
« Reply #13 on: May 16, 2019, 04:05:30 pm »
Thanks for that, Rob!  I did want to do that sort of automated process, but I was going from the paid version to the Unity official version, and things were just all over the place in terms of GUIDs and also fonts in general being outdated.  Thankfully I was able to go through and manually edit everything, part of it via the little Debug unlock mode for the Inspector, to just reassign the missing scripts as needed.  I had to update all the fonts anyway (well, I was choosing to) as I went, so I just hit all of it at once.  So far so good with that... I think!
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!