Author Topic: NAT Punchthrough discussion - and request for programming help.  (Read 10294 times)

Offline Admiral

  • Hero Member
  • *****
  • Posts: 547
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #15 on: January 23, 2018, 08:47:39 pm »
I came by to contribute to this, but noticed that ahferroin7 already mentioned many of the things I was going to say, especially as regards UPnP and NAT-PMP. Download the source code to a BitTorrent client (e.g., Transmission) to see practical and extensive use of these protocols. Of course, I have to imagine there are almost certainly C# libraries that you can use in Unity for these purposes.

That said, I also once built my own "MRUDP" protocol on top of UDP in the past; a "More Reliable UDP." This was back in the 90s for streaming financial data over (extraordinarily slow) CDPD links to mobile devices more reliably (in the face of packet loss and latency) than TCP could do. There have been numerous published protocols as well in the interim, including Google's recent QUIC. Here's an article from some years ago which provides an overview of several. Targeted searching will of course reveal others. http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/Reliable-UDP-(RUDP)-The-Next-Big-Streaming-Protocol-85316.aspx

Here's one that targets Mono/.NET and also includes setting up holes in NAT routers: https://github.com/RevenantX/LiteNetLib

Anyway, now that you know what to look for, I'm sure you will find lots of suitable things without having to reinvent this wheel.

Cheers,

Doug

Offline Qoojo

  • Newbie Mark III
  • *
  • Posts: 27
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #16 on: January 23, 2018, 08:49:05 pm »
You can send a binary frame.
Yep, that's what I'm doing. As raw as I could get it.

We're not using any network-object synchronization. Sync is achieved by the simulations running identically and deterministically on each machine independently of each other, and everybody being sent a copy of each player's commands along with instructions on when to apply each command (so they all do it at the same point in the simulation).

Excellent choice. Layering the data I see it like this

Game Data -> Raw Data -> Send over Socket -> Receive Raw Data -> Unpack/Decipher -> Update UI in multithreaded manner

Then you have to consider fault tolerance. You can't expect every packet to reach destination. But some data must always reach destination like commands to move object to x, y or attack, etc... This is a difficult problem.

Are you going to have the host PC be the hub and control everything? Like client user interacts with object, then command is sent to host PC, then game data is updated by host and sent out to everyone? I see you answered this somewhat in quote, but seems like there could be situations where client is acting on objects that do not exist.

From what I googled, I don't think you have to worry about big endian vs little endian, and using host/network byte order.

I think you have a good grasp of all this already, and your question or request for help wasn't really in this area :)
« Last Edit: January 23, 2018, 08:54:48 pm by Qoojo »

Offline Admiral

  • Hero Member
  • *****
  • Posts: 547
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #17 on: January 23, 2018, 08:51:45 pm »
We're not using any network-object synchronization. Sync is achieved by the simulations running identically and deterministically on each machine independently of each other, and everybody being sent a copy of each player's commands along with instructions on when to apply each command (so they all do it at the same point in the simulation).

As I recall, this is how AI War (the First) worked.

Cheers,

Doug

Offline Chthon

  • Sr. Member Mark II
  • ****
  • Posts: 398
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #18 on: January 23, 2018, 08:58:04 pm »
Well, I'm currently learning database access at work through JDBC and TomCat, and likely will be learning antagonistic neural networking for my Senior Design project. I'd love to learn how the low level stuff actually works from a programmatic standpoint, and unfortunately I can't learn from either TomCat or Jetty because they don't provide the source code for me to peruse. It's endlessly infuriating. :(

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #19 on: January 23, 2018, 09:01:49 pm »
Doug!  Always great to see you. :D

Here's one that targets Mono/.NET and also includes setting up holes in NAT routers: https://github.com/RevenantX/LiteNetLib

Well... holy smokes on this one.  This one is basically implementing everything we'd need, from the looks of it.  It's very much like what I recall of Lidgren, except this looks a bit more evolved (to say the least).  I'm quite happy to see the multicasting on there for LAN discovery.  Dual mode for IPV6 is also of course very nice, since that's likely to (any year now, sigh) become more common.  Basically running in "reliable mode" over UDP and then passing data of our own choosing is what we're doing.  To some extent, that makes the network library itself a bit interchangeable.

Keith -- what might be nice is to migrate all the FORGE stuff and our networking code that goes with it out into the open-source moddable part of our code.  All of that is open source now, anyhow.  Depending on how similar the system above is, perhaps you or someone else could program that in as an option.

What might ultimately be a nice thing is a bit of an abstraction layer where we can swap out the underlying networking layer by just having it implement various interfaces or similar.  That way in theory we could have a FORGE implementation and a LiteNetLib implementation, and the user can switch between them.  Not useful in the long-term, probably, but in the next few months it would allow for some A:B performance testing in an easy fashion.  Lidgren is also open source and has evolved in the past years, so it might also work as a third option to check on.

I imagine we have some folks here who would happily profile the transport layer for us if we had some options on that, while the actual "what we're sending via the transport layer" stays in its own logic area.

Basically kind of like how we can easily switch between DirectX and OpenGL and Vulkan and nobody notices a difference in actual functionality.  Those in turn are running on top of variable drivers, etc.  In our case, FORGE or LiteNetLib or Lidgren would be analogous to DirectX or OpenGL.  The data we're passing in and getting out should be identical in all the cases, and these libraries tend to have a pretty similar setup for the bare-metal type approach that we take to things.

@Qoojo: Yep, endianness doesn't matter to us, partly because we're all in managed code and that abstracts that bit away from us.  And yes, the host is the global manager for all things, and the clients talk to the host, which is also a client of itself.  Clients request for things to happen, and then the host says "do this in future simulation step X" to all of them, and it happens on that step. 

All of that logic is well above the transport layer, and pretty transport-layer agnostic.  Even how Keith is serializing those commands into bytes is transport-layer-agnostic, because he has his own way of encoding things that is a lot more efficient than ascii, or even something like just raw int32s.  We did a writeup with some table comparisons in an earlier design doc, and it's referred to as "BuzzSawBinary."  It's actually smaller than gzip compressed data in a lot of cases, surprisingly.

Our problems are solely of the "hey I need to talk to you and I need you to talk back to me" variety with clients and hosts, not about knowing what to say after that's established.  For customers, one of the big problems was always "who am I?  who are you?  why is there barbed wire between us??" ;)
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #20 on: January 23, 2018, 09:04:08 pm »
We're not using any network-object synchronization. Sync is achieved by the simulations running identically and deterministically on each machine independently of each other, and everybody being sent a copy of each player's commands along with instructions on when to apply each command (so they all do it at the same point in the simulation).

As I recall, this is how AI War (the First) worked.

Yep!  And our current networking already works fine, just like that, but with a different transport layer library.  The NAT punchthrough and potentially multicasting are the main hiccups.

My original implementation was based on this paper: https://www.scribd.com/document/43559444/1500-archers-on-28-8k

The same principles are in place in most every RTS game with any substantial number of units to this day.  Action games are a whole other beast, and vastly more complicated.
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Chthon

  • Sr. Member Mark II
  • ****
  • Posts: 398
Re: Neural network AIs?
« Reply #21 on: January 23, 2018, 09:20:23 pm »
Well, the cool thing is that the AI is all open-source, so anyone is free to add this sort of thing as a mod.  And if it works well and people prefer it, we can bake it right on into the official builds.  As an AI type or otherwise.

As it stands, right now our priority is polishing what we have and reducing our workload, not increasing it if we can avoid it.  We're already massively overbudget and over schedule, and we'd like for this to release in as pristine a state as we can.  Keeping away from scope creep at this point is the big lesson for us in terms of trying for any kind of cleanness at 1.0.

But the good news is that we've built an overall engine that is so darn flexible that we can start contemplating things like this post-1.0 if the game is selling well enough; and if it's not, modders and similar can contemplate it if they would like to.
Well that is definately interesting. It would only be part of the AI, and I think it'd be cool to have it learn your habits and how you play, and surprise you as you play.

Only problem is I found out my computer is far too old to run Tensor Flow on writing recognition. It ends up exiting out while training with no message as to why. I expect I'm missing some instructions needed for that build of Tensor Flow.

Maybe I'll get on it if I take an AI course next Fall, or if I get the project I want in Senior Design.

Offline Qoojo

  • Newbie Mark III
  • *
  • Posts: 27
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #22 on: January 23, 2018, 09:28:39 pm »
@Qoojo: Yep, endianness doesn't matter to us, partly because we're all in managed code and that abstracts that bit away from us.  And yes, the host is the global manager for all things, and the clients talk to the host, which is also a client of itself.  Clients request for things to happen, and then the host says "do this in future simulation step X" to all of them, and it happens on that step. 

All of that logic is well above the transport layer, and pretty transport-layer agnostic.  Even how Keith is serializing those commands into bytes is transport-layer-agnostic, because he has his own way of encoding things that is a lot more efficient than ascii, or even something like just raw int32s.  We did a writeup with some table comparisons in an earlier design doc, and it's referred to as "BuzzSawBinary."  It's actually smaller than gzip compressed data in a lot of cases, surprisingly.

Our problems are solely of the "hey I need to talk to you and I need you to talk back to me" variety with clients and hosts, not about knowing what to say after that's established.  For customers, one of the big problems was always "who am I?  who are you?  why is there barbed wire between us??" ;)

Oh nice. Yea, if you don't repeat data, then compression algorithms can have a difficult time. ;)

Looks like you have the barbed wire solved with these libraries.

Offline Qoojo

  • Newbie Mark III
  • *
  • Posts: 27
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #23 on: January 23, 2018, 09:33:47 pm »
My original implementation was based on this paper: https://www.scribd.com/document/43559444/1500-archers-on-28-8k

I guess we know the min specs now, "Support a target platform of: 16Mb Pentium 90 with a 28.8 modem".

I will over that link later. Interesting stuff.

Offline Admiral

  • Hero Member
  • *****
  • Posts: 547
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #24 on: January 23, 2018, 09:40:16 pm »
I can't learn from either TomCat or Jetty because they don't provide the source code for me to peruse.

Tomcat has been an Apache project for... decades, and open source: https://github.com/apache/tomcat . I used it for the Space Empires IV Play By Web in the late '90s.
 (https://web.archive.org/web/20010626023644fw_/http://seiv.pbw.cc/)

Likewise, Jetty is also an open source project. I use it to this day (as part of my Clojure web stack). Code: https://github.com/eclipse/jetty.project

Enjoy,

Doug

Offline Chthon

  • Sr. Member Mark II
  • ****
  • Posts: 398
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #25 on: January 23, 2018, 09:46:48 pm »
I can't learn from either TomCat or Jetty because they don't provide the source code for me to peruse.

Tomcat has been an Apache project for... decades, and open source: https://github.com/apache/tomcat . I used it for the Space Empires IV Play By Web in the late '90s.
 (https://web.archive.org/web/20010626023644fw_/http://seiv.pbw.cc/)

Likewise, Jetty is also an open source project. I use it to this day (as part of my Clojure web stack). Code: https://github.com/eclipse/jetty.project

Enjoy,

Doug
Thanks, but when I have maven pull in the project, it doesn't have the source files with it, which won't let me F3 through them in Eclipse.

Maybe if I find time to download the binaries and then tell Eclipse where to find the source.

Also, I swore TomCat was Oracle, as every time i'm getting the Oracle license agreement splash like on every other Java file.

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #26 on: January 23, 2018, 10:09:07 pm »
My original implementation was based on this paper: https://www.scribd.com/document/43559444/1500-archers-on-28-8k

I guess we know the min specs now, "Support a target platform of: 16Mb Pentium 90 with a 28.8 modem".

I will over that link later. Interesting stuff.

Ahaha.  That was Age of Empires 1 -- we're a bit beyond that, at this point. :D
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!