Author Topic: NAT Punchthrough discussion - and request for programming help.  (Read 2009 times)

Offline x4000

  • Chris Park, Arcen Games Founder and Lead Designer
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,237
NAT Punchthrough discussion - and request for programming help.
« on: January 23, 2018, 12:13:28 PM »
I figured I'd make this discussion on the forums so that it's not just a discussion between Keith, Badger, and myself via email.

Badger notes that we really should have NAT punchthrough prior to starting the beta, because that's a big pain in the rear for getting multiplayer set up otherwise.  I've noted in the past (prior to the kickstarter) that this was a possibility, but not one we've fully investigated yet, and it's something potentially very time-consuming as well as something with potential ongoing costs.

Basically, the way that NAT punchthrough works, to my knowledge, is via a few methods, which are similar to the private VPNs like Hamachi and whatnot:

1. There's some sort of server hosted somewhere, by Arcen or otherwise, and it keeps track of "game hosts" that say they want other people to be able to connect to them.  So, for instance, Badger fires up the game and says he wants to connect.

2. Badger's friends then try to connect to him, via that relay server.  At that point the relay server tries to broker a direct connection between Badger and his friends.  If it's successful, then that's like getting a green dot in Hamachi.  Basically there's no middleman, and because of the respective firewall and router configurations on both ends, things are all happy.

3. But it's also possible that even with the central server, the routers are not going to allow for this type of direct connection.  There all manner of router and firewall setups, and there's no real centralized way to keep track of that stuff.  So what happens in this case?  Well, Badger can talk to the central relay server, and so can his friends, so they can pass messages to one another by talking to the relay and having it... well, relay... those messages on.  This is the "yellow dot" situation in Hamachi.

As you can imagine, this introduces a lot of lag, but it ALSO means that all the bandwidth of the gameplay for Badger and his friends is now costing Arcen money directly while he's playing.  We can't afford that sort of thing, and this is also the sort of thing that leads to game servers being shut down a few years down the road by AAA companies, and then everyone complaining.

The really large publishers maintain somewhat game-agnostic relay servers, as I understand it, so if they have 2 people playing a game they published 10 years ago, that's just a drop in the bucket on top of the 200k people who are playing their latest titles.  Wheee.  Steam also maintains relay servers and similar, but we don't use Steam networking, nor do we have any intention of doing so -- that locks us to Steam.

Remember my commentary about the "yellow dot" scenario, above?  Well, in those circumstances ALL of the traffic is being routed through Steam instead of going through Forge Networking or similar on our end.  This is... suboptimal.  What about DRM-free copies of the game, for one thing?  Also, Steam's networking is... not my first choice, for a variety of reasons.

Right now we're using a customized version of Forge Networking Remastered (https://github.com/BeardedManStudios/ForgeNetworkingRemastered), which we bought (https://www.assetstore.unity3d.com/en/#!/content/38344) but which is now apparently free and open source.  They have a version of NAT punchthrough, but I don't know how robust it is: http://docs.forgepowered.com/nat-hole-punching/

There are other NAT Traversal solutions, such as this one, that are generalized, but mainly geared toward UNET: https://www.assetstore.unity3d.com/en/#!/content/58948

UNET is something we've found to be generally bloated and unreliable, so that wasn't something we wanted to go with.  I think that the solution above could be made to work with FORGE, but I'm not sure it would be any better than the stuff built into FORGE.  And in ALL of those cases, I'm not sure that this would be zero-cost to Arcen, which is the only way we can guarantee that 20 years from now the networking will still work.  We're small!

With that in mind, I'm open to solutions and suggestions.  There are various personal VPNs that might have APIs that could be integrated, for instance.  Ideally whatever is going on is invisible to the FORGE networking layer (it just thinks it's talking directly to the clients, in other words), like was the case for Hamachi and Wippien and similar.

Next we come to what is essentially a manpower issue:

Now that FORGE is open source, if there are any programmers in the audience here who would like to take a stab at seeing what they can get working with it, and if they can improve things over what Keith has had time to do, then it's certainly worth an attempt at least.  We're shortstaffed in this area, and I think that this particular feature is pretty much just not going to happen (I have a gut feel) if someone external isn't able to volunteer their time and expertise.  Personally I'd rather have Keith focusing on the game itself, and the AI and whatnot, and other things he's more directly suited to.
 
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Boost

  • Newbie
  • *
  • Posts: 1
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #1 on: January 23, 2018, 01:22:41 PM »
Hi,

I found this thread through the kickstarter project update message and I'm very interested in doing networked games so I thought I'd reply.
I haven't actually done a full networked game, yet, but I've done basic object synchronization and messaging. I've been more working on the game engine side more ,
and I've been distracted by other projects.
But as I've been interested in networking I've also read about NAT quite a bit.

So I haven't actually tried this approach in practice, but from what I've read the best chance of success for NAT punchthrough (without needing a server for constantly relaying messages) is to have a server where the game hosts register and clients can then ask the relay to relay a message to the server to initiate the punchthrough. Then the client and the server use the same socket (so that the port is the same, this increases the chance of success as some firewalls are really strict) that they used to send a message to the relay server and send each other 10 UDP packets  with 1 second intervals. I'm not 100% sure that this would also work with TCP connections, if both sides try to open a connection simultaneously, but it has a good chance to work. This way the firewalls on both sides should see incoming and outgoing packets to the same peer and allow the communication. This would be very light on server resources as it would only need to receive registrations (and optionally keepalive messages) from the game host and relay one request from a (verified) client to the server per connection attempt.

The approach of ForgeNetworkingRemastered from a quick look seemed that they won't use the same socket (or that they don't have to, not sure which one is the case as I didn't dig into the NAT client side) and basically the server is told that client from their specified port wants to connect (in a json message) and it then handles that somehow (perhaps by sending some sort of message to the client). At least from this quick look it seems that depending on the server implementation this may be the exact same approach.

The approach I described should work for basically any type of firewall (even ones that only allow return packets from the exact ip and port combination) to establish connection between two clients. Of course this needs some additional tweaking if all clients need to communicate with each other and some of them are behind the same NAT so that they see each others' private address instead of the public address, but this shouldn't be impossible to get working.


Offline ahferroin7

  • Newbie
  • *
  • Posts: 1
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #2 on: January 23, 2018, 01:39:48 PM »
OK, so some random musings on this from a netops guy who's done more than his share of raging at poorly implemented network multiplayer systems, with the disclaimer that I've not looked very far beyond Forge's public API documentation (which conveniently leaves out a lot of the actually useful info on what it's actually doing):

There is no single silver bullet that will make this work, period.  Luckily, you appear to already understand that given the content of your post.  There are however a handful of things you can do to make this work in almost all cases and make it easy for users to fix when it doesn't:
  • STUN (https://en.wikipedia.org/wiki/STUN) and TURN (https://en.wikipedia.org/wiki/Traversal_Using_Relays_around_NAT) will cover a vast majority of cases.  They work pretty much like you described in your explanation of NAT hole-punching.  Most likely, Forge uses some variant of these.
  • Provide a way for the user to specify an arbitrary, fixed port to use for the listening socket.  This sounds stupid and rather useless to a lot of people at first,
     until you realize that a lot of consumer routers make you jump through hoops to expose ports externally, and it's often easier to just map one and do all your multiplayer stuff through that regardless of game.  Minecraft is actually a great example of this being done right, when you go to host something locally, you can manually specify a port to host it on.  This bit is largely independent of the networking layer you use under the hood.
  • UPnP and NAT-PMP will help cover a vast majority of cases where STUN and TURN do not work.  While those of us who are tech savvy and care about security over convenience turn these functions off on our routers, most people don't, and supporting them will make a direct connection just work in a vast majority of cases.  I'm not certain from looking at the Forge documentation whether they have this or not.
  • Please, for the love of all things holy, allow local multiplayer.  This will allow people to reduce the load on any kind of central server, allow multiplayer even if you eventually have to shut down the central server, and probably most significantly, let people have real LAN parties with the game.
  • Make the central server component (relaying if you're going to actually do that, or just discovery if you won't do relaying) available for regular users to run. This serves two purposes, first you remove the absolute dependency on you hosting a server for multiplayer, and second you open up the option of a community supported relay pool (which will save you money).  Take a look at https://syncthing.net/ for an example of this particular aspect done right, pretty much their entire relay network is community hosted, and they provide an easy option for users to run their own private relays.
  • Don't use Hamachi as a model for how to do things under the hood.  They did a great job on the end-user experience, but did a lot of things below that wonderful UX that made those of us who have to handle actual network infrastructure hate them with a fiery passion.  In particular, do not try to pull crap like they did for indirect relaying (they essentially hijacked a publicly routable but infrequently used block of IPv4 addresses that they did not own).
  • Make sure you have proper IPv6 support.  Like UPnP and NAT-PMP, this will get you a lot closer to things just working in most circumstances (if two players are on real IPv6 networks, they are likely to just need the central server for discovery and nothing more).

Now, if you're just going for the bare minimum to get NAT traversal working, then I would suggest specifically focusing on something similar to STUN and/or TURN, plus UPnP and NAT-PMP and IPv6 support.  Combined, that should make things work in at least 90% of cases, and in most of the cases it doesn't it should just require a trivial manual port-mapping on the user's NAT router.

Offline x4000

  • Chris Park, Arcen Games Founder and Lead Designer
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,237
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #3 on: January 23, 2018, 04:10:24 PM »
Holy smokes, this is more complex than I'd even expected.  We're really not network programmers in that particular depth.  We also don't have any central servers that would be good for this, at least not at the moment.  This has to be a relatively common problem to the point that just doing the STUN-style initial brokering of a connection is available in some sort of organized middleware?

I believe we're using just a single port (UDP) with Forge, but I don't know for sure; we were with Lidgren, which was my project to implement.  But Keith implemented FORGE and customized it a bunch when he did, so I don't know where we stand on that precisely.  And yes, definitely local multiplayer is always something that we prefer above all else!

The sort of work that you're describing definitely sounds out of scope for what we currently have on our plate in terms of time and funding, so I'd be very interested in anything that can help us cut those hours down.  Whether that's some form of middleware or hosted server system for this, or whether that's someone willing to donate some time showing us how it's done from a programming level.
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline The Tycooner

  • Newbie
  • *
  • Posts: 2
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #4 on: January 23, 2018, 04:22:25 PM »
As long as the option to swap port (for those who know how to setup a router), UPnP (for those who don't) and direct connect by IP exists, you don't really have to bother too much with this, just add the port, local IP and outwards IP to the hosting screen (in a copyable fashion) and most people should be able to host and connect with that information.

I'm not really seeing AI War as a game where you just pick a random game in a list and play for hours on end with randoms, and while I do realise there is a demand for that kind of thing too, that's where communities and other social interactables comes into the picture.

Offline x4000

  • Chris Park, Arcen Games Founder and Lead Designer
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,237
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #5 on: January 23, 2018, 04:29:44 PM »
With the original AI War, we had the ability to choose your port, but it was set to something specific to start with.  I'm not sure what you have in mind with UPnP; I've not heard that term used in this specific context before.  Being able to get their local IP or their global IP is definitely something I want to put in directly, which we didn't have previously.  The one problem with local IPs is that usually people have multiple adapters these days, and those can give strange results.

People have infinitely been asking for a global place you can go play with random people, but I'm not inclined to do a lobby like that.  It would perpetually be a ghost town, because people are either already playing or not, for the most part; the matches are too long.  But for people who don't know how to set up their router, the ability for them to connect to a friend without having to set up their router is really all I'm looking for.  Generally my understanding is that that's NAT punchthrough -- but is there something related to UPnP that serves a similar function...?
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #6 on: January 23, 2018, 04:42:41 PM »
I believe we're using just a single port (UDP) with Forge, but I don't know for sure; we were with Lidgren, which was my project to implement.  But Keith implemented FORGE and customized it a bunch when he did, so I don't know where we stand on that precisely.
Single UDP port :)
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Qoojo

  • Newbie Mark III
  • *
  • Posts: 27
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #7 on: January 23, 2018, 04:45:11 PM »
I have a lot of experience coding TCP sockets on linux and windows for 20+ years in C++/Java/C#. Never really needed UDP.

Forge seems pretty standard. You can send a binary frame. The connection stuff looks the same, and "Setting the correct NAT host in the Connect method of the ServerUDP and ClientUDP is all you will need to do in order to enable NAT hole punching in your application." So if the hardware is setup properly, it appears you can use this library like a normal socket library. Just have to figure out the specific of server and client specification. I am at work, so I cannot watch the NAT Hole Punching specific video, and they did not list the code.

With that said, looking briefly at the Forge documentation, I am not sure NetworkObject will handle the data load given the immense number of ships this game has without serious performance issues. Considering AI War has thousands of ships, and the network object seems to be used to sync between clients, including RPC calls for things like exploding, I think the first step is to forget the hardware and NAT Hole Punching, and attempt a proof of concept project.

Check out how it performs with all the bells and whistles in the best situation, on a Lan. Then introduce the NAT hole punching, then full remote. If you find an issue with that, then you are going to have to go full binary or a mixture of the two.

These are my thoughts with 15-20 minutes looking at the docs, so take them with a grain of salt.  ;)

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #8 on: January 23, 2018, 04:55:29 PM »
You can send a binary frame.
Yep, that's what I'm doing. As raw as I could get it.

We're not using any network-object synchronization. Sync is achieved by the simulations running identically and deterministically on each machine independently of each other, and everybody being sent a copy of each player's commands along with instructions on when to apply each command (so they all do it at the same point in the simulation).
« Last Edit: January 23, 2018, 04:57:15 PM by keith.lamothe »
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline x4000

  • Chris Park, Arcen Games Founder and Lead Designer
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,237
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #9 on: January 23, 2018, 05:05:46 PM »
Keith, it appears I have thrown you into a thread with a bunch of people who know a lot more about this than I do.  Hopefully this yields something constructive, and in the meantime I'll yield the thread to you. ;)  Back to the GPU I scurry!
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #10 on: January 23, 2018, 07:20:34 PM »
Quote
As long as the option to swap port (for those who know how to setup a router)
Already in settings, but can also expose on MP screen.

Quote
UPnP (for those who don't)
How do we enable this?

Quote
and direct connect by IP
That's already in.

Quote
just add the port, local IP
Easily done.

Quote
and outwards IP
Would a quick request to icanhazip.com, sanitization of the result, and output of same suffice?

Quote
to the hosting screen (in a copyable fashion)
Copy the whole chunk, or just the outwards IP, or what?


Thanks very much for the advice :)
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Chthon

  • Sr. Member Mark II
  • ****
  • Posts: 398
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #11 on: January 23, 2018, 07:40:41 PM »
Sorry I haven't been around for a while college and life suddenly got serious.

In my work and in my classes I've had the opportunity to become familiar with some networking, but I'm not exactly sure how to do NAT punch-through myself. Instead I've been working using websockets, which act like a file you just write to another computer somewhere out on the web. Websockets literally handle all of that for me.

Is this a possible protocol to use in AI War 2, or are you guys still using REST or SOAP?

Implementing websockets for you would be a little bit tough for me right now, as I'm mostly using Jetty or TomCat to perform them. Jetty is the Eclipse native libarary for Java, and TomCat is the Java native library. I don't know the library I would use in C#.

One of the features I believe that Websockets takes advantage of is IPv6 addressing, which would eliminate the entire need for NAT punch-through. After all, if we can address every atom in our galaxy, what's the chances that 2 computers are going to have the same address, right?
« Last Edit: January 23, 2018, 08:26:30 PM by Chthon »

Offline x4000

  • Chris Park, Arcen Games Founder and Lead Designer
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,237
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #12 on: January 23, 2018, 08:06:18 PM »
We can't use REST, SOAP, or even WebSockets -- our data has to transfer far faster than that.  Basically we need data to go out and come back within  about 100-200ms at most, and none of those protocols are set up for doing that.  We also need the data to go as leanly as possible, because the more packets that are sent, the more you get data loss and resends, etc. 

So it has to be a protocol built above UDP (because TCP stinks), where it keeps messages even if they are out of order or one gets dropped, and just requests the missing ones, etc.  FORGE and Lidgren both basically provide a framework like that, as do things like UNET and most other game-based systems.  TCP is a good 4x to 8x to slow at minimum, and the more packet loss there is, the worse it gets, as well as the larger the data, the worse it gets.

But it was a thought!  Just letting you know the parameters of where we are.
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Chthon

  • Sr. Member Mark II
  • ****
  • Posts: 398
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #13 on: January 23, 2018, 08:26:13 PM »
Well, I'm familiar with the concept of UDP, but not any UDP protocols. Thought websockets was leaner than that though, as I said, it's like writing to a file. It's you who determines what information goes into the file. Other than the headers on the packets of course.

What about my suggestion of IPv6? Is that possible over UDP? If so then you are addressing computers directly behind routers.

Offline x4000

  • Chris Park, Arcen Games Founder and Lead Designer
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,237
Re: NAT Punchthrough discussion - and request for programming help.
« Reply #14 on: January 23, 2018, 08:45:12 PM »
Well, in terms of IPv6, support for that is somewhat implicit in the modern libraries (to my knowledge), but we don't get to decide how clients are set up.  I wish! ;)  Clients and hosts have their IPs set by their ISPs.  We're talking about your computer talking to your friend's computer.  I don't know what your setup is, or who you are, or what year it is, and I certainly don't get to touch your configuration.  This isn't a server I'm setting up, or new drivers I'm installing on your machine, etc.

UDP and TCP are basically just the two lowest-level protocols that you can get to.  Websockets and all the rest of those are implementations on top of TCP, IIRC.  There's a whole stack of stuff that has to be stripped away for games to work, basically.
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!