#network programming

From a high level perspective, what rollback does is hide the lag time between when a remote player (my opponent) presses a button and the local player (me) gets that button press notification by skipping animation frames. It will probably help if you read [this old post about delay-based netcode] first in order to understand the kind of problem we're trying to solve first and how a delay-based system works to solve it.

Let's take a look at some sprites from Street Fighter 2 in order to illustrate how different types of netcode work. First, we'll look at the animation frames that a game with zero delay plays starting from when the fierce button is pressed during Ryu's idle animation to the first active frame (when the fierce punch can hit). Let's assume that each of these animation frames takes the exact same amount of time to play (1/60th of a second or roughly 16.66 milliseconds each).

When the button is pressed, it shows the first startup animation frame for 50 milliseconds (3 frames worth of time), then the second 33 milliseconds, finally reaching the first frame that the fierce punch can actually hit an opponent.

A delay-based netcode approach would build in a few frames of input delay in order to give the remote player time to receive the notification. A three-frame input delay would look like this:

The important thing to note here is that the button press info from the remote player can arrive on any of the "DELAY" frames and still fully play the animation on time because the game can figure out what button was pressed on which frame so it knows when to start the animation. The longer the input delay, the more time the game has to make sure everything is synchronized between players. Unfortunately, it also makes things feel bad because it makes the controls feel sluggish to players.

What rollback does is save the entire game state over the last few frames and then insert the remote button press info to retroactively figure out what the world would look like if that button press had arrived on time. So, as an example, if my opponent presses Ryu's fierce punch on frame 0 and it arrives to my game on frame 3, the game would skip the first three frames of startup animation for the fierce punch and jump directly to frame four so that the fierce punch will have its first active frame at the same time on both games, like so:

This guarantees that both players will see the important event (when the fierce punch can actually hit and do damage) at the exact same time without needing to delay the game. However, this is not perfect. Observe how several frames of the startup animation have been skipped entirely for player 2 in this case. Instead of seeing the full fierce animation, player 2 instead sees that Ryu was idling for a few frames and then he throws a fierce punch with a visible two frame startup instead of its usual five frame startup. The larger the rollback window, the weirder it can get. If the button input doesn't arrive until frame 5, for example, there's no startup animation at all and the fierce punch can hit with zero visible startup:

If we have longer rollback windows and animations with short startup times, we can have significant troubles because it means that players can retroactively get hit without ever seeing what hit them until it is too late. Ryu's jab animation has only two frames of startup. If we have a rollback window of three frames or more, it means that I can be retroactively hit by a jab I never saw coming.

Rollback is very good at keeping important events happening at the same time across two different remote games as long as the window isn’t too long. Most of the time, going more than three rollback frames starts causing weird timing issues. However, rollback is not a perfect solution. There are significant gameplay issues that must be accounted for if a game runs with rollback. The biggest issue with being able to retroactively change the game state is that false positive events can trigger. If the game thought that the round was over because I hit my opponent with a killing blow, it normally does something like play a sound and visual effects - the announcer can yell KNOCK OUT, the opponent’s character groans in defeat, the game slows down to show the animation, visual effects play, etc. What do you think happens if that round-ending event gets retroactively changed three frames after it started because my opponent actually hit me during the rollback window and my game didn’t know it? The game has to handle this weird case when it thinks somebody has won the game/round/match and actually hasn’t.

Furthermore, rollback requires significant processing resources. The game must be able to remember the entire state of the game in order to retroactively simulate everything that would have happened if a particular input arrived at a certain time. This isn’t just hit and animation frame data, but the visual effects, lighting, and other details that might affect the game - a retroactive attack might trigger a trap placed by their opponent, which might cause other effects to trigger in order for the game state to be correct. This gets significantly more difficult as you add more players, which is probably the reason that Smash Bros could not add feasibly rollback online play. When there are three remote players that can all potentially cause a visually-intense game to roll back at any time and a huge number of additional permutations that can happen as a result of those retroactive rollbacks, you’re looking at some major potential performance issues (especially on a Switch). As such, rollback really is only important for precision input timing for games that have startup animations within which the lag can be hidden. That’s basically limited to fighting games (that allow two players at most) and not much else.

[Join us on Discord] and/or [Support us on Patreon]

The FANTa Project is being rebooted. [What is the FANTa project?]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Many first person shooters are plagued by a netcode artifact known as the peeker’s advantage. I propose here a technique for correcting this bug, based around normalizing gameplay in such a way that human reflexes and skill decide the outcome of competition (as opposed to network latency or artifacts of varying netcode designs, discussed shortly).

When a player in a first person shooter is moving around the game world they exist in a position on their own game client that is slightly ahead of their position on the server. This is a solution/side-effect of clientside prediction which is ubiquitously used in first person shooters giving the player instantaneous movement and controls that feel like a single player game despite controlling a character that is moving around on a remote server. 

If we were to visualize the difference between the clientside player position and the serverside player position it would look like two characters chasing each other. How far apart the two characters are depends (in descending order of usual importance) on how fast movement is in the game, the latency of the player, and the tickrate of the server. But how big are these differences in the actual games of the current era? Are the two states of characters practically overlapping? Or is one several meters behind the other? The answer -- which varies by game and by internet connection -- is that the desynchronization between these two positions is significant. Over amazing internet connections in games with slow moving characters the desynchronization is usually on the order of 1 player length. So imagine any fps game character (valorant, cs:go, apex, overwatch, fortnite, cod, etc) -- and then imagine them creeping or walking around slowly. In this scenario the desync between the two states is such that one could picture the character being followed by its clone, touching. If they move *really* slowly then they’ll be overlapping. However as the characters break into a run their clone will trail them by more -- maybe 2-8 player lengths depending how fast characters are in the game. If a player has a high latency the clone will be ever farther behind in all scenarios except holding still.

Now when a player shoots their gun in all of the above games, the game engines will calculate the shot based on where the player perceives themselves. That means that as you play the game what you see is pretty much what you get. You don’t have to manually correct for lag while aiming in modern shooters -- just aim for the head right where you see it. However this introduces the peeker’s advantage. A defender can hold a corner with their crosshair primed to shoot anything that appears, but an attacker (the peeker) who comes around that corner is ahead of their server position and thus they get to a bit of extra time to "peek” and shoot at the defender before they themselves are visible to the defender. Depending on the actual amount of lag and the game itself the defender perceives themselves as either having been shot insanely quickly right as the attacker appeared, or maybe if the lag is not as bad they perceive themselves as having gotten to trade shots with the attacker, but ultimately they lost. The attacker perceives nothing special -- they just walked around the corner and shot the defender b/c they were playing aggressively and have superior reflexes (or so they think).

How big is the actual peekers advantage? Well it varies by game, but an article put out by Riot Games about Valorant goes into detail about how much the peeker’s advantage affects gameplay, and how their engine attempts to minimize it. It’s a great read: https://technology.riotgames.com/news/peeking-valorants-netcode. But to summarize, using 128 tick servers (very fast) 35 ms of internet latency (fast) and monitor refresh rates of 60 hz (standard) they calculate an advantage of 100 milliseconds after extensive optimization. That’s fast, but is it fast enough? Well back in 2003 I used to be a competitive Counterstrike 1.6 player at around the same time that I was obtaining a psychology degree with a particular interest in human perception (reaction time, how our eyes work, perception of subluminal images that are shown very quickly). I tested the reflexes of myself and all of my teammates. Competitive gaming didn’t have the same structure back then as it does now (everything has a ladder now -- back then it was private leagues), but by modern standards we were probably top 3% ladder players or something like that. Generally speaking there isn’t much of a speed difference in the whole pool of pro-gamers, at least when compared to new players. They are all pretty fast. Response time for watching a corner and clicking as you see a player (already perfectly lined up) fall into the range of 150-190 ms. Tasks that involve moving the crosshair to react quickly (as opposed to having had perfect placement already) slow that down another 50-150 ms. But generally speaking the competition between two similarly fast players with good crosshair placement comes down to very tiny units of time with even 10 ms producing an advantage that is measurable (both Riot and I agree about this). This means that had the server allowed for a double K.O. (which these games do not) we would find that in fact both players were very good and would’ve killed the other just 20-50 milliseconds apart. Valorant and CS:GO don’t work like that however, and instead the game essentially deletes the bullets from one of the players and leaves the other alive. Unfortunately this difference in human reflexes amongst competitive gamers is entirely gobbled up by 100 ms of peeker’s advantage -- meaning that at high skill levels the peeker will very often win and the defender will very often lose. So while the Riot article celebrates the success of engineering that allowed Riot to reduce the peeker’s advantage as much as they did, if you read the fine print you’ll find that the peeker’s advantage remains huge.

I don’t mean to pick on Riot, far from it. They’ve clearly done an amazing job. The other games I mentioned earlier are presumably in the same approximate ballpark, though I can tell you from personal experience that some of them are a fair margin worse. Not all of the games I mentioned use a 128 tick server (only one does). They also have longer interpolation delays and other little engine details that slow things down further. Riot is also an insane company that literally owns/builds the internet just to reduce latency for its players -- so if we want to take away a general sense of how bad the peeker’s advantage is in most games we should assume it to be worse than the scenario described above regarding Valorant.

Now that I’ve discussed at length the peeker’s advantage, allow me to present a related netcode model that attempts to solve these problems: The Peeker’s Compromise. If we delay the time of death on the serverside by the timing difference between the attacker and the victim, then we can allow the defender an equal opportunity to shoot the attacker. The server can then determine the winner (and the remaining damage) based on the performance of the human (instead of using the ~100 ms of engine-related advantage and internet latency). So let’s use some numbers for a hypothetical situation. Let’s say our game has 128 tick server, the players have 35 ms of latency, and 60 hz screens (like the Valoran example from earlier). Right as a player peeks another player they essentially get to shoot 100.6 ms sooner than their victim. As their shots arrive at the server the server might calculate that the victim has died -- but rather than killing the player it will keep them alive for 100.6 ms PLUS their own latency, which in this scenario puts the total at 135.6 ms. If during these 135.6 milliseconds the server receives shots where the defender hits the peeker, it will enter a section of code that attempts to settle this discrepancy. First off, it is entirely possible that after compensating each shot for the difference between the players we find out that one truly was faster than the other -- the game could use this information to decide which one lives and which one dies. It also might make sense to allow damage to legitimately trade kills and to build double K.O. situation into more first person shooters.

Let’s talk about the artifacts of this new and proposed system. In low-latency games with a high tick rate this change would be subtle -- we would just have no more peeker’s advantage. As latency increases all way up to 200 ms we will have a new artifact. Instead of having a more severe peeker’s advantage we’ll end up with a scenario where it looks as if players are taking 1-3 extra bullets beyond what would normally kill them -- although if you stop shooting early they still end up dead a few milliseconds later. The Vandal in Valorant (similar to the AK in cs:go) fires 9.75 shots per second, which is one bullet per 103 ms. So in a best case scenario the player death is delayed by the time it takes to fire one extra shot at full auto, and in a worse case scenario we add 1 bullet per added ~100 ms of waiting done by the engine. It would also make sense to cut off certain shots from being counted from a laggy player (existing systems already do this in their own way).

Gameplay at lower skill levels wouldn’t really be affected one way or the other. It isn’t affected much by the peeker’s advantage either -- players have to know where to aim and thus be involved in legitimate reflex test before we’re down to something so close that milliseconds of delay have an effect. If players are oblivious to each other, or place their crosshairs incorrectly as they come around a corner then the added slow down of the human having to make a new visual-search-decision-plus-adjustment is too slow for any of this to matter. But at higher skill levels there would be some actual changes to gameplay. The most significant change is that players would be able to hold corners -- and if they’re truly faster than the peeker they would win. In such a design it really would make much more sense to allow two players who fire at essentially the same time to kill each other which if adopted would need to be addressed at a game design level. Also games that had an alternate method of very indirectly addressing peeker’s advantage, such as weapon instability during movement as a major element (arma, h1z1, pubg, tarkov, etc), would have more options, and may need to tune existing timings to get the same feeling back.

Previously I benchmarked a node.js first person shooter game server on smaller Vutlr VC2 instances (1024: $5, 2048: $10) comparing them to the dedicated $60 offering.  My conclusion was that the dedicated cpu came at a premium of 3x, perhaps too pricey for my specific game. I abandoned the dedicated offerings, and then the question evolved to be: which of the VC2 instances offers the best value? Previous estimates suggested that renting multiple 1024s would be the best. My answer now is that renting 8192s is the fastest, which comes as a surprise!

The Geekbench Scores listed by vultr (detailed in previous post) suggested that each tier of VC2 was roughly *but not quite* twice as fast the previous tier. Using their own benchmark data, I calculated that running 8x 1024s would result in ~28% more processing power than running 1x 8192 albeit spread across 8 different servers. 

Rather than take the theorycraft at face value, I have now rented an 8192 and run my own game benchmarks on it. On the 1024 I was able to run 3 game instances at ~85% cpu. On the 2048 I was able to run 6 game instances at ~95% cpu. And much to my surprised on the 8192 I was able to run 24 game instances at ~78% cpu. These all come out the same dollar-to-player ratio. 

I had to write a special node.js child-process-fork version of the nengi.Bot to connect 240 players to 24 separate games-- which was the number consistent with a full round of ZombiesWithGuns.io.

Screenshot: 73% total cpu usage, 24 instances and 4 nginx workers at 10-16% cpu each (out of a total of 400% cpu).

So the 8196 really was at least 8x as strong as the 1024, and 4x as strong as the 2048. This runs counter to the Geekbench Score, which would’ve predicted only running ~17 games instead of 24. In fact in my test, it actually came out to be the strongest per dollar by a 5-10% margin. Technically I may have been able to squeeze 25 or 26 game servers for https://zombieswithguns.io/ onto this machine, but I like to leave a little breathing room.

On a related note, that breathing room is a much bigger deal. Having 15-20% of the cpu allotment idle isn’t a whole lot on a 1024... but having that percentage on a 4vCPU 8192 is quite a bit of processing available just in case one of the game instances spikes.

Why did the 8192 perform better in my game benchmarks than what was projected in Vutlr’s Geekbench scores...? Well the real answer is I don’t know. I’m not even sure what a Geekbench score is. But here are some theories. Multi-threaded benchmarks often suffer some overhead. Four cores never registers being 4x as fast as a 1 core, etc. But I’m not running benchmark software.. I’m running nengi.js game instances which are designed to scale horizontally as entirely separate processes (!!). In my case this is obviously superior to a benchmark because it is the actual program that I intend to run. Other theories are perhaps that nginx does a better job with more workers, or that the CPU is somehow prioritized over other tenants, or that the RAM or network i/o was more of a factor than anticipated and thus benefited from being improved. I don’t really know about any of that.

Well that about sums it up. For ZombiesWithGuns.io the bigger VC2 instances (up to 8192) have been better, not worse! I’ll be deploying several 8192s across the next few weeks and I’ll write another blog post if I find out more.

A networked cube via nengi.js and babylon.js. The upcoming version of nengi (v0.3.0) finally has some decent client-side prediction, which was my prerequisite to opening nengi up to integration with 3D libraries. This is the beginning!

Client predicted projectile @ 300 ms of latency, followed by the server side shot (white is client prediction, black is server-confirmed reality, though seeing it is subject to the latency as well). 

Out of the 7 projectiles, the first 4 have been created on the server as far as this client knows. The 4th projectile has a line demonstrating its relationship with its counterpart. The last 3 white dots are projectiles that have also been fired on the client but due to simulated lag have yet to be created on the server and confirmed again on the client (which is the point at which the black dots appear).

Getting these things really deterministic (and therefore consistent) with date/time related math seemed inherently flawed, so I’ve favored a fixed timestep + deterministic simulation approach instead.

Example/process:

When the client is determining if it can fire a projectile, it does so by comparing the weapon cooldown with an accumulator variable. e.g. is our accumulator greater or equal to our cooldown of 0.500 seconds? Each frame it adds 1/60 of a second to the accumulator.

The server also uses the same logic to determine if it should created a projectile -- at a rate of 1/60 being accumulated for each frame it receives from the client.

 The result being that the number of frames experienced by the client is the number of frames processed on the server... and the small aggregate of 1/60s being added up will pass 0.500 on the identical frame for both computers (deterministic). 

The server can also decide if the frames it receives from the client are valid... did they contain actions that are feasible for that player? Are they coming in at a believable framerate? 

From there each computer (client and server) can simulate their own highly-similar result for the projectile. The server might determine that the projectile hit another player, and deduct hitpoints. Meanwhile the client, which for cheating reasons does not get to say anything about anyone’s hitpoints, might instead determine that the projectile hit another player and draw a purely aesthetic blood effect.

Oh and there’s a hook for player authentication as well. It is up to the dev how/if they want to authenticate though. I need another day to clean/up test before release.

There is no formal TLS / SSH / WSS support, but it is easily added if desired.

It came out pretty interesting in the end. An instance can transfer a player to another address and transfer some data along with the player. The instances actually speak to each other before the player gets transferred. The instances do this via the same websockets that the players use, they just identify themselves with a password. The source instance generates a transferRequest, and the target instance responds with a transferResponse (or fails under a variety of circumstances). If the transfer is accepted, then the source instance sends a transfer message to the client which goes ahead and connects to the target instance, bearing a code that identifies the client. The target instance recognizes the client upon arrival, and gains access to the data that was initially transferred from the other server. Thus nothing from the client is ever trusted -- its the servers who have a conversation with each other, and they merely give the client a receipt (a reference to the conversation) which does not divulge any details to the client. This follows the security model in nengi of making the servers the authority on the game state.

Transfers are pretty quick, taking slightly longer than the round-trip latency between the machines. Transfers are asynchronous.

This enables some really basic but scalable functionality -- such as having a entrance to a dungeon, building, city, portal, etc. where the end point may reside on another server... be that another physical machine or just another instance running in another thread on the same machine. It also offers a way to secure instances, because the game logic can define and enforce rules about who can enter which instance. Lots of features rely on controlling player access (pvp arenas, party-based instances, games that require membership, etc.)