#game cluster

Let's go on a small ride into the past. A long time ago, everything had it's place. If you meant server 7 had the file important.txt in the folder management, you knew that file was on one machine, in one place, on that metal box in the closet behind the stairs. Then people realized that this makes it vulnerable and added a second box that would backup that first one. Then the first one died and they spent a lot of time rebuilding it from the backup box. So they started to use load balancing and fail overs, once one box dies, the other takes over and nobody noticed, except the technical staff.

Fast forward a bit and now we have the thoughts behind the cloud. One of them is that although there is still metal running and storing your things, you don't really care which one it is. It's an abstraction between the actual machine running it and the application or service.Instead of one web server delivering your webpage you can have two or two hundred if you need, but you don't really care much about the details of each physical box, as you just need enough to make it work. One physical one, can house multiple virtual boxes as well.

Let's jump back to games. To this day, you still have game servers (say in some game like Battlefield) where the server loads a map (let's put a pin into instancing for now) and people connect to that server with that specific map. Early MMO games, like for example Ultima Online had a similar concept, where a server was responsible for a certain area. You could notice the server boundaries when you walked across them on the map. You could even exploit them, because monsters would not cross the boundaries. A lot of things happened with these boundaries but mostly they were annoying to the players when they encountered them. I bet they were also annoying for the guys at Origin back then, who had to deal with them. Some servers would do pretty much nothing, while others (the one housing Britain for example) were bursting. To recall Battlefield, you can find at any given time a ton of full and a ton of empty servers, but very few that are only half full. So either you are wasting servers that do nothing, or have overloaded servers. Eve Online has a similar problem. The star system Jita is the main trade hub for the entire Eve universe. Because the traffic is so high, they dedicated an entire server to just that one system.

The core troubles, if you break them down, are in segmenting areas (invisible server boundaries, or star system only, like in Jita) and wasted versus overloaded resources on servers. I might go into detail about the different kind of nodes and how else the system is setup, but for now I want to concentrate on these two issues.

To understand its approach, I need to explain two of the many node types the cluster has.

The user nodes take care of 3 essential things. Authentication and authorization of users, in other words, login, character selection and injection into the world, and lastly character creation. This last one is simply the act of having some character editor where you can make your own personal character, choose its name and so on. We ignore that for now, as after creation it will select that character and inject it into the world.

The zone nodes handle the actual interaction of the player with the world. It knows where everyone is inside the world, enforces the rules and does the calculations for combat and skills. The term zone is chose because a zone is not fixed.

So let's go through this how it would play out in action and I explain a bit more along the way.

There are always 3 user nodes or more online. When a player tries to login, the system will send that person to one of them for the interaction, using a load balancer, making sure the least busy server will get the most requests. If for example there are a lot of logins happening (or expected because of past experience) more user nodes are started. If there are few logins, user nodes are shut down.

So now the player is on a user node, sends the credentials and does the authentication bits. The user is now logged in and gets presented with the selection of a character and/or the character creator. Say this player has never used the game, they would get directly to character creation. Now they create the character, all in the client mind you. When done, the client sends the data for the new character to the server and there will be some checks as to the name, and so on. Assuming all is peachy, the character gets created and the same process starts as if the user would have simply selected a character earlier.

The user node checks where the character is (or should be) and calls up all the data the servers will need but also the client and transmits that. Then the node asks a controller node for the server that runs the zone this character is currently in. The controller will either locate that node, or simply commission a new zone on an existing zone node for this and return the details to the user node. With the character injected into the world, the user node hands over to the zone node and the player is inside the game.

Let's say the player is in a city. The entire city (and close surroundings) are one zone at the moment. Someone throws a party in a location in the western part of the city and lots of players attend. The traffic is rising on the zone node. So now the node starts shifting zones it handles around first. It hands off the second most busy zone first, then starts handing off other zones as traffic keeps rising. Once a zone node handles only one single zone and it gets still too much, a zone split happens. A split can be in two or even multiple parts. It will try to keep the largest masses in one zone, but it will fracture them behind the scenes. The player knows nothing of this of course. In the end, everything should at all times, run smoothly and fast.

With that the world can be very very big, but you only ever have to load the places where there are actually people. Also one server node can handle multiple zones where only a few players are, while the busy ones always stay fast as well. Close together zones can be merged if there is little traffic. Certain areas where there is a repeating pattern of medium traffic (think a road, or a pass through the mountains) can be configured to be one zone in itself and only be split when truly needed. Should there be fewer players, then you can shutdown hardware until needed again.

The experience for the player is the reason for all this. No splitting, merging, shifting or crossing boundaries of zones have any noticeable effect to the player. To them it will be one continuous world, no gates, no loading new levels and waiting, just going on and on.