MMO Developer

As always, major spoiler alert. You have been warned.

I've been curious about this game since I seen the first videos about it. The basic concept sounded good but the videos brought it home. Delaying it though I'm not sure they actually accomplished making it better, then it would ahve been back then.

I really need to tackle expectation management. This is where Watch Dogs fails miserably on a wider scale. It has strangely a few of the notions from GTA V in it as well.

So the game is you being a hacker on a revenge course. Your niece got killed in an accident because a job went bad and the ones you stole from wanted to kill you. So now you go after those that ordered the hit. It sounds like a solid story, made better by the fact that Aiden is not avenging his wife, but a little girl that wasn't even his. Still slightly tropy, but far enough away to feel different enough. In his quest he hacks, steals and kills his way through the city of Chicago. I've never been there so I can't tell how much it feels realistic. He is a vigilante, keen on kepeing his identity a secret.

And here is where the game already tanks majorly. At the very beginning is a guy that almost gets killed and survives. Later you have to get into prison and convince the guy not to talk. Meanwhile the radio blares at you that the police is hunting for the suspected vigilanty Aiden Pearce. Why the hell did I just go through all the trouble when everyone already knows? That little bit of info that on occasion gets played over the radio just broke so much of the world that it's unfathomable why anyone though this was a good idea. To make matters worse, his sister is apparently rather easy to find. So why did nobody, ever, stake out her place? Or just use one of the thousands of cameras and send a car when the pixelated version of Aiden shows up? Doesn't take a genius for that and it causes a lot of suspension of disbeliefe in even the simple most common sense parts.

So the story is actually not too bad, but just often gets a bit odd. A lot of effort is spent on crap, that just can be seen a mile away, is not worth it. For a quarter of the game you setup some elaborate scheme to get someone into a place that is heavy fortified. In the end you go in anyways. Alone. Why couldn't I just get it over with in the beginning? The player also seems to have quite a lot of explosives, so being able to blow up a wall or a door should be a rather simple task. Yet that doesn't work that way.

Watch Dogs falls into the same trap as GTA, it lines up some nice pay offs and then doesn't pull through. The beginning of the game with the stadium, that would have had potential for epicness, and it was completely squandered. The hunt for the guy that survived was pointless, though the mission was very well done. Most of the things you do for T Bone seem even more pointless or way too simple (the HQ of the company is less secure then one of the CTOS stations?). The biggest let down was the auction and the complete zero impact that had. What happened to the girls? You never hear about it again. You can do side missions and then do one final clean up mission, but that one is not even very original or challenging.

Which brings me to the two main videos. The one where Aiden hacks into some underground place and kills people that sell girls. Well, that doesn't happen. And secondly the club. Yes you go to that club, but for an entirely different reason. The mission with De Marco as seen in the video is interesting and even tactical. What you really have to do is find Defalt, a hacker/DJ in that club. Well it's not hard to tell where he is. But the game will of course not let you just pull your weapon and shoot him, or get close to him. First you need to hack some phones randomly, some cameras unil you hack the camera of his laptop. Then he takes off and now you are allowed to chase him. Again, this entire mission feels bad in so many ways. Let alone that this guy hacked into your "Top Secret and secure bunker" and he hacks everyones data while being a DJ at the same time, seems very weird (also reminds me a lot of DeadMau5).

The love interest which is clearly there is rather down played and then eliminated in a even more stupid and so overly tropy way that I just couldn't believe it when I was made to play through it. Nough said on that one.

Expectations, failed.

So how about the mechanics? Here is hope for Watch Dogs and why I hope they make another one. With better writers and with better principal oversight.

Driving is okish. Cars feel different. though most high performance cars feel just over the top. There is only sliding, spinning out high speed or nothing. So they don't feel as nice to drive as they could. Handling is not too bad either. You can flip cars and total them, but both require an amount of force that feels realistic. It could handle better. Also there is no shooting while driving, which makes a lot of chases and other battles very hard to accomplish. You can hack while driving but not shoot. The hacking while driving works half automated. You don't have to look back to a blocker to trigger it. You just drive over it, and the game will prompt you to push the hack button when the enemy would get caught by a hack of that installation.

Shooting is where the game shines. This feels good and responsive. Hits feel solid. It's sad that most guns are highly inaccurate even with single shots at medium distances and that there is no silenced sniper rilfe (for the player). For good kill shots you need to be close and then the silenced hand gun is your best tool. The only one that tops it is the silenced vector which you get for 5 online rounds of trailing someone. You can die quickly and you can do damage quickly if you aim right. So the fights feel more so that you need to find good cover and positions instead of running in guns blazing. You also want to checkout the area first with cameras and tag all enemies so you can see them even through walls. This makes the combat highly tactical.

General moving about is ok, but for some reason I keep feeling like Aiden should be able to do simple movements from the Assassins Creed series as well, like jumping over a gap and hang on to a ledge. That one move kills a lot of potential. Aiden can only swing himself over an edge, not stand on it and jump forward. He never really jumps over a gap or anything like that anyways. Still gives him a large range of movements in the puzzles at the CTOS towers or when assaulting a position.

I mentioned unlocks. The towers are some (the story forces you to unlock most centers anyways) but there are more collectibles. Some have an impact, like the chess games will allow you to gain maximum skill later in one particular thing. Some unlocks you only get through online play. It feels ok, but could have been done a bit better.

Many of these side things feel very tacked on. Not very coherent either. This gets worse though.

On one side Aiden has this meter that tells him how "good" he is. Killing innocents and cops makes him bad. But also injuring the pedestrians does that. By doing vigilante things like stopping crimes etc, you get "good points" again. Now this has very little impact actually. The only thing that happens is that if you are too bad, that people will randomly call the cops on you. Over all your actions seem to make little impact. You can invade peoples privacy and steal all their money, without any impact whatsoever. Moral system gone wrong.

Which brings me to the online play. You can disable it, but that means you reset your score and you lose any and all unlocks you might have gained that way. That's harsh. It's also forced upon you. You can play a day without a single person attacking you and then 3 in an hour. I think there is a limit, but I haven't seen it yet. Also there is a thing where you can't be invaded on missions. But when you are just trying to unlock a tower, that's free movements and it can happen that you just get invaded a moment before you hit the button. Very annoying when it's hard to get to a spot. Too bad most of the puzzles are that way, you need to find the right spot, get into the right position or hack the right camera. All of it can be interrupted at any given moment. The finding of the culprit who did the invasion is also often just wildly running around and not very interesting. I think the game would have benefitted from those features being completely absent and that effort being invested into a better more coherent story and some better missions.

"You cannot purchase this item because somebody else has already purchased it."

I've seen this text thrown at my face by the steam market too often now. It just occurred to me, that this is a premium bad example. Not the market itself, but how items are handled.

Imagine an item, identical (since digital) and there are about 88'000 offering it for a small price, like 10 cents. Well not all, there are some charging 8, others 9, lots 10, lots 11 and some even would love to get you to buy it for 307 dollars. The problem is, each item is individually there until someone clicks buy. You also have to check a box and click again buy, which makes it even worse in using it, but the really awful part is this: Someone already bought it. While you clicked the first button and tried to hit the check box, someone else already did that and clicked the second buy button. Item gone, but of course your client hasn't caught up.

Amazon has that problem even worse. They have not just the same items but from different sellers at different prices. What do they do? Show you the best price and let you buy it until there are no more items of that price. Actually that's how most work. Except the steam market. An even better model would be to implement a buy order. Like the item is there for 10 cents, but I wont buy it for that. Maybe for 8 I would. So I put in an order for one item at 8 cents. Once an item is offered for 8, it will be sold to me as I'm the first who put that order in. This is how general markets work when it comes to currency exchanges and even would open the doors for speculation on the market. Maybe that is what they try to stop with not having that feature.

I start a new series today I will call Examination, that fills a different slot than Bad Examples. In it I pick apart a game or part of one, from my personal view point in regards to development. To start this little journey, here is the latest blockbuster: Grand Theft Auto V.

Just for those who have not played it yet: Spoiler alert. You may proceed.

Did I like GTA5 as a player? Yes. Was it what I thought it would be? No.

The handling of cars feels much better than in GTA4 which I played a bit in anticipation of this release. The floating properties of the previous game are gone. This usually is good but also leaves you with less options in trapping a car in a mission.

Game play where the target is on rails and magically is way faster than you because GTA wants to show off it's nice scripted sequence of events, so who are you to ruin them with taking out the driver too fast, still exist. That was one of my peeves from GTA4 and it's still there, but thankfully not too often. Though being there not too often makes them glare more.

Three protagonists is nice, it works rather well. One major drawback is when you switch they move about and sometimes are not dressed anymore how you wanted them to be or drive something shitty suddenly, or are at the opposite end of the large map, where you didn't want them to be. The second one is that they play virtually the same. Their stats mean little in driving, shooting or running. The only really useful special is Franklins magic driving ability, which makes racing rather easy but also get always almost trivial. The hard way to activate them makes it also less user friendly to use in a pinch. It feels like something tacked on just to justify the stupendous amount of activities you can partake in. Some are fun, some just annoying. Also, why can't I transfer money between characters?

The new cops system is pretty clever, though sometimes still guided by too much magic. The cops know where you are even if they "can't see you". They will drive pretty straight at you making a clean get away almost impossible. The best strategy seems to be take a new car and then just drive anywhere you want.

Adding sway to the sniper is the only major negative point I could find in combat. It usually works well and the aiming system is actually much more preferable to the old one. I would like to ply this on the PC though, free aiming with the sticks is still something I try to avoid. I liked the options to mod your weapons but was expecting a bit more, both in options quantity, in comparison capabilities and in impact of the options.

The radio feels less engaging as in GTA4. My guess on that is that there is too much of it and not enough standing out features. The songs repeat faster as the entire track seems shorter for some stations. Breaking news is often overshadowed by scripted calls or character conversations so you don't actually get to hear the often hilarious things being said.

Which brings me to the last point I want to touch: The missions. As usual, they are great. Varied and often surprising. But there are multiple large holes here. A pet peeve are the assassination missions for Lester. You are tasked with about 5 of them, each one having an impact on the stock exchange apparently, but by the time you get to have an impact and it actually making sense for you to use that knowledge to make a lot of cash, it's already too late. Not that it matters too much. After that first heist, you have enough to buy most weapons and upgrades immediately. Tricking out cars only makes sense for each characters personal vehicle, as they come back over and over. Other cars are robbed from you often throughout the game play. Also only 5 assassination missions? Perhaps a DLC addon? Now there are heists in the game and a whole system of crew members in it. The low end crew members always screw up and die. Always. So using them is useless, but you can only select between two, the useless one and the one that wants a big cut. Having the ability to make a low end crew member into someone better would be nice, but you are never given that chance. It also only matters twice (if you don't count the FIB heist). But that is not the problem with heists. The big problem, throughout the game is that your characters don't get paid. Except for the first big score, and the last one, your heists are nil. You make neglect able amount of money on missions and some "big scores" turn out to be dead ends too, like the exotic car collection mission. The stream of "no payout" really goes on my nerves. It leaves me not wanting to play more but makes me feel hollow and "why did I just do that?" in the back of my mind. The disappointments in these missions and even in  "the big one" near the end, make me feel ripped off in a bad way. Same is true for following through on missions. In one mission you essentially down a whole plane into a lake. There is some talk "you could get that" but the game makes not a beep about how one could do such a thing. And then it's forgotten, forever.

Music is wonderful. Because of it's properties, it goes into our brains and tickles the imagination. Through that we imagine emotions from slow and heavy music to be sad, while prickly and jumpy notes seem happy, even cheerful. The notes themselves do not do that, it's the context we give them in our head that does the trick.

The same context makes a song feel shallow or epic. I think everyone has heard "Bolero" or "The flight of the Valkyre". As Bolero starts with an almost inaudible part, it swells up into this fantastic musical masterpiece and ends with a big bang. By then you are moving with the rhythm of it, tap along or nod to it, because it's just so gripping. It has the feel of a procession, a march of some kind, something that gets bigger and bigger and comes to a clashing finale. The Valkyre goes in bombastic from the start. It makes you feel like there is something big happening, but all the time. A battle, a flight, something grand of massive scale. Now these are the things in my head, but it's just those notes that create it, no song texts along with it, just notes evoking images and emotions.

Games can do the same.

If you haven't grabbed Spec Ops the Line yet, do it. It's one of the best games at making you feel emotion. A warning, it's a shooter, but on easy mode, it's not very hard. The entire thing is a roller coaster of emotions and just for that, gripping. You will not feel good and grand after it, in fact, the game makes you feel really shitty at times, but that it manages to do that it simply amazing. No it's not frustrating, but it evokes real emotion. It sucks you in and then slams you in the face.

Now other games are there to make you feel proud of yourself or simply that you managed to put three equally colored things in a special formation. But that's very basic. A very good game can give you a variety of emotions and you can even pick them more often then not. The thrill of combat if there is really something to loose. A sense of accomplishment if you built something. Sorrow about a friend loosing something. And even a simple, almost boredom like state, where you can do something to further your progress, but will not require overly much of your mental capacities.

Yes I just said that it's a good thing to have something "mildly boring" in your game. With something I mean not everything. If the major game mechanic is based on that, you might as well put it away, forever. But if its an ancillary activity you can do, but are not forced to do, then it can be enjoyable in that exact state. Some will grumble about it, for certain. Why put it in when it's not fun? This fun only focus kind of makes us relying on pure fun to make the game usable. If the player has no fun, he hates the game and stops playing.

So for a normal game, that might be ok, usually you have relieved the player of his money by then. For an MMO this is an entirely different matter. Once they stop having fun, the game is becoming a financial liability and gets canceled. The truly successful games have retained users for years. Ultima Online is counting 16 years so far. There is a way to do this using the skinner box, or the gear treadmill as some call it as well. But once you see it, you cannot unsee it and stop playing in most cases.

If on the other hand the game offers you fun and also more relaxed activities, then the chances are they are retained a lot longer.

Back in 1997/1998 I was playing Ultima Online. A lot. It made sense that my character and everything else is not stored on my computer, as that would mean I could manipulate it and so could others. It made less sense to me that my macros and other handy configurations were not stored on the server as well.

Today we call it cloud storage. But then so is everything in an online game. With many games on Steam you get cloud storage of your saves. I recently reinstalled my computer and didn't even think for a second about my save games, because they are all in the cloud. When I reinstall, it will all be there again. Something similar happened with Battlefield 3, where my settings were remembered from when I had it installed before.

So there is really very little that should still be stored locally only. What version the client is for example, but not what the CTRL+H macro does. This brings up a tricky point though. What about graphics settings?

Say I play the same game on two machines, one is a high end gaming monster, the other a low budget laptop. Different graphics cards, resolutions, everything is different and so the graphics options should be different too. In such cases, there should be an option to choose if these should be local only. So likely I might use the laptop to game on a boring trip but my main stay is my gaming rig, so I want that in the cloud. Or I could have multiple performance profiles, if I replace my crappy laptop with a new crappy one, I still don't want to go and search for all the right settings again.

So it's a trade off between choice, which needs to be offered in some way, and convenience which in this case would be low as the replacement scenario is not that often a problem.

Don't you just hate it when you install a game or a software and the first it does is telling you the software is outdated? Well the game handles all of this slightly different.

Updating and installing is very similar. Roughly spoken, it's copying files form one place to another, with an update being just a tiny bit more selective of the process. This is why there is almost no difference to it internally for the game.

At first the user get's the launcher. I call it that, because it wears many hats, but only on the surface. You can download this file later, for free, from our servers, just execute it and it will install the game for you. It's also the same application that updates your game, and incidentally, launches it. So the exe file you download, is the same one that you start every time you start the game. You could take that file, give it to someone, and they can get the entire game just from that little thing.

And little it is. The point is to have an as small as possible file in this, for exactly all of these purposes. It also means an as lean as possible file. So how can it do all that? The short answer is, it can't. But it can seem like it.

There are two main use cases, the installing and then the launching of the game with included update check.

So how do we install? The launcher is executed and it first tries to connect to the servers, to see if it's the newest version itself. It will self update right then if it's not. If it is, it will also get a small file that contains information about the next steps needed. The launcher reads that the game is not installed according to the registry, so it does the install mode. Thanks to the file it previously got, it knows everything about the game, like how much space it needs, etc. It also has the latest versions in that file, so no old things get installed. The user selects his install options, like where they want it and tell the installer to go on. The first thing happening is now the launcher copies itself and the file I mentioned to the new directory it just created. Then it looks around. Say you launched the launcher from a DVD you bought in the store or a USB drive where you backed up an old copy, it will then match what it has locally in the correct version and queue up what it doesn't to be downloaded. All that information is displayed to the user. Now while it downloads the new parts, it copies the already existing ones over to the place they need to be. This supports scenarios where the DVD contains all, or all parts need to be downloaded. Every single file is checked against a checksum to ensure they are correct and exactly what they should be. While the installer is working, it also displays a message, that the user should now create an account if they haven't already, with a link to the page where they can do that. So while the launcher does his thing, the user can do theirs and doesn't wait around too long. To finish it off, the launcher creates icons and shows a "I'm done" message, before it terminates. If you start it again, it will recognize that the game is installed.

This brings us to the second scenario. With the game already installed, the launcher does a check on the servers if there is a new version. If there is not, it simply launches the game. Eventually there is. So it downloads the same file that it did for installing, that contains the information about all files. Then it compares what it has locally to what is in that file and downloads what has changed. It replaces the needed files and then starts the game, all updated to the newest version.

And now for the bonus round. Since the game constantly talks to the servers, they can tell the client "hey there is a new version" as well. That will cause the client to start the launcher with a special switch. In that mode it will see the game installed and start the update process. What it does though is only download the new files it needs but not yet replace them, as for that you need to restart your client. When the downloads are done, the client will inform the user about it. I can hear the cries about higher pings and such things, fear not. That functionality is entirely configurable by the end user. Either you can disable it on your own, configure the maximum bandwidth for downloads in the background, or let the game itself make educated guesses on what would be a good maximum, so your gaming doesn't get impacted by it. the last one is the default and it would for example download faster if you just stand around in a dead area then if you would be fighting in a thickly packed city.

Steam is wonderful. I love it. I own over 400 games on Steam. But Steam is by far not perfect, though it does a lot of things very right.

During the holiday season they ran a lot of sales, hence my enlarged games collection. Games I loved and played a long time ago I could now snap up to own "forever" without hassling over the CD. I bought about 30 games for less than 50 bucks in one week. Then suddenly Steam told me I can't buy anymore games, just when a great game was on sale I just needed to have.

Turns out, it's not a bug, it's a feature. A security measure if you spend a certain amount of money, or in a certain amount of time, you get blocked form further translations. I can still use Steam, but not buy anything. And that during the most active time of the year.

I had to register at their support site (don't share user data) and tell them that it's truly me, including some personal data. It took almost a week and then I could use Steam again, missing plenty of good opportunities.

The problem is not having a security scheme, the problem is how it is implemented. If there is a threshold, you should make it a soft one, and require information first, clearly stated, with a warning, maybe even a display of how much more you can spend before your account gets blocked until you provide the information.

I was less miffed at that it happened, more at how it happened. Communication is key. Expectations even more so. If I had expected it, I would have done that instantly. If I didn't I would have blamed myself "it's right there, I'm stupid not to do it". Instead I expected Steam to work and this interruption went against my expectation.

If something could go against their expectation warn and inform. Create grace periods either by volumes or time. Make certain they can work, like if you need them to call you and you only work weekdays, don't make the grace period 24 hours on a Saturday morning.

I just posted about logins and security, then I came across this noteworthy post from Tobold.

He essentially notes the wonderful account safety hoax going on out there. EA switches from emails to names, Ubisoft from names to emails. Both claim it's for better security. I claim it's pure marketing, to be seen "doing something".

The most essential part of account security is to keep your credentials hidden. EA does the opposite here, your login name is also your forum name, the player name in games, etc. So potential attackers already know possible targets.

Why not use emails? Today we are afraid of spammers, so we keep them hidden. Therefore, by nature, nobody in general knows this information. Except those I'm in contact with.

Though you might have multiple email addresses and they can't be sure you use this one. Also if your mail is "[email protected]" you can write it "[email protected]" and still get the email, though the game company will think it's a different one. You can add dots at random really, as gmail filters them out for delivery. On Gmail you also have the + hack available. Simply take your email like [email protected] and add a + and then something else before the @ like [email protected] or [email protected] for example. Now although you have used your email, you have created a unique login name.

In the end, email is more secure than a public viewable name any day. you are not secure against social engineering attacks, but that's the majority anyways. Brute forcing is not very often used on these kind of things, as they have multiple protections against it.

To be even more secure, you have another name entirely different from everything else. A login name, that is not what anyone ever can see anywhere, but only used for login, which is not a character name and not an email. But that requires you to remember yet another name. Email is still secure enough.

But that's all just the account name. The password needs to be secure too. Everything below 8 characters is laughable. If you don't use lower and upper case, as well as numbers and special characters then your password is weak too. If it's a pure English word, you might as well tell everyone, as those can be hacked in no time at all. Choose secure passwords, no matter what your account name might be.

True security only comes with better methods. The best there is, which is trusted by online banks even, is a two factor authentication. It requires a phone for SMS messages or a device that can run an app (smartphone, tablet) which generates the two factor number based on a pre shared secret and a time based algorithm. That way you have to know your credentials, but also have a physical device. Even if someone knows your credentials, they can't login since they miss the device. Or if someone listens to your traffic, they can't get into your account as once the number given is used, it is blocked by the system and will not allow entry with the same credentials at the same time.

Recently I spent an ungodly amount of time with user and login code. Since my mind was a lot on that, I concentrate my thoughts here. This is like other posts a sort of general "how to do it right".

First off let's define parts that come again a few times. The login is the username of the user, identifying the person. This can just as well be an email address, it doesn't matter as long as it's unique. The password, well, that's clear. Then there is the session id, which is used to identify an active session. Then the token is there to authenticate the user, like a username password, but all rolled into one, without it being the username and password. Important is their meaning, how they look like exactly, is not really important as long as some ground rules are kept. I'll cover them as those things appear.

This process can be used for securing your game but just as easy for your webpage. It contains as much security as needed, but not more. If you do less than this, you should change your application. More is possible, I mention something near the end, but for most, more is overkill.

Let's start simple. A user provides his login and password, clicks a button and they are logged in. A lot happens behind the scenes now. Let's pick it apart.

Now a login must be unique. I suggest using email addresses as those are more memorable. If you chose to give out names, you can use these as well. I would suggest using email to login and usernames given separately for their profile on your application but not allowed to use it for authentication. That way users can't guess someones login.

The password is a place were most already fail. A password below 8 characters is not acceptable these days. Longer is even better, but never allow users to make shorter ones, the chances of hacks are significantly higher. Include some "password strength meter". Suggest, but don't enforce the use of small, large letters, as well as numbers and special characters. If you enforce it, then brute force attacks have a higher chance to succeed than if you only strongly suggest it. Use a password strength meter, add text that it would be beneficial to use longer password and more varied characters. Link to articles like this one, about password padding. Disallow the most common passwords (there are tons of lists out there and new ones coming out every year). Simply block the passwords that are most likely to be cracked. Use a dictionary attack on the chosen password and suggest to use a different password if you can crack it in a very short time. A good way to do this is while the user enters other information, do it in the background.

So now the user has a more or less secure password. Now comes the tricky part on what to do with it on your side. Clear text saving is out. There are many methods like salting, hashing, pepper, static and dynamic salt, but all of it just really pales compared to true encryption. If you are fluent with PHP there is a new feature in PHP 5.5 that will make your life easier. If you use the Symfony2 Framework I did some heavy lifting for you and made a password encoder that also runs in PHP 5.3 and up with a compatibility library.

Using blowfish (or in the future a more secure implementation) this will encrypt the password using a given salt. You can also specify the level it should encrypt, which doubles the length it takes to encrypt it with each step. There is multiple reasons why this is superior to everything else. It is based on encryption designed for this job. The password, the hash, the level as well as the algorithm are all stored in the output string. So you save one "password" string, not multiple parts. The string stays compatible even when you update to a newer algorithm or update the level. Next time the user logs in, it will simply update the string to the new version (which means you just run a small function on each login to see if the parameters are still the same). But even if you do change, it can still verify the old passwords. This way you have secure and future proof passwords.

We are still at a simple login. With the successful login, a new session is created. The session id identifies the session. That id is shared with the client. For web pages usually in a cookie. All future communications happen with that session id sent over the wire. Your system will then look up if it has that session and also check if the session is still alive. If you have some storage that does time to live on its own, then you are good. Otherwise you have to check if a session has expired. For not so critical things a couple hours are ok. For more critical applications 10 to fifteen minutes are more common. Worst case, the user get's logged out and has to authenticate again. The session id is an all in one key that has an expiration date. If it's used, that date refreshes. On a webpage this might not be desirable, but say you have a game and a user is now allowed to be logged in twice at the same time, the second login would kick the player that was logged in first from the game. The session id is "insecure" but also vanishes after a while. If the communication channel is encrypted (SSH, HTTPS) it's secure enough for pretty much anyone. Since sessions vanish after a relatively short time they are not too much of a concern.

But what about the the "remember me" or "keep me logged in" feature? Here we come to a bigger problem. On one hand the user should get the comfort of not having to authenticate all the time, but you don't want to make it insecure. Here comes the token into play. The token works as a one time key. It's username and password in one. Once logged in, while wanting to be remembered, the system creates the key and sends that to the client. Now the browser would take a cookie, and some ini file or the likes for a game. Next time the client connects, it will check the session and see that it no longer exists. Then it will send the token and see if that allows it to be logged in and get a new session. If not, the user has to authenticate again the usual old ways. If the user can login with the token, the token get's deleted and a new one created for the next time. If you login by hand, the token also gets deleted and a new one created. So even if someone can steal your token, one login with your username will kill the old token. Again, communication should run over secure channels either way.

So where does that leave us? The user can login, for normal use the session is his representation with a expiry date and should that date come, a token to login again, without sharing the password at all. This is currently the secure way to handle things, but it doesn't stop there.

With a web browser you could also check ip's or ip ranges upon login or session reloading. The same can be done with game clients of course, but there we have even better variants. The client can get a signature from the computer by reading some freely available data such as the main network cards mac address or some serial number of other hardware. With that you can create a fingerprint of the computer and hook a token or session to that specific fingerprint. If someone else uses it, they will not get in, so stealing the token wont help getting access.

And of course there is two factor authentication. Using a simple app like the Google Authenticator that is freely available allows you to very fast enable that for your own application. You could set a flag that two factor is only needed every 10 logins or every 30 days or once per hardware fingerprint, but always on the user management website. Alternatives are to use SMS instead of an app in a phone and of course allow two factor to be disabled all together if the person has no phone capable of either. There is the "poor mans" two factor which is employed by Steam, just send an email with a code to enter on the page. Since most people use one password for everything, if someone has your account password they oculd just as easy read your email, which is why proper two factor is always an external device (phone, dongle).

Fingerprints and two factor are more icing on the cake and depend largely on your use profile. Keep in mind the accessibility, so you don't lock out users by too strict guidelines and don't skimp on the parts where it's important. For example, in the game client you can't change password or set payment details, but on the page you can. In that case ask for the password on changing the password again and strongly suggest two factor for sensitive details.

I just finished Bioshock infinite. Steam tells me I played it for 9 hours total. I played on medium difficulty as I remember the previous ones sometimes being punishingly hard. Now something over 9 hours, but under 10 hours later, I'm done. About half an hour of that was just the epilogue.

Now ok, I'm pretty good at these games and catch on fast, but I didn't rush through it, but went exploring and finding treasures, lockpicks etc. So if I would have pushed forward, the time would have been even shorter.

For a full price game, this experience was way too short. Again, for that money, I expected more. For the same cash, I could buy movies that would entertain me longer. Or multiple games. Including the first Portal. That one was short, but on purpose. It still takes a while to get through it and get it right. Bioshock Infinite was too short.

The tear game mechanic I used about a dozen times. Often it got in the way because it would block the view of the enemy I tried to shoot. The wonderful mechanics of docking houses is totally ignored after the first 30 minutes, where it's done showing off it's feathers. running back and forth through mostly linear levels is also not my thing. The weapons are nothing to write home about, but using the Z key for zooming in is just crazy. The vigors are mostly a gimmick or can be used as crowd control in the later game.

The writing is fantastic. The story is good with a few blotches in there, making it overly complicated at times and throwing logic out the window.

What ever should come as DLC I have no clue. I really wonder where there is possibly room for it in Bioshock, but I'm sure they find a way to shoehorn in it.

This is the fourth and last post about SimCity (for now). Now that I covered distribution, servers and regional play, here comes the one thing where most companies, including EA, fail the hardest: Communication.

I stipulate, if there was a constant and open communication going on, if the communication was proactive and apologetic, this entire screw up would have been a tenth as bad as it is now in the eyes of their customers, including me. The worst is that when there is a problem you feel like you are not taken serious or even ignored.

Things got worse over time. I pushed this post back to see how it unfolds and I was not disappointed.

The overall communication after way too long a time was "we screwed up". Yes, no kidding.

They, to this date, refuse to acknowledge how unplayable the game is, the many showstoppers and overall horribleness of the game. If you can call it that.

So what do they do? They try to buy people with a free game. Now free stuff is always a cheap and good solution, the execution of it however speaks volumes.

You get to pick a game, if you "activate your game on origin". Now some people still can't do that, blocking them from getting the game and from getting the free apology gift.

They presented a list of games to pick from, all older titles, most of which gamers either already have or are not interested in (or so cheap they might as well be free already). That list though you only get after some voodoo happened on Origins side. They said that a week ago, everyone should have access to that list. In reality many many players are to this day plagued with not getting access. Next comes the deadline. Activate the game with a deadline and then you also have a deadline to choose. One that runs out in 4 days.

If it would stop there it would be great. It doesn't. After looking at the selection I decided I would like to own Medal of Honor Warfighter. Now there is a small problem. According to Origin the game is in Germany (I'm in Switzerland, but let's just ignore that) only available to people over 18. Yeah no problem here. But Origin makes it extra secure by not allowing you to download the game unless you do so between 23:00 and 06:00. Again, I can't get the game unless I do so in a 7 hour window, where I usually sleep or get ready for work.

To sum up. I couldn't get the game to work until I did a full reinstall of my computer, then got a free game, that I couldn't get for a week after "everyone should be able to" and then I can only get it for 4 days between the hours of 23:00 and 06:00. Awful.

Admitting mistakes is good, but admitting only half is worse then admitting none at all. At least in the later case you can still say you are trying to assess the problem or damage, leaving the customers mystified. In the former you completely show how much you are willing to admit.

Communicate honest and open, but also appropriately often. If there is a huge problem now, communicate multiple times a day, or if it takes longer, at least daily. Look at how service providers sometimes give updates every ten minutes "still working" and so on. When you are introducing a new feature, announce it to be coming soon, then introduce it, and maybe communicate some tweaks or questions answered. Don't go on a communication frenzy for things like that, people have enough on their plate.

Think about how you want to be communicated to about another product. If you can't use it, you want to know now, why not. If there is a new color to the product you probably don't care overly much.

Long text walls (like these posts) also are entirely a no go. Keep it short and simple.

In the previous two parts, I have covered some of the unfortunate things happening with SimCity, the distribution and the servers. This time I'm talking about a particular game feature, regional play.

One of the big things in SimCity is regional play. In essence, you play your city inside a region, where you can have more cities, or rather other players have those cities and you play with (or against) them. Basic idea good. SimCity manages to mangle this basic idea though.

Most glaring is the fact that your city limits are rather small. Cities in SimCity 4 could be huge. Now in this new incarnation, it barely is a quarter of the previous game. The small sized map is about what you have now. Clearly you are supposed to do different things in different cities and grow the overall size of everything to a large metropolis. So the big city is all your cities together. At least that's the theory.

In practice you create suburbs, but not even coherent ones, but the suburbs are divided by areas as big as a city itself. So cohesion between the cities is tenuous at best. Each city runs in it's own space time continuum and although you can gift things and even offer services to one another, the implementations are absolutely horrible. Money get's transferred delayed by physical hours, or simply vanishes. Services like power and water that should be available never show up. You can't produce goods in one city and automagically supply another with those. Gifting them by hand in tiny quantities technically is possible, but I have not been able to actually track a shipment making it from A to B.

People travel from one place to another. So does crime, fire bugs, students, well everyone. If you want the people to stay in one place or not allow people from outside the city to go to your school, blocking your own people using the facilities, you can't.

The whole thing is buggy as hell.

What something like this needs is control and transparency. You need to be able to see your actions. Either "shipment will arrive in 10 minutes" or something similar shown to the player. Then something needs to show where the goods arrive. Boundaries and rules need to be open and understood. The manual is rather small and does nothing to explain this.

Welcome to part two. In part one, I talked about the awful distribution of SimCity. This time let's look at the servers.

That's harder then it looks, as they are down or full more often than not. I wish I was joking.

Let's take a step back. I say servers and SimCity in the same sentence and people look at me weird. "Isn't it a singleplayer game?", I get asked. Why yes imaginary person, it is. It has features though that allow you to sort of, but not really play with others. Also DRM and cloud storage, flash some more fancy words here, it's a really good feature, bla bla. In short, you have to be online, at all times. The game saves are all on the remote server, there is some sort of "global market" on the server and you can share a region with other players, which means their cities can help yours (and are more likely to screw up yours in truth). So in a region, one of 8, there can be a fixed number of cities, 2 to 16 and a fixed numbers of great project spots (1 to 4). So in the end, if you want to play just for yourself, you still have to be online on a server, no matter if you play just one city (which will be discussed in post 3) or not.

Each server has capacities. when you start the game, you actually start a launcher and updater, mentioned in the first post. And now you select the server you want to be on.

Why would you need to select a server? The server you select will store your saves. Also the global market is always for one server the same. No big deal, right? Wrong.

Servers have capacity. At this moment all servers are telling me they are full. So I have a single player game I can't play. Worse, even if one would be free, it probably isn't the one I spent hours in working on my city. So I would have to start new somewhere else. And then go back later. And I can't pull my game from one server to another.

So EA has managed to fracture people, remove their ability to play a single player game or continue their single player game where they stopped before.

Should you be able to get into a server and click play, the actual game starts. Funny though, if someone else logs in while you do that, the game might tell you that it can't talk to the server anymore. Should you survive that, you start a tutorial, which is forced on you on each new server. After that, if you start a new region and city, the game still might not allow you to continue and block your progress by saying that it lost the connection to the server.

If it looses the connection while you are in game, it has a short grace period before it can reconnect and it might work then, but more likely, it simply kicks you out of the game and you might loose some playing time that goes unsaved. Worse you probably can't connect back in because it's at capacity.

Another huge mess. But how to do it better?

First, no "servers". I shouldn't be made to select one, but instead have all servers have all my saves and allow me to connect to all my friends. Real time games like EVE Online can do it, so why not a single player with marginal multi player aspects like SimCity? Behind the scenes, these could be scaled up as needed. Or scaled down. No need for server splits, merges etc. If I play SimCity and my friend plays SimCity we can play together. No Server hijinks. Make it transparent and invisible for the player. They will like it much more.

Now once I press play, I need to have a guaranteed spot. It needs to work and I need to be able to play. A simple session handling comes to mind there. If chances are I will be kicked out after suffering through the loading screens, it just makes me quit the game and reload it. Very bad customer handling.

Recently (of this writing) was the release of a game people were waiting for, the next incarnation of the SimCity series. Removing any numbering (like in the past) and just calling it SimCity was in my not so humble opinion the first error, but it's not the first offender in this regard, nor will it be the last. The whole thing is one huge mess. The mess is so big of proportions that it needs multiple posts. I will try to keep my strong feelings out of it and concentrate on facts, but please excuse me if I can't hack it. You might understand at the end of these posts.

So here we go to the very first post, about distribution of SimCity. Now I personally favor Steam, and you can even get the previous SimCity 4 there, but after a quick look over, it seems like EA didn't want Steam involved for some DRM and DLC money reasons. Though nothing concrete is there, so it's anyone's wild guess. The end result is you are stuck with Origin, the Steam clone of EA. Now it's a clone, and a bad one at that. The fact of that alone could be worth a few posts, but let's concentrate on SimCity.

After buying, you get to see it in Origin, like in Steam, though it takes awfully long to appear. I have yet to see the purchase in my order history as well. On the web that is. In the client I can see it. Again, another issue. Now I have the game there but it only says "not yet released". Ok, I can live with that, I pre-ordered after all. But where can I see when it does get released? Not in Origin of course. I found the info hidden in the flowing text of the shop page. The release would be one minute after midnight the next day.

I decided not to stay up but leave my computer running, maybe it would download it for me? Certainly not. Giving me the option for it though would have been nice, as it's 12 GB. So I get up in the morning, check, no download, ok so I start it myself. I did a double take, it only downloads 256 MB? Both the strangely fitting binary number and the small size made me think something is wrong. Turns out, yes and no.

See those 200 MB are just the downloader. So you had to install and authenticate on the origin client, which in turn is "just" a fancy downloader itself. To download another downloader.

Well fine, let's install it. I clicked install. The bar went to 25%, then vanished. Strange. I clicked again, chose install again, the same thing happened again.

The EA help is no help at all. Their suggestion is laughable at best, insulting at worst. Of course it doesn't solve the problem and many more crop up with the same.

Being the geek that I am, I of course have more than one computer. Out comes the notebook and installing Origin there is done in a jiffy. Click download, the same 256MB downloaded again, clicking install and it works. After collecting my jaw from the floor I actually got it to run.

What you run though is not the game, but the SimCity Updater/Server selector. In that updater it actually downloads the true game, all 12 GB of it. So if you don't get it working that far, no download for you.

So I download a downloader, authenticate, to download another downloader, which I install, which then downloads the actual product. It takes years of experience the likes only EA can provide to dream up something as awful as this.

To make things even more fancy, the servers are down more often than not, so you don't even get to the update stage because the server is down. The servers issue is the next post, this one is long enough.

Now how to fix it? First show exactly when the game unlocks, or do it like Steam on the store page with a large block below the header that tells you in a count down how long you have to wait. That also eliminates the problem with timezones.

Next let people select a game for download when released. By that time, you probably know the size of the game and can simply select that to be reserved on the hard disk. Even better allow preloads, like Half-Life 2 did, by letting people download large chunks in advance and only the last bits at the end. Or a key to unpack the encrypted package. That way you can also distribute the load better.

Since you download it already, why not let it also install in the background? Need a "I agree" thing, let them do that before they even start saying they want the game (and it's components) on the disk in the near future.

Let's go on a small ride into the past. A long time ago, everything had it's place. If you meant server 7 had the file important.txt in the folder management, you knew that file was on one machine, in one place, on that metal box in the closet behind the stairs. Then people realized that this makes it vulnerable and added a second box that would backup that first one. Then the first one died and they spent a lot of time rebuilding it from the backup box. So they started to use load balancing and fail overs, once one box dies, the other takes over and nobody noticed, except the technical staff.

Fast forward a bit and now we have the thoughts behind the cloud. One of them is that although there is still metal running and storing your things, you don't really care which one it is. It's an abstraction between the actual machine running it and the application or service.Instead of one web server delivering your webpage you can have two or two hundred if you need, but you don't really care much about the details of each physical box, as you just need enough to make it work. One physical one, can house multiple virtual boxes as well.

Let's jump back to games. To this day, you still have game servers (say in some game like Battlefield) where the server loads a map (let's put a pin into instancing for now) and people connect to that server with that specific map. Early MMO games, like for example Ultima Online had a similar concept, where a server was responsible for a certain area. You could notice the server boundaries when you walked across them on the map. You could even exploit them, because monsters would not cross the boundaries. A lot of things happened with these boundaries but mostly they were annoying to the players when they encountered them. I bet they were also annoying for the guys at Origin back then, who had to deal with them. Some servers would do pretty much nothing, while others (the one housing Britain for example) were bursting. To recall Battlefield, you can find at any given time a ton of full and a ton of empty servers, but very few that are only half full. So either you are wasting servers that do nothing, or have overloaded servers. Eve Online has a similar problem. The star system Jita is the main trade hub for the entire Eve universe. Because the traffic is so high, they dedicated an entire server to just that one system.

The core troubles, if you break them down, are in segmenting areas (invisible server boundaries, or star system only, like in Jita) and wasted versus overloaded resources on servers. I might go into detail about the different kind of nodes and how else the system is setup, but for now I want to concentrate on these two issues.

To understand its approach, I need to explain two of the many node types the cluster has.

The user nodes take care of 3 essential things. Authentication and authorization of users, in other words, login, character selection and injection into the world, and lastly character creation. This last one is simply the act of having some character editor where you can make your own personal character, choose its name and so on. We ignore that for now, as after creation it will select that character and inject it into the world.

The zone nodes handle the actual interaction of the player with the world. It knows where everyone is inside the world, enforces the rules and does the calculations for combat and skills. The term zone is chose because a zone is not fixed.

So let's go through this how it would play out in action and I explain a bit more along the way.

There are always 3 user nodes or more online. When a player tries to login, the system will send that person to one of them for the interaction, using a load balancer, making sure the least busy server will get the most requests. If for example there are a lot of logins happening (or expected because of past experience) more user nodes are started. If there are few logins, user nodes are shut down.

So now the player is on a user node, sends the credentials and does the authentication bits. The user is now logged in and gets presented with the selection of a character and/or the character creator. Say this player has never used the game, they would get directly to character creation. Now they create the character, all in the client mind you. When done, the client sends the data for the new character to the server and there will be some checks as to the name, and so on. Assuming all is peachy, the character gets created and the same process starts as if the user would have simply selected a character earlier.

The user node checks where the character is (or should be) and calls up all the data the servers will need but also the client and transmits that. Then the node asks a controller node for the server that runs the zone this character is currently in. The controller will either locate that node, or simply commission a new zone on an existing zone node for this and return the details to the user node. With the character injected into the world, the user node hands over to the zone node and the player is inside the game.

Let's say the player is in a city. The entire city (and close surroundings) are one zone at the moment. Someone throws a party in a location in the western part of the city and lots of players attend. The traffic is rising on the zone node. So now the node starts shifting zones it handles around first. It hands off the second most busy zone first, then starts handing off other zones as traffic keeps rising. Once a zone node handles only one single zone and it gets still too much, a zone split happens. A split can be in two or even multiple parts. It will try to keep the largest masses in one zone, but it will fracture them behind the scenes. The player knows nothing of this of course. In the end, everything should at all times, run smoothly and fast.

With that the world can be very very big, but you only ever have to load the places where there are actually people. Also one server node can handle multiple zones where only a few players are, while the busy ones always stay fast as well. Close together zones can be merged if there is little traffic. Certain areas where there is a repeating pattern of medium traffic (think a road, or a pass through the mountains) can be configured to be one zone in itself and only be split when truly needed. Should there be fewer players, then you can shutdown hardware until needed again.

The experience for the player is the reason for all this. No splitting, merging, shifting or crossing boundaries of zones have any noticeable effect to the player. To them it will be one continuous world, no gates, no loading new levels and waiting, just going on and on.

In the last servers post, I told you about the general servers the game company needs to run its business. Now servers and non-servers really. Although there is metal somewhere behind it, usually, it's more a service that is gotten from the ominous could.

Web is even more such a thing. If you as a game company buy a physical webserver, with the intent to serve out a web page, you are doing something horribly wrong.

That whole web pages serving thing is pretty much licked. You can choose from thousands of locations and providers to host with. Services range from "yeah, we are online" to full Orwellian control.

But of course we are not talking about the blog that has 50 hits in a year, but one that gets thousands of hits every hour. You also have trouble around peak times, say the release, or big additions, christmas or simply the evening of your most busy customer group.

So, to give you a short overview, the webpage is coded in PHP, has a MySQL database backend, varnish front end caching, a redis session storage and item caching, a rabbitmq message queue and that's about it. Now there is a small cluster of servers that just run the database and the redis instances. Each web frontend node has it's own stack with php, webserver, varnish in it. Message queue and workers run a smaller cluster on it's own.

The database are 4 servers. If there are too many connections, or too high a load on existing nodes, more get added.

The message queue is 3 servers, running the queue as well as the workers on it. More are added if the queue is too long or loads get too much.

The frontend nodes are 3 servers. More get added if the load is getting too high.

And when I say, get added, I mean the wonderful magic of auto scaling. If certain levels are exceeded, the cloud will create new instances. Once the levels fall down below certain levels, the cloud will shut down instances, to save money. That way you can quickly scale up to 5 or 50 times your traffic and scale down in no time again after getting over the big rush. It also means if an instance is unavailable, crashes or something else happens, the cluster will add a new one on its own.

So your website suddenly learns to manage itself. Anything you do to enhance it, will mean it takes longer for more resources to be provisioned for it. Now you get into the position of where an optimization that will mean you can handle the same amount of people with 5 instead of 6 instances, real traction. You might get into the habit of spurning over optimization and rather concentrate on making handling easier for the customers, which in turn means more customers and happy customers. Decisions over optimization versus everything else, get a real fiscal meaning. Another instance running is pretty cheap.

Now the whole thing does a self healing self scaling mambo, making sure your page is blazing fast, no matter what comes. You need copies of that, at least 2, for testing and development. But those don't need to run 24/7 and you don't have to run as many of them (2 of each are enough). Again, you can shut it down over night and the first user that needs to run them, can press a button to power it all up again.

Of course you can also have a local copy of this, which I hinted at in the testing cluster, on my first servers post, but there are always things you can only truly be sure are exactly the same, when you use something that is exactly the same. So an online test installation is a must.

It's prudent to keep frequent backups of the database around. Those find a perfect home on the backup server in the company. Of course you also need to back up any additional data, for example uploaded files. For the later, some sync daemon software that runs in the background and keeps your data synced over the net with a backup server makes this a breeze.

On the internet everything is somewhere on a machine. If it's big, it's on multiple machines.

The game knows 3 different kinds of server environments. The first you see is probably the website. I will talk about that in another post. The second is the game cluster. That one get's it's own post too. This is about the third one, the servers we need for the company to run.

Let's talk about the non-servers first. Back in the day everyone had to have a ton of software and servers to do even simple things. Email, contacts, calendar and documents, all needed individual and/or linked systems. Today you can do most of that externally, cheaper. Google Apps for Business is one such thing that does all of that. You can have your own domain, lists, groups etc with these tools, all simple and fast to setup and reasonably inexpensive. Now this is good for doodles, documents, mails and such things, just the average day to day stuff pretty much anyone needs. It also has the nice effect to be a "per head and month" cost so it simply scales with employees, no 5 digit server and license fees to pay.

For development you need a bit more. Github for example is a good way of a "non-server" here. You can buy small numbers of repos that are private for not much, or go with a self hosted enterprise version, if you are paranoid, or need more speed. Of course there is other version control, but personally I'm most happy with Git at the moment. Github also includes issues and a wiki, which is good for basic documentation and issue tracking. It's not very handy for full on project planing though.

In these agile times, you need agile tools. Again, here you have the pick of the litter, which is tightly tied to your method, Kanban, Scrum or anything else. You can forgo digital and just do physical cards. Or make an entirely digital board. The jury is still out there, but I found it invaluable to be able to search through ideas (possible features) and then reorder priorities, down to single priorities, in a linear fashion, which is first and which is number 150. Add to that your common metrics, like days, hours or story points and you can see a lot clearer. In the same vein come metrics. Being able to have it somewhere digital, you get most metrics for free, by just having it in the computer. Let's assume another non-server tool.

So far there is not a single server locally. This is not good. No it doesn't have to do with paranoia, but there is a significant amount of data, especially in a game, that you don't want to outsource on a storage like Google drive, because if everyone has to download it from there, they could just as well download it from another source site on the internet or would be faster handing a USB stick around.

A file storage server is surely something you will have local. What you put on there is to be shared. You might keep a couple of sensitive files in a more secure place, but the high-res sources of textures, find their ideal home there, along the latest builds for everyone to test. Although that one should go into another category, read further down about the test cluster. This is a very mission critical piece of equipment. It would be good to have this redundant in a fail over configuration. If one breaks, the other jumps in and takes it's place, keeping the files in sync between the two.

Where there is files, there is backups. Backups are important. Backup early, backup often and be paranoid with the backups. You can use that to backup real game data as well, away from the game cluster. Simply use a dedicated machine for the task of keeping backups of everything, including the archiving and save encryption of files, which you want to do when taking something out of the office. Which is what you want to do. In one way you want at least one physical disk drive to take out of the office in a rotation, usually done in a 4 piece dance, so you have a physical storage device you can get to at any point. You keep one always connected to the backup server, one always not connected to the server in the company, one always out of the house and one always in transit (either coming to the company, unconnected, or leaving the company). So at times you have two disks connected to the server, but still one unconnected disk laying about (the transit one). And when you remove the second disk from the server, you have two unconnected, one of which will go home with you. At home you store the new arrival and take the one you had home with you to work again the next day. You could make it more secure by using two rotations, for two locations, but instead I simply use the power of the cloud. Amazon Glacier is a storage solution for longterm storage. Upload is free and storage comes to one cent per GB and month. Same as with external disks, encrypt what you put online. The storage is safe and cheap relatively, adding an additional safety net. Once a backup is too old, you can simply delete it again.

Telephone systems these days also need a server. Well, this one depends a lot on means and needs. It's one you can't get around very simple.

A note on non-servers is being able to reach it. An external service can be down. Lines can be cut between your computer and that external service. That is one reason why many want to keep their stuff as close to the vest as possible. In my experience that has not really paid off, ever. You want to make sure the things that make a full stop to your system are redundant, say a file server where your source files are. Your internet connection should have a backup (most ISPs can deliver that) in case your main line dies, usually with just a reduced bandwidth. Most professional online services have near to zero downtime, or long in advance scheduled ones, you can work around with, or simply go for a coffee break until it works again. So don't waste too much time thinking about that part.

A very real one is privacy. You have to understand your online settings, your sharing of files and folders and keep an eye on it (people that left the company no longer access to those resources, external users, etc). Also the company you give your data to, can see it as you give it to them (except backups you encrypt). I have not heard of abuse in that field, but it is a potential possibility.

On the other hand, non-servers give you in most cases a predictable and linear growth in costs. No huge administration and operating needed. You can spend time and money on the game at hand, not on more hardware.

Which brings us to the last point, the testing cluster. Here we replicate the website and the game cluster, on a few beefy machines with virtual boxes running on there. Technically, everything needed to run the website and game is rebuilt as virtual machines inside that cluster, including fail over nodes, loadbalancers, caches, message queues and so on. That also includes download and update procedures for clients. That way you can get new clients out to testers internally, testing the entire release procedure along the way, eating your own dog food. This will make it possible for you to test new updates, experiment with new software and before you even touch a single machine in a live environment, know already that it will work.