DevBlog

“Just another IRC crash.” We heard that way too many times by now. If we are lucky, the issue is solved by now with todays update. But god dammit, that was a trip!

As some of you might or might now know, FAF is running on a single dedicated machine where all applications are managed in Docker containers.

A Docker container is like a tiny virtual server inside the main server and it’s usually supposed to run only one service (like our MySQL database or our lobby server). The idea behind this is, that this containerization solves the issue of different applications requiring different software dependencies.

Docker containers run on images. An image is a predefined set of files that contain the operating system (like a tiny ubuntu) + all the files required for the applications. Docker is so popular by now that almost all applications offer Docker images or some 3rd party developer takes the application binaries, builds the image and makes it runnable for us.

Unfortunately none of this applies to our IRC software bundle. Our irc services contain of two applications: unrealircd as the irc server itself and anope as the additional irc services like chanserv used for authentication, moderation etc.

So the developers don’t offer a docker image. Furthermore they don’t even offer binaries for their application. That means, you need to compile the applications from source. When compiling the ircs apps from source they ask us a lot of questions like which modules to use or what settings to enable. However answering questions is not a thing that can be done in an automated way right? For each version change you need to check for a change here. In addition a compilation of these two apps took around 30 minutes.

Fortunately there are people on github who did all this, so we could rely on it. For a moment. So we forked their github repository in 2016 and adapted it to our faf needs.

And then on faf side the thing happened that always happens if everything is working: nothing. Two years nobody in faf cared about the irc server. Unless recently it started crashing regularly for no reason.

When I started to investigate this issue, the first thing I noticed was our unrealircd version. It was 16 releases behind. Even though they were minor versions a lot of issue seem to be fixed in this. But minor releases are supposed to be full compatible. So time for a fast update right?

Wrong!

First I checked our original source of the docker image. They were also stuck on the old version. Furthermore they developed some other stuff that was now fully incompatible to our setup (and is not even working as of today). After a few tries to make our setup compatible with theirs I gave up. The differences were to big, the benefit too low (since they didn’t update their docker image anymore as well). So I digged into the build process.

After multiple compilations I was sure that nothing had changed in the build process. So just get the new source code, compile it and run it right?

Wrong! The new version suddenly refused to work with root permissions. While running as root is a bad idea in general, it’s not that important if you run it in a docker container. However, the unrealircd devs decided to enforce this and give no option to turn off this check. Rackover was so kind to deliver the right patches to the Dockerfile so we could run it under a different user. And she promised that everything was running well on her machine.

So I took her changes and threw it on our test server. Again I was greeted by the already know error message “DON’T RUN AS ROOT, DUDE!” (ok not a real quote here, but you get the idea). What the heck? So I double checked the changes and ran it on my local machine. And there it wasn’t working as well? So was rackover in delirium when providing the fix? No she was not. There was a tiny difference. Rackover ran their changes with docker while I was using docker-compose - a management layer on top of docker which we use on all of our servers. An investigation showed, that we had overriden the user with root again. Simple fix on my machine and it was starting! Awesome. So finally run it on the test server.

Boom! Error message. WTF? It claimed that an ssl could not be validated. So just check the files? Not so easy. While you can enter a docker container like any other operating system, it needs to be running to do so. But the container ends, as soon as the main applications stops. And since unrealircd crashed instantly, the container ended instantly. Workaround: don’t start unrealircd, just read unlimited from /dev/null. Silly but it works. So as it tourned out, one of the certificate files was missing. But on my machine it was there? Reason: docker-compose trolled again. We setup the config folder to be mounted to a directory on the root server (outside the container). And the file was missing there, as it came along with a newer version of unrealircd. So copy the file run it again! Gnaaa! Now the file permissions weren’t set since unrealircd no longer read them with root permissions. Some chmods later the server was finally booting! Using some dev-faf-client I could connect. I started QAI and irt also connected. Yes! Go along and deploy to prod? Naaa, better get some testers along and try it out. Success??!!

No. Many of our test users couldn’t connect. Whaaat? Why could I connect? Why could QAI connect? Helplessness and despair was rising. But then we compared ip addresses. As it turned out only IPv4 users were affected. WTF? Usually it’s the other way around?! But a port scan confirmed. IPv6: open, IPv4: closed. On production both ports were open.

Sheeo to the rescue. He noticed that the new unreal version was listening internally on IPv4 and IPv6 while the production server was only listening on IPv4. So why was the IPv6 port on production open? This again was docker-compose magic. When you open an IPv4 port only, docker-compose will internally route IPv6 traffic using an internal gateway (so to the docker-container it still looks like IPv4 traffic). And if you open both ports in docker-compose IPv6 is unsupported or broken. Whatever, it’s a known issue with no simple docker solution. So the solution must be in unrealircd. Why did the port even open out of the sudden using the same config? I don’t fucking know. This is the kind of stuff I expect not to happen in a minor release. There was also no easy option to turn it off in compilation. However I found out a simple workaround. I could make ports listen to ip 0.0.0.0 instead of *.

Finally all users could connect. Afterwards I packed all the changes into git, changed the configuration on prod and made the release. 1 hour later 450 users online. Looks like our odyssee finally ended after 8 weeks of investigation and try&error.

What did I learn in this 8 weeks?

1. The benefits of docker get lost if you rely on unmaintained images.

2. Update as soon as possible. The more releases you skip, the more problems stack up.

3. Docker-compose can fuck up your container if configured improperly.

4. IRC is a dinosaur that should finally extinct.

That’s it for today! If you want to know what comes next on my roadmap check out https://trello.com/b/aYXcrRBF/brutus5000s-faf-task-queue

Today I got an interesting report of a moderator. Our moderator tool (short name: mordor) crashed directly after login.

So unfortunately mordor had no error handling for making calls against the api, since I programmed it under the optimistic assumption, that no errors will come up on tested api calls (mordor was built using ugly shortcuts since it is a tool for only few users & a lack of dev time).

Ok so booting up IntelliJ and starting the app I could reproduce this behavior. The logs in IntelliJ now showed that the mod client threw an exception because it couldn’t handle responses of our API with a HTTP status code 500 - internal server error. This is highly unusual, since the api has pretty good test coverage. But something was up.

So time to ramp up putty and login to the server. Docker logs being helpful at always show us the cause of the API error:

2018-07-20 20:25:10.331 ERROR 1 --- com.yahoo.elide.Elide : Internal Server Error com.yahoo.elide.core.exceptions.InternalServerErrorException: javax.persistence.EntityNotFoundException: Unable to find com.faforever.api.data.domain.Game with id 8275969

Hmmm. If the game with id 8275969 does not exist, then why are we querying it? Climbing up the stacktrace in mordor I found that the error was thrown on querying teamkills. So are there invalid teamkills? This reminded me of an issue I had a few weeks ago and resulted in a github issues (https://github.com/FAForever/server/issues/339). A few queries later I was sure: the cause is not the teamkills. The cause are missing games. But we only noticed because in one of these game a teamkill was reported! This is severe bug in our server and is probably related to complaints about games not being rated.

So what do we learn here? A simple crash on one application can have it cause in a completely different application. In this case the root cause was actually server -> db -> api -> mordor.

Since a short-time solution to mordor was required, the only way to deal with this was to finally add error handling (which should have been done in the first place). So at least the moderators are back to work. The original server cause is still unknown :(

Overall work was around 4 hours. I guess there are more interesting ways to spend a friday night.

- Brutus5000

Planning the Galactic War Server

As some of you might know, Brutus5000 is working on the server side of Galactic War (GW) in his free time.

From a technical perspective, the GW server will be a separate software, allowing us to start/stop/update it without affecting the FAF server software. 

Nevertheless, the GW server needs to be able to communicate with the FAF server in order to see who’s online and have the server launch people’s game and have them connect to each other.

This raises several technical questions:

How does the GW server (or any future, external service) connect to the FAF server?

What communication protocol will be used?

How does authentication work?

Which messages do they pass forth and back?

This is a common problem in software engineering and a state of the art solution is OAuth2 together with WebSocket.

The new V2 Server Protocol

Since I’m not touching the current server implementation anymore for many reasons, I implemented what's required into the upcoming “Java Server”. Along with it, I implemented what I call the “V2 Server Protocol”.

But what’s a protocol worth if it’s not documented? That’s one issue with the current one; there is no documentation and you have to look at the server code to figure out how it works, down to the byte level. 

This is not the case for the V2 protocol. Not only is it much more intuitive (plain JSON over WebSocket) but also does it come with a tool that generates the documentation directly from the source code - ensuring that it’s always up to date.

Have a look at the V2 server protocol documentation.

“That all sounds boring and I have no idea what you just said” you migh say. I get that, and I’m sorry. However, I need to emphasize that - at least from a technical point of view - this is a big step forward for FAF as it basically allows everyone to rather easily write applications or even websites that interact with the FAF server.

Think “plugin-ins for FAF”

Sample use cases

Here’s some inspiration of what could be built:

A web-based admin tool that allows seeing who’s online as well as sending messages or kicking players

Displaying real-time information about online players and/or games on the website

A betting system

Any meta-game someone comes up with

Chrome or Firefox add-ons that interact with the server

A web application with real-time statistics 

Integration with IFTTT or Zapier

And most importantly, it allows GW to proceed.

You’ve probably all heard of GDPR and hopefully received an email from FAF about it. What you probably don’t know is how much time the whole thing cost us, to do it “right”.

The journey started with “we want to send a GDPR info mail to all players”. How hard can it be? Well, how do you send an email to 200k recipients? Certainly not using your personal Gmail account. So you need a paid service like Mandrill (which is rather expensive) or Amazon SES (probably cheapest), or whatever.

Or you send emails for free from your own server and risk being seen as a spammer and getting blocked.

Along with sending out the GDPR email, we also wanted to introduce the FAF newsletter to increase communication with players and player retention. So I planned to set up “Mautic” which is an open source email marketing tool. The same tool could also be used to send out a nice looking GDPR email.

Setting up and configuring Mautic was rather easy, but still took several hours (I had to learn how things work first).

Creating a decent looking email template took another few hours, especially since first I had to figure out how it needs to be created and because the trial-and-error turnaround was rather slow. In the end, I was happy with the result:

Next task was to feed all our email addresses into Mautic. Sounds easy enough, but took me quite a while and long story short, I had to split the CSV export from our DB into files with max 9500 lines and upload ~20 files individually. Cumbersome, but it worked.

Now, how can we allow users to sign up for newsletters? Another long story short, the best solution seemed to be to create a custom “preferences page” that allows users to select which categories they want to describe to:

So that’s done, too.

Now, which service do we use to send the emails? So far, we’ve been using a Mandrill account for which Visionik was still paying. Another long story short, after a few thousand emails none were going out anymore because we probably hit some limit which we couldn’t check because we didn’t have the Mandrill login.

Which by the way, also blocked our forum notifications, registration emails and password reset emails. You can imagine that this caused some additional pressure which I initially relieved by creating accounts manually in the database per user request. Totally not a waste of time.

Alright then, let’s finally (and for the very first time) use our own email server, which gladly I already set up using the great open source software Postal:

Let’s hope email providers won’t instantly block our email server’s IP address because it sends out thousands of emails.

Finally, all GDPR emails went out - at a rate of about 40 - 80 messages per minute. With 200k email addresses, you do the math. Only a few providers blocked us, some of which I contacted and asked to get unblocked.

Setting up Mautic and starting to communicate with our players via email has been on my list for a long time - thanks to GDPR the pressure was finally here to make this happen.

At this point, I’d like to thank Sheeo as well who spent a noticeable amount of time writing the privacy policy and terms and conditions that everyone needs to accept now.

Today I published the initial version of the long-promised tool for our hard working moderators. Until now all ban requests, ladder pool changes, question regarding finding your username or email have been posted in our slack moderator channel and processed manually by one of our few server administrators.

This was not only a lot of unnecessary work, it also led to an unapreciated delay (especially ladder pool or bans).

After selecting a user you can gather all the relevant data from the tabs below.

The ladder map pool is very ugly yet, but was the first functionality implemented (coincidence?).

The recent activity feed finally allows our moderators to work proactively.

The application makes full use of our json-api compliant Java-API. In the long run, we want to put the data objects and connectivity code into separate packages, so you can easily write your own tools to create statistics or whatever you want.

Until then, don’t get involved into too many teamkills, because from now on we are watching you!

In the last post https://faforeverdevblog.tumblr.com/post/158977673087/2017-3-29-gw-game-code you could see I managed to get the GW game mod launch a game. It was still pretty buggy but it was a start.

I made some progress since that and after some more refactoring and code cleaning it seems that the mod should be fully working now. Local multiplayer game ran fine.

So I moved to improving things. Starting with fairly easy ones. One I want to present you (since you won’t get too many pictures from brutus and his server code :D)

The reinforcements beacons

Each faction now has a unique beacon. In the old version all factions had the Cybran one. With that we need to thank Exotic_Retard for making the model for the Seraphim beacon and CookieNoob for making the unit previews that will be shown in the build bar.

Compared to the old beacon the models were slightly increased in size, fixed all kinds of stuff like the strategic icon, hitbox, selection box, sounds when selecting, destroying the beacon, adding a wreckage when it’s killed and maybe some more that I already forgot.

Expect more to come in the near future,

There are many ways in which the client code is not as nice to work with as we would like. Let's talk about one of them - how the client manages things the server gives it.

Intro

There are three kinds of things that server tells us about - games, players and IRC users. We receive these when the client is launched and they are added, removed and updated throughout the client lifetime. The way they are handled right now is unfortunately quite a mess - bits of logic concerning them are put alongside widget code or in global objects and dicts. This makes it pretty hard to make changes to the code, as a change to a widget might cause a player to no longer be reported ingame, or for a game to no longer show that the player is playing it!

To clean it up a bit, we intend to move all the code related to players, games and users into one place.

Flow

All the info about players, games and IRC users comes from the server in one way or the other - to track that, we'll have Player, Game and IRC User objects that receive this info, change their state as required and emit signals. We'd like to be able to keep track of all players, games and users, too, including getting notified when a game starts, for example - for that purpose we'll have three more (Player, Game, IRC User)Set classes that will keep track of things and signal when things are added and removed.

Relations

Of course, that's not all - we'd like to get notified when a user joins or leaves a game, or when a game changes players, or when an IRC user that corresponds to some player appears. To keep track of that in an organized manner, we'll have two relation classes that will keep track of relations between games and players, and between players and IRC users.

Signal order

To make the code more simple (and make it easier to use it), there will be an order imposed on the emitted signals. Relation change signals will be sent first, then object set signals will be sent, finally object update signals will be sent.

More

You can find detailed info on FAF client github wiki here. If you have ideas or want to help implement it / write tests, do chime in!

Last year’s effort on the new API is finally bearing fruits. I could catch a preview of the new map vault in Downlord’s client. The mod vault will use a similar layout. 

Downlord is still polishing it, but it is definitely coming this summer™

Today I’ve implemented quick tips to the game loading screen.

Hi everyone! IceDreamer here with some news about the next big bugfixing and features patch:

The Deployment API route has been completed! This means that I or my team can deploy micro-fixes for problems found without needing to wait for a person with server access!!! Woohoo! :D

The patch is called 3680 and has been in development for some months now. We’re at the point where I need the entire community to band together and play real games using the FAF Develop mod (It’s at the bottom of the list of Featured Mods)

There is a forum post HERE

 where you can list feedback about errors you encounter, bugs, or just general feedback

The Changelog can be found HERE

Part of this patch is what’s called a “Refactor”. Basically, to make modding easier in the future, we’ve moved some files around a bit. Unfortunately this means that some mods will be broken by the change. Below I will put instructions for mod-makers to be able to get their mod ready for the change.If you notice your favourite mod doesn’t work in the patch, LET THE AUTHOR KNOW!

Modders – We have refactored away the /modules directory, moving the files within it to appropriate places inside the lua directory. You can see the changes made if you scroll down THIS list, looking for commits marked with “Refactored … out of modules”

Have you ever checked how many megabytes you need to download if you want to watch an old replay? It’s around 60mb.Why does this happen? Every FAF version has a set of files it needs to run. But a single bit of change, and you need to redownload the whole thing.

Now imagine a new FAF version released: 10.000 unique players download this patch from our server. That’s a total traffic of around 600 GB. For one new patch. Now people still watch the old replays. So they download the old version, than the new, than the old... so even though we don’t know the exact numbers, it’s probably more than 60mb per player per month. And traffic of that scale costs money.

In the last 3 month I have been working a new patching mechanism called BiReUS. BiReUS is a general purpose patching tool, that can create binary patches of one ore more repositories on a server side and download and check out any version on the client side. It’s written in Python3, open source and available on github.

BiReUS compares files on binary level using the bsdiff4 algorithm. But while bsdiff4 can only be used on single files, BiReUS can be used on whole folders of files, and even more it can unzip zip files and compare the contents, patch them and zip them up again. This makes it possible to create FAF patches with a size of around 200kb.

Some of you may ask, why we don’t use git instead. Well, git has 3 main issues:

1. It downloads the whole repo before you can checkout any version. So instead of downloading the 60mb of FAF you need to download 400mb of FAF repository. As we have a loot of player with slow internet connections, that should not be the way to go.

2. git doesn’t work very well with binary files. Every new version of a binary file increases the whole repo size way more than a simple bsdiff4 patch.

3. Last but not least, the most important issue: Most repositories except for FAF mod itself don’t have a github repository with correctly tagged releases back to the first version it was introduced in FAF. This would break replay compatiblity to all replays that aren’t tagged in their current github repository.

Yet we are able to re-create patches for all features mods that are stored on our servers.

So how far are we from using it in production? BiReUS itself is stable and tested against the whole FAF repository. On my machine it took ~6 hours to create all patches once and the whole folder of patches took 4 GB.

There is no official decision to use it, but I hope we can all agree on that in the next dev conference.

Yet we are lacking client implementations. The current python client is not ported to python 3 yet, though I hope this is a new motivation to do so. Downlord’s java client can’t use it either, as there is no Java implementation yet. But with the documented formats I am pretty sure this is only a question of weeks. And last of all we still need to do more testing, because we don’t want to break things.

I am pretty sure, we all want to get this going before we switch to the new servers, Maybe we can save some money on bandwith and put it into some more powered hardware instead

If you are interested in helping out, as always: join us in Slack or fork it on github.

While Brutus is still busy working on the server code for Galactic War, I took a look at the game code that was left. After several hours of rewriting and debugging I got it to launch.

I had to use some dirty hacks to get it working in single player as it was done to work only when launched from the client. Now, since I dont know that much about that, I need to find out how to make it playable over LAN so some proper testing can be done. After poking a bit more in the code I was also able to tell Brutus the game requirements, what information has to be sent to the game, so he can implement that. There is still a lot of code cleaning and rewriting to do to make it possible to add new things as the current code is very messy, but that should be now easier as it at least works.

As you know the tutorials in the client are not in the best shape right now. And outside of the client there’s no source that would keep all the tutorials together. Most up to date would be a wiki, but not many people actually update it or visit it as a source of information. Not everyone is intrested into reading pages and pages or watching dozens of videos to get some piece of information about how to play this game. So I decided to use what I’ve learnt from working on coop missions and make a new set of tutorials out of it. You can watch a preview of one of them:

As you can see, the first group of tutorials will be basic build orders for the most known 1v1 maps. They already expect the player to be a bit familiar with the units but they could still teach a lot of things.

I made a mod for the tutorials which makes adding more very easy. It does require working Map Editor and very basic skill in lua. Most of the changes required to the coding is simply replacing unit names according to what you named them in the Map Editor.

Having these tutorials working there’s still a need for more basic or advanced one. Some that would explain the basics like introduce all kinds of commands or all elements of the UI. From the advanced ones: how to react to special situations in the game.

This is the part where I could use a help. Write down everything that we should teach player, preferably split it into categories where each category would be one tutorial and then I could try to code it. Also if someone feel like coming up with some nice design for the client tutorials page I dont think you’d be refused.

You can find everything what I have for the tutorials in this google document: https://docs.google.com/document/d/1dZShYfOeiSvRSTmQZ6G3rHPZ0UdpsgG6GqvgYIDYGPY If you want to help out, even with the non coding task, be sure to join us on the Slack!

Here’s some info on what’s happening with the client right now.

For now most client work involves cleaning up and refactoring existing code, although we do have some features planned as well. We will probably make one more 0.12.x release - it will fix some themes breaking in 0.12.4, refactor (re)connecting to the lobby (if you’ve ever used the logo in the top left to disconnect / reconnect, it should behave more consistently now), and give you a fancy update dialog where you’ll be able to pick a (old/stable/beta) client version you want to update to.

The next big feature is the ICE adapter - we had a successful testing session last Monday, and we’re working out the last remaining issues / bugs. After that we’ve got plenty of other features queued up. Apart from some significant under-the-hood changes, we might have:

- IRC user friends / foes,

- Browsing subdirectories in the “Local replays” tab,

- Themable login dialog

And more in the future. We encourage you to try out client prereleases - any testing and bugreporting helps us out immensely!