#inotify

Serveur dédié : résoudre l’erreur “tail: inotify cannot be used, reverting to polling: Too many open files”

Ce matin, je me suis aperçu que le serveur était un peu moins réactif que d’habitude.

Ni une, ni deux, je lance le terminal et commence par vérifier les fichiers log. Un message attire alors mon attention :

tail: inotify cannot be used, reverting to polling: Too many open files

C’est bien étrange puisque très peu de services sont censés lancer des tail. Nous allons donc lancer quelques…

If you are using pyinotify, you might be familiar with the error "No space left on device (ENOSPC)". What you have to do when this happens, they say, is to increase the maximum amount of watches on your system.

You can see the max number of watches with

sysctl -n fs.inotify.max_user_watches

And set it to a new value with

sysctl -n -w fs.inotify.max_user_watches=16384

But wait, there's more. inotify using processes have a link to "anon_inode:inotify" among their file descriptors in /proc. So, you can use the following command to identify them:

find /proc/*/fd/* -type l -lname 'anon_inode:inotify'

To have more information (thanks to good folks here):

When writing code which needs to be built before it can be used, whether that's a transpile step like ES7 JavaScript to ES5, or a compile step like with Go, you're likely going to want to do this when a file in your application tree is modified. There are a large number of project and language specific tools which were developed to tackle this problem but did you know that there are system level packages available that you can use across all your projects?

Introducing inotifywait, which is an efficient and easy to use cli tool which uses Linux's inotify interface to watch for changes to the file system. And fswatch for those on OSX. Most of the language and project specific tools are built using wrappers around these two tools.

If you're like me and don't like to add more build tools and layers of abstraction than necessary then you're probably already using Make to build and develop your application, and you'll be happy to know that using these tools with it is trivial. Make has no way to know when a file has changed until the next time you run make so because of this many have tried something like the following which will run the build target every 2s.

watch -n 2 make build

Or they will build the loop into the makefile.

.PHONY watch watch: while true; do \ make build --silent; \ sleep 1; \ done

This works, but it's performing a lot of unneccesary work being run in a loop especially since the file system is able to tell us when a file or directory has been modified using inotify. Instead of automatically looping we wait for a file system event that we're interested in and then run our build target. In the following code we're able to create a make target in our makefile which will watch for file changes under our specified directory recursively.

.PHONY watch watch: while true; do \ inotifywait -qr -e modify -e create -e delete -e move app/src; \ make build; \ done

This works by creating an infinite loop which is started when you run the watch target. Before the first loop can finish it hits inotifywait which sets up listeners on all of the files and directories in your app/src directory. These listeners are waiting for any files to be modified, created, deleted, or moved in or out of app/src/.... When a file or directory changes inotifywait lets the loop continue triggering the call to your build target. That target executes in its entirety building only the file(s) which changed (assuming you've properly set up your makefile) and then the loop starts again with inotifywait waiting for those files to change again.

Working on a lot of resources for Endless, so decided to package the structure up for others to use. While not the most integrated or powerful system, it’s very quick to setup and relatively simple to use.

As it uses bash and inotify, its currently written for Linux systems. I’m sure it can be tweaked to work on OSX via a different mechanism, and probably rewritten in PowerShell or something for Windows.

The attached scripts are just examples of how to implement it - they’ll need to be tweaked for your specific project, since you probably wont have the same resource system or format I do.

Overview

The goal is to save a file in an editor (such as Krita), and a few seconds later see the changes in the game without having to run build scripts manually. A video of what I mean:

In my game repository, there are 3 main folders that are important for this:

Resources/ : Location of all the engine readable assets go. It has a final compile step with WolfRez, but its only straightforward conversions. You can think of it as final assets.

ResourceSources/ : The source files for assets. For example, I do all my art with Krita, so the kra files are in here, while the generated PNGs go into Resources.

Build-Linux-Clang-CPP11-Ninja-Debug/ (or whatever) : Build directory of the game which can compile, package, install the game.

Now in my case, I also have the ability for my game to automatically reload resources when they change : but thats outside the scope of this setup (since its very engine dependent). If you dont support that, worst case you have to restart the app as an extra step to see your changes.

TopLevel - rebuildResourcesOnChanges.sh

This script basically is run once in a terminal and forgotten about. It monitors the path hardcoded inside it using inotifywait. If any files change, it’ll call fileChange.sh in the given directory. If that returns 0 (success), it then rebuilds the app. Pretty straightforward.

Directory - fileChange.sh

Every directory underneath ResourceSources can have a fileChange.sh script. If it exists, its called whenever a change is detected in that directory. The script then does whatever actions are needed to build that file. For example, the one i’m using at the moment for a treasure chest graphics is:

#!/bin/bash HACKERGUILD=$HOME/Repositories/Hackerguild TEK=$HACKERGUILD/Endless/Tools/Scripts/tek.rb RES=$HACKERGUILD/Endless/Endless/Resources/Endless.tek/ if [[ $1 == "TreasureChest.kra" ]]; then ${TEK} convert TreasureChest.kra ${RES}/Entities/Common/Sprites/TreasureChest/TreasureChest.png exit 0 fi exit 1

If “TreasureChest.kra” changes, it runs a helper tool I have (tek convert) to create a png out of the file, and returns 0 to rebuild the app. Otherwise, it returns 1 (nothing changed). Other fileChange scripts do things such as slicing a tilemap up, or converting a song file. This flexibility allows it to basically perform any command when a file changes.

So high level:

App saves its file

rebuildResourcesOnChanges notices a change

fileChange does whats needed to process it

rebuildResourcesOnChanges rebuilds the app

App detects the file changed, and reloads the resource automatically

Nice and easy.

Scripts:

Fixed: Is there a working Linux backup solution that uses inotify? #development #fix #it

Is there a working Linux backup solution that uses inotify?

It takes forever to back up. Before we can trust btrfs or ZFS to backup incremental snapshots, wouldn’t it be nice if there was a daemon that used inotify to keep track of which files had actually changed so backups would run more quickly? Where is this program?

How do I backup my Linux box without having to crawl the whole filesystem…