Automatically kill an application if it takes up too much memory on Linux / Debian / Raspberry Pi
I use a Raspberry Pi 3B to do video encoding ā see this post.
However, under certain circumstances, ffmpeg might end up ballooning in memory, and this can get serious very fast given that a Raspberry board has a relatively small amount of RAM.
In my case, this can happen if my local network goes down in a specific way... I think ffmpeg ends up not getting an instant refusal / timeout on connecting to the server and buffers forever instead. This could also potentially happen if whatever your camera is pointed at suddenly becomes much more complex to encode, and your options cause ffmpeg to buffer the input instead of dropping frames.
So I figured, maybe I could look into running a script that kills ffmpeg as soon as it takes up too much memory. I found a base of a script somewhere on the Internet, but it didnāt work; hereās a fixed version.
TOTAL=`cat /proc/meminfo | grep MemTotal: | awk '{print $2}'`
USEDMEM=`cat /proc/meminfo | grep MemAvailable: | awk '{print $2}'`
if [ "$USEDMEM" -gt 0 ]
then
Ā Ā USEDMEMPER=$(( (TOTAL-USEDMEM) * 100 / $TOTAL))
Ā Ā echo "Current used memory = $USEDMEMPER %"
Ā Ā if [ "$USEDMEMPER" -gt 80 ]; then
Ā Ā Ā Ā killall -9 ffmpeg
Ā Ā fi
fi
To be honest, this makes the title of this post inaccurate: this script evaluates the total active usage of the entire system, not how much one application takes up by itself. But generally, since youāre trying to avoid a āone application makes everything else go OoMā disaster scenario, this is for most intents and purposes identical. And you could change it to use a different memory type, maybe, but MemActive is the one that would be relevant to me, since I leave a tmpfs (ram disk) worth a couple hundred megs running on the board.
And, of course, this script explicitly runs a killall command on ffmpeg, notĀ āthe application that is taking up too much memoryā. Itās not a smart script; it assumes you already know which application(s) is susceptible to misbehave.
So, weāve got our watchdog script. Letās save it at /home/pi/watchdog.sh
We could add a loop right in the script but thatās probably not the most elegant solution. We could also useĀ ācronā, but the instant I learned it was possible to wipe all of its data simply by calling it with no parameters, I figured I would stay away from that thing as much as possible.
Letās use the services system instead! (Itās systemd & systemctl.)
We have to create the service definition and its timer, separately.
First, the service: letās sudo nano a new file into /etc/systemd/system/ffmpeg_watchdog.service
[Unit]
Description=FFMpeg Watchdog Service
After=network.target
[Service]
Type=simple
User=pi
WorkingDirectory=/home/pi/
ExecStart=/bin/bash /home/pi/watchdog.sh
Restart=on-failure
[Install]
WantedBy=multi-user.target
In case youāre wondering, ExecStart hasĀ ā/bin/bashā in front of the path to your .sh file, because it doesnāt magically know that this is supposed to involve the shell / bash.
Now, letās create the timer, which will define when and how often our newly-created service will run. Create the same file but replace .service by .timer!
[Unit]
Description=Run FFMpeg watchdog
[Timer]
AccuracySec=1s
OnBootSec=1min
OnUnitActiveSec=10s
Unit=ffmpeg_watchdog.service
[Install]
WantedBy=multi-user.target
Itās fairly straightforward once youāve figured out what to doāsomething that describes at least 90% of Linux-related tasks. Now you just need to enable all of this with two systemctl commands:
sudo systemctl enable ffmpeg_watchdog.service
sudo systemctl enable ffmpeg_watchdog.timer
And now everything should be active! Make sure to test it by forcing the application to devour all your RAM; your new protection should kick in. 10 seconds and 80% is a good interval for me, since the bloat happens slow and steady under my circumstances, but depending on your application, you might need to adjust the .timer to check slightly more often.
Anyway, the usual caveats of my ātech solutionsā posts apply: I donāt pretend to know all the answers, I donāt know if this is the proper way to do something, and I donāt know if this will work for you, but I do hope it will help you, or at least guide you towards what youāre looking for! Cheers.