Pay Attention, You Might Learn Something

The shell is an odd beast. Although it goes against every current trend in software engineering (strong typing, compile checks over runtime checks, ...), shell scripts are here to stay, and still constitute an important part of every developer's life.

Time again to change your passwords, sensitive data, and your faith in internet security.

You may have heard about the recent Gitlab outage and their great openness in publishing the outage report. If not, the short version is that while working on database replication (and several other things), an engineer unintentionally deleted all the production data.

Only then did they discover they were also having problems with their backups.

“So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.”

Luckily, they did have a successful backup 6 hours before the incident and have been able to restore most of the lost data.

As I read through the report, one line stood out to me.

“YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com”

The report does not include the actual command that YP ran, but I think I most linux/unix folks can guess it was something like this

rm -rf /var/opt/gitlab/postgresql/data

This is not the command I would use to remove an empty directory. If I think I am about to remove an empty directory, I always use the following command

rmdir /var/opt/gitlab/postgresql/data

The difference is simple. If I am mistaken and the directory is not empty then rmdir will do nothing. It will never remove data

$ rmdir /data

rmdir: failed to remove '/data': Directory not empty