Zero syslog.severity:"Error" logs
A couple of days ago I started to configure our syslog daemons to push the logs to https://www.loggly.com/ . Loggly allows you to search through the logs, easily identify issues and sends alerts when user-defined thresholds occur.
At one point I noticed something really annoying: It was difficult for me to define alerts for log messages with error severity since many of those were already flowing in. This made me investigate each issue, just to understand that most of those "errors" were caused by rsyslogd warnings, missing files inside my docker container, permission errors, etc. in other words: nothing that would take our site offline nor affect performance.
The perfectionist inside me couldn't deal with it. I had to fix all those issues, and so I did. This allowed me to define really low alert thresholds (>0 errors in a minute send an alert), which is great!
Finding a real problem in a sea of false positives might be hard, but that's not an issue we'll have at TagCube!
