Rainy Days and Unreachable Nodes
With an absence of somewhere to put this, I'm back.
I'm in an authoring mood once again and refuse to let my wake of abandoned blogs that lay through the years put me off from a return. Whilst I always hope to live up to the blog's title and discuss many wordly adventures; for the time being I am at Aberystwyth leaving such big stories small at hand.
In an attempt to re-re-re-re-kindle my blogging habit, I will try and post about my findings of my study into Computer Science and Statistics and interesting things I find.
I'm in the process of writing my own blogging platform that'll organise these various topics nicely, but all personal projects are a long way from completion given the little time I have outside of assignments and being a lazy student.
In which case, Posterous will probably do for now.
We begin on a rainy day and a spot of server outage.
It would seem Aberystwyth has been holding back on us all this time and is dowsing every square millimetre of the town with a vast volume of rain which more than makes up for the rather dry (if not cold) weather we've been having.
To add insult to the day's horribly moggy feeling, a glorious one hundred and thirty four days of server uptime have come to an abrupt end.
As context, I currently have a virtual private server (VPS) housed with the rather lovely BigWetFish who I would recommend; they offer excellent prices for students and respond quickly to any node issues.
Over the past few weeks, I've noticed a trivial issue with my apache server; whereby it refuses to perform its job of serving web pages to users from any domain hosted on the VPS, only offering a mere internal server error for their trouble.
Happy to exploit one of my more useful terminal aliases, I grabbed the tail of apache's error log whic featured many iterations of;
(12)Cannot allocate memory: couldn't create child process:
Really, out of memory? The node has 1024mb of RAM!
But sure enough, a check with htop - which I often leave open to confuse any nearby Geography students - confirmed there was no memory available for anything.
Restarting the web server suffices; but only acts as a quick fix.
The errors rear their heads in the server logs just a week or so later.
Following a service restart of apache today, I decided to check the error logs and noticed that whilst apache was starting it was creating some child processes for the mod_python module; a module designed to integrate the Python language into the functionality of the apache web server.
I'd originally only installed the module as a simple learning exercise for deploying Django to the web, although it is now recommended to use mod_wsgi for this purpose instead, which I have implemented, leaving this python module a little redundant.
So, I decided to remove it.
Mostly in the hope of solving the phantom apache memory leak.
Unfortunately, the VPS didn't seem particularly keen on parting with mod_python this afternoon whereby my terminal locked up during the yum remove task. Force killing the process seemed to completely bork running any form of yum command.
So I forfeited my uptime record and rebooted the VPS node.
After a little bit of whiteboard drawing, I figured the server should be back up and running to continue my newfound vendetta against mod_python.
My SSH connection request timed out, and pings were returning "Host Destination Unreachable".
Terrible words fell out of my mouth as I realised all I could do was submit a ticket and wait patiently. I taught a few freshers how and why to use getters and setters in Java to pass the time.
A Real Analysis lecture and a wander home later, the server had been tended to and brought back up to continue investigations. This time the python module was removed by yum without issue but rebooting the server still loaded mod_python.
But, but, I just removed it?
sam@improbabilitydrive ~> sudo /usr/sbin/httpd -M | grep python
Syntax OK
mod_python (static)
It would seem it was not removed after all - though at least my include syntax was okay, I guess?
I located the rogue module...
sam@improbabilitydrive ~> cd /etc/httpd/modules/
...sensibly hiding in plain sight in the httpd modules directory, far too busy chilling with the likes of mod_suphp and laughing at the embarrassingly present mod_frontpage to be removed by yum, I presume.
Stripping one of the many additional httpd.conf includes confusingly provided by CentOS/WHM of the import statement to load mod_python, the ordeal is now over. Though I'll have to wait a week or two to see if this has solved the memory issue.
Edit: This seems to have worked, yay!