Software Gunslinger

So, it’s been in the news that Google is going to remove extensions from the Chrome Web Store that don’t comply with their new privacy guidelines.

I guess I'd believe there’s good intentions behind this. That Google wants to protect users' private data and all... but I'm afraid this won't have a happy ending.

Since this is uneasily familiar, let's start with a little story first.

I would have assumed that there should be plenty of examples to solve this problem, but there isn’t. So, here we are.

I really like state_machine. I know it very well, and I know its caveats 1, and still like it. With that in mind, here’s a specific use case that becomes obvious after thoroughly reading the docs, but there’s no examples.

A couple colleagues I talk to (yeah, literally two) often say that I should write a book of replies I give to recruiters 1. A book would be overkill, plus I don’t think they’re that special. So, here we go. Let’s start with a classic one:

“We’re looking for a rockstar engineer”

Every real developer has been reached out with this a million times.

"Sir, I was offered this before. It usually involves performing like a rockstar (including the no-sleep part) while being paid as the average developer. Unless we’re actually talking about a 6-figures monthly salary, I think it’d be way better if we both could agree on more realistic terms.”

“The project is overly interesting”

Ah, yes, I’m sure. Let me guess: the project is in stealth mode and you can’t tell me about it. Let me guess again: it’s a WordPress-based PHP project.

“Excellent! You’re talking about the NASA, right? Because unless you’re talking about the NASA, no project can be that interesting. Even if you are, I certainly won’t work for cheap. And you’re probably not talking about the NASA, so I’ll add an extra fee to cope with the boredom of the project.”

TL;DR: Software gunslinging isn’t (by far) a dream job.

Being a (software) gunslinger is terrible. There, I said it. Still today, after all that’s been said, there’s the perception that it’s a dream job, full of freedom and never-ending happiness. Most of those ideas are myths, and just a handful ones are true.

Let’s define (software) gunslinger first: a (mostly) independent (software) worker dealing with projects for a (mostly) limited timeframe, setting its own price per hour and/or goal, and (mostly) working completely on his own, (most likely) in a remote fashion.

There’s several, real or truly imaginary, additions to that.

TL;DR: Keep reading if you don’t mind some cursing about teaching normal people how to code. It may not be the kind of cursing you’re expecting, though.

There seems to be very strong antagonist positions about teaching programming to not technically-aware folk. Depending on your point of view, that's the greatest thing since the invention of printing, or worse than a nuclear armageddon.

Gunslinger peers tend to be just infuriated by it. Among their reasons are that:

Technically-unskilled people don't need to know how to code at all, as programmers don't need to know how to fix a broken pipe, they just call a plumber.

This is just a push from corporate management and recruitment suits to fill the big gaps in "engineering resources", because there's high demand everywhere.

The same corporate plot is also intended to cut jobs, displace capable people and replace them with cheaper coders.

There's already too many mediocre software "engineers".

Many, many capable engineers are mostly self-taught, and they're constantly learning, everyone should be able to do the same.

Formal education in computer science is so complex that it's only for universities and above.

Online programming courses are silly, they just scratch the surface, and you're set to start with programming the wrong way.

I may agree with some of those ideas in a broad sense, but I don't consider them valid reasons for anything. While online tools and courses -which I don't care about- seem to be blooming, I'm interested in implementing the more classical, usually ignored approach: include programming in formal education -- elementary and high school. Give it the same status that math, physics or literature have.

TL;DR: In software, current meaning of the word "entrepreneur" is generally tied to an non-technical hypocrite.

In our daily gunslinging tasks, we hired keyboards have to deal with many self-entitled "entrepreneurs" 1. Without putting everyone on the same basket, I'm going to focus on (what seems to be) the most prolific kind: non-technical (or, in all fairness, barely technical) ones.

They usually share the same speech, with small variations, along the following:

"I want my idea to become a reality, change the world for the better, and get rich in the process".

Let's dissect the phrase. It is, at the same time, so widespread and hypocritical that I can barely hear it anymore without laughing maniacally or being overwhelmed by profound, true gunslinger hatred. Still, people I talk to is happy to say the words in a daily basis without the most basic sense of absurdity.

You want your idea to become a reality. Explain to me again how you can't really implement it. Oh, you think the idea is the only thing that matters, and implementation are just practical details? Cool, I have a old term for you: businessman. You want something done, that you can't do yourself (not even given infinite time), you have access to money somehow, and you trade it in exchange for work. Engineering work. That's what businessmen do. I don't see how that's really "entrepreneurial" 2. Oh, you want a "technical co-founder"? Sure, I love working for free! I always agreed that expecting proper compensation for my hard-earned knowledge was silly.

You also want to change the world -- I'll assume for the better, if you didn't say it. Explain to me again how the seventh million iteration of an application for sharing pictures ("but, but… with a twist!") is going to make the world a better place. You can't really think that unless you've had some serious brain damage in the past. To me, it just looks that you want to carbon-copy something that's been done a million times and then, inexplicably, you'll get acquired by Google or Yahoo or whoever for a ridiculous amount of money that surpasses NASA's yearly budget. Later on, they'll shut down your application, all your engineers will fly away, and you'll retire to your own private island until you die. That may certainly change your own world for the better (if that ever happens, which I doubt), but for the rest of mankind it'll be the same old dusty wasteland.

Then, lastly, you want to get rich in the process. The shorter the process, the better. Explain to me again how a new rich person is going to make the world a better place. No, seriously 3. Disparities in wealth distribution are one of the main reasons because the world is in such an horrendous state. Isn't your argument self-negatingly hypocritical?

There's a laughable and childish variation of the above: in all seriousness, some guys want to be the next Steve Jobs or something like that. Yes, I've heard people above their 30s saying that, word by word 4. I don't know how anyone with a working brain would invest money in such argument.

Certainly, there are exceptions to the above, but those are unique, that's why they're in the news. I'm not referring to any of them, but exactly to the opposite -- the vast hypocritical majority.

In the end, my answer to all this is always similar to:

"Give me money, I'll buy some coffee and turn it into code. I wish you the best in earning a decent living, every day, for many years to come."

I'm sorry (no, wait, actually I'm not) if anyone gets offended by this. But at the same time it's sorrowful that so many people fail (sometimes, conveniently fail) to see the hypocrisy in those arguments. They look like little kids trying to impress their friends, but then enormous amounts of money, time and resources are incomprehensibly wasted in a stupidness continuum, and I start wondering, isn't this the technological equivalent of war?

I had to look up for the word to write it correctly. ↩︎

I had to look up for the word, again. ↩︎

No, "benevolent" philanthropist organizations created to circle around taxes and mock about solving problems that they've actually helped to create themselves are not doing any favor to the world. ↩︎

TL;DR: PHP is still a poor choice for continually-running processes. See this post for context. Read on for more proof.

There's been some reaction to the previous entry. Some people agreed, mostly seasoned PHP folks, some others just didn't. Yeah, apparently it pleased or infuriated more than two folks in the entire Internet.

First and foremost, a correction to previous statements: PHP's 5.3+ enables garbage collection by default, it's an opt-out feature, not opt-in. So it's probably enabled in all your scripts, even the fast-dying ones, if your PHP version is recent enough and you didn't do anything funny to your php.ini file.

Yes, I already acknowledged that PHP has a garbage collection implementation starting 5.3.0 and up (opt-in or opt-out, that's not the problem). I also acknowledge that garbage collection works, and is able to take care of most circular references just fine. However, if you're one of the many that think "hey, no one should be using anything below PHP 5.4 nowadays", you're clearly too young to remember how much blood, sweat and years it took to get rid of PHP 4, even when PHP 5.0 was reaching end-of-life, 5.1 was healthy, and 5.2 was already in the works.

Anyway, as previously stated too, garbage collection is a great thing, but not enough for PHP. It's a borrowed feature that does not play well with old fundamental decisions inherited from the original design. Garbage collection is not a magical solution for every problem, like many tried to argue about. Let's illustrate with another example.

React

In between the sea of praises and insults from the previous entry, some constructive-criticism folks suggested that I should try out projects that use PHP in a continually-running loop. One, specially, was recurrently suggested and caught my attention: React, an event-driven, non-blocking I/O platform, somewhat similar to node.js (ok, maybe more than just similar).

Before starting, I can't stress enough how much of a good thing I think React is. No, this is not irony. No, I don't think it's great because it aims to bring the wonders of node.js to PHP, which may be cool, but I think it's truly exceptional because it's the kind of project that pushes the boundaries of the underlying programming language further. Myself, as I'm sure many more, want React to be successful, and for that PHP should step up and support this kind of development… which, currently, doesn't, and I think the project will inevitably suffer from that. Let's continue.

Why React, and not something else? Because is the kind of innovative stuff I'm talking about from the beginning, it uses PHP in a continually-running loop that isn't just meant to recurrently run time-consuming database queries in the background, something that has been already solved by garbage collection (and probably a patched up solution involving periodical restarts, for sure), or worked-around with cron jobs.

A preliminary note on deployment

Other React-like projects usually have a production deployment guide in the lines of "this is good for development/debugging, but you need something more robust for production deployments" 1. I can't find anything for React, except a few notes for Ratchet, a PHP WebSockets framework built on top of React, so I made sure to do my homework and follow those tips to simulate a production-like environment. That means we're assuming React is following node.js steps: the production server is already built-in, at most you should be able to reverse-proxy the requests, but ultimately the continually-running process will manage them.

Besides that, I'll be using the latest stable PHP release to date (5.4.13 by the time of this writing), with garbage collection enabled by default, of course.

Testing it yourself

Assuming that we'll be using React for more than printing "Hello, World" in the browser, I took two examples from their codebase itself (here , and here), mixed them together, and added a print line for quick debugging. No circular references, no special tricks. The goal is just to serve a shitload of data (in React's own terms), sending 20MB of a single random ASCII character as response to each HTTP request. Here's the result:

https://gist.github.com/anonymous/5357469

Running that file (granted that you have installed React2, of course) will start the server in port 1337. Now, we can test this from the shell:

while [ 1 ]; do wget -qO- http://127.0.0.1:1337; sleep 1; done

So, we're sending one request after another, waiting a second in between, and repeatedly getting the following debug output:

Allocated memory usage: 23,174,168, Memory Limit: 128, PID: 72928 Allocated memory usage: 23,174,168, Memory Limit: 128, PID: 72928 ...

So far so good, right? It allocates 20MB to send the response, plus a few more megabytes for PHP own requirements, and never gets past that, so the garbage collector works just fine, freeing memory after each request.

Sadly, this test is not even close to a real production environment. Requests won't gently queue one behind the other and wait for the previous one the finish, just to behave nicely to your server. So let's test this with a slightly less trivial, more production-like scenario (verbosity and multiple lines for clarity):

while [ 1 ]; do for i in {1..4}; do \ wget -bqO- http://127.0.0.1:1337; done; sleep 1; done \ done; sleep 1; done

Now, we're sending 4 request per second, with one twist: using the -b option in wget (meaning "run in the background"), all request are executed in parallel, not one behind the other. So, approximately, we're sending 4 parallel requests per second (yes, I'm still gentle enough to wait a second in between). Here's the debugging output:

Allocated memory usage: 23,174,168, Memory Limit: 128, PID: 64873 Allocated memory usage: 39,974,960, Memory Limit: 128, PID: 64873 Allocated memory usage: 46,866,264, Memory Limit: 128, PID: 64873 Allocated memory usage: 62,603,296, Memory Limit: 128, PID: 64873 Allocated memory usage: 72,787,752, Memory Limit: 128, PID: 64873 Allocated memory usage: 87,614,320, Memory Limit: 128, PID: 64873 Allocated memory usage: 105,745,480, Memory Limit: 128, PID: 64873

And then...

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20971532 bytes) in /react-test/vendor/react/http/React/Http/Response.php on line 82

Wait. What just happened?

The process died, because PHP's default memory limit (128MB) was reached, just as expected. Everyone knows that.

You mean that the request died, not the process.

No, I mean the process. Everything. The server, the event loop, the request that went over the memory limit, and all other ongoing requests by the time too. All dead.

So the garbage collector didn't work?

The garbage collector was working just fine.

But aren't you being specially tricky or aggressive with the React server?

I just mashed up a couple of their own examples and figured that in a production environment, a few requests per second is not too much to ask for. I took less than five seconds to die. To make things worse, remember that this was tested in against your loopback interface, in your own machine, with virtually no network latency.

So, should I increase the memory limit?

Sure, you can. And I'll increase the number of request per second, or the HTTP response size, or I'll just put this in a real production environment with real network latency and your process will die anyway. It's a matter of math.

Can't this be fixed?

It may be worked around, with clever or dumb techniques. None of them validate PHP as a suitable language for continually-running processes.

Can't we just monitor and restart the process when it dies?

Sure, that has been previously suggested, too, as a workaround. It's a good thing to monitor your background processes, generally speaking. However, monitoring is intended to prevent your infrastructure to collapse in the event of an unexpected error, hopefully with enough details to identify, debug and fix the problem. In this case, we're talking about a known, expected, inevitable issue. Restarting every time it happens may be good enough for your needs, but still doesn't validate PHP as a suitable tool for the job.

Isn't this a problem with your own machine because it isn't a real server?

What? No. You may get slightly different results in your environment (maybe you need an extra request per second, or a larger HTTP response), but specific by-the-byte numbers aren't important here. You should be able to bring death to the process quite easily. But more importantly, please, read on. It's substantial to understand what we're trying to fix in the first place.

An attempt on understanding

So, what exactly happened here? PHP veterans may already know, but we're going for the full explanation.

First, let's understand how React works. Identically to node.js, they use an event loop. Also, as well as node.js claims (with maybe not the best terminology), it's "single-threaded". For React, what's important to us is this: while it'll be able to manage requests simultaneously, there's no forks, and everything will happen within the the same process, the original one, the one that runs the event loop. See the debug output from the previous test, the PID never changes.

Secondly, I think we can all agree that garbage collection frees memory once it's not being used anymore. Yes, I know this is a very basic concept, and not specific to PHP. That said, if your data is still being transferred through the network, it's obviously not ready to be freed.

However, the significant result of those combined factors is that allocated memory for each individual request you send to React will add up to the one and only process that's running. If somehow you have enough requests that, individually, have allocated enough memory to sum up a number that go over the memory_limit, your entire process (server and ongoing requests) will die. In "classic" PHP terms, it's the equivalent of a fatal error in a single page that brings down Apache completely. Go figure.

Also, two factors were taken out of the equation that would make things even worse. One is real network latency, as previously mentioned, which would keep allocated memory for way more than a fraction of a second until a slow transfer is completed. The other would be a service that takes more than a fraction of a second to generate its response. In both cases, your allocated memory will rise, won't go down quickly, and will continually add up 3.

What is this memory limit, anyway?

You probably figured it out by now, but this is not a React's problem. memory_limit, as well as max_execution_time are old relics inherited from PHP's original dying nature. You can read the manual for their formal definition, but their spirit is this: if your PHP process is running for too long, or consuming too much memory, it needs to be killed. Why is that? Because PHP processes are meant to die rather sooner than later. That's what the language was designed for from the very beginning.

You can argue that this kind of monitoring (for a process that's consuming too much memory, or running for too long) is closer the operating system toolset rather than an application's programming language setting. I concur, but those settings have been there forever, and they now reveal PHP's true nature. Their defaults haven't changed in a while. In the latest stable release, PHP 5.4.13, it's still 128MB, a perfectly valid value to serve web pages.

Is there any hope?

There's always workarounds. There's been workarounds before garbage collection, and there are some after it.

You can disable memory_limit completely, setting it's value to -1. That's been strongly discouraged throughout the years, and by the very PHP's literal manual definition, memory_limit "helps prevent poorly written scripts for eating up all available memory on a server.". Judging by the default value, PHP thinks that any script going over 128MB is poorly written. You can disable the limit anyway, if you're reckless enough.

Or, maybe you can set a high memory_limit -- ridiculously high enough to suit your needs, making sure that your process won't die that quickly. I've tried that with the above example, and made it last a couple of additional seconds. Back when PHP has no garbage collection, I clearly remember being forced to set a limit of 4GB in a server. Now we're talking of a similar solution. See why garbage collection isn't a magic solution after all?

Whatever you do to the memory_limit setting, keep in mind that it's global, and it is that way for a reason. It means that if you raise the memory limit to cope with a particularly troublesome request, you'll be raising that for every other single thing that runs through PHP in your server. That's also strongly discouraged, because in the name of a very specific problem, you're opening for potential problems everywhere else, where the rest of your PHP code will probably run just fine with a small portion of the allowed memory.

But wait, there's also ini_set. You can change configuration values dynamically in your code, without affecting server's PHP global configuration setting. That's true, but in our very particular case, it's a non-solution. Remember, React runs as one, and only one process. If you change the memory_limit with ini_set inside a very specific request handler, you're changing it for the entire process and all other requests too. No, no joke. It's the same thing as before, but instead of doing it at PHP's global scope, you're doing it at your application's global scope 4.

Yeah, you can raise the memory_limit, serve a particularly memory-consuming request, and lower it again. But, in the meantime, all other React's request handlers will run with the raised level, including the ones that doesn't need it. Remember, global setting, one process, simultaneous requests. Same thing over and over. Oops.

Or, maybe, in the near future we hear about a deployment option for React based on workers (like Gunicorn) instead of an omnipotent forever-running process. We only can dream about that today.

Or, just maybe... you may be able to configure a web server as reverse-proxy to proxy (uh?) only a few simultaneous requests, keeping the others waiting in queue, in the way of "don't send me more than two of those requests at a time, or I'll die", if that's performant enough for your needs. The question is… why you would ever need to apply those half-baked tricks? They have nothing to do with anything.

Additional questions, no answers

Why on earth are we listing those workarounds?

How come that by this point, there's still people that doesn't ask themselves "Darn, is there maybe, just maybe, a fundamental problem in the language I did choose for this?", or "Why I'm spending more time on workarounds for a process that doesn't like to keep running, than the time in implementing the process itself?", or "Why this ugly memory settings have to be set or unset in the middle of my business logic?"

Why people keep arguing that PHP is perfect for continually-running processes just because they were able to work around its limitations once, believing that one particularly working but constricted idea ("I just restart it when it crashes!") is the magic solution for any other possible problem?

I have no clue. But there's more.

Sequential nature

PHP, born as a scripting language, is meant to run in a sequential manner. Fundamentally, almost all "classic" programming works like that, and the problem doesn't lie there at all.

It starts being troublesome when the language puts all the eggs in one basket and aims to die early: if everything is part of the same sequence, an early error will kill whatever else comes later, related to the error or not 5. Because of that, in our particular React example where the everything is part of the same process, an individual handler raising an non-catched exception will kill the the entire server 6.

But the real problem comes when you combine that with PHP's totally incoherent amalgamation of global error handlers mixed with a very late implementation of exceptions. They don't even talk to each other. Some of them are just uncatchable. A lot of things have been said about how this particular part of PHP is horribly designed, so I won't add much else. It's part of its dying nature. PHP dies, sometimes you can't even know why, and the best you can do is restart and hope for the best 7.

But, really, is there any hope?

React may be enough for your needs. If your responses are small and quick enough, or you don't expect too much load, or you architect the solution in a way that the actual request is just a lightweight interface to something else, it may work. You can also take advantage of the non-blocking I/O (ignoring React's own example for a shitload of data) and serve large data sets in smaller chunks, giving the garbage collector some air, which will probably be slower, but lively.

We're not judging React here, we're doing a technical analysis of PHP as a suitable tool for continually-running processes. In my opinion, the risk of having to craft half-baked last-minute workarounds is too much to justify it against other tools. I'll be delighted to see React's guys to find a totally unforeseen solution to this, but I think PHP is blocking their way.

Finally, PHP has been slowly improving. This isn't irony, either. It's been around for 13 years, it's one of the most widely-used programming languages, there's no argument against that. No matter how ill-designed it is, flagging it as totally useless would be incoherent, short-minded and a complete denial of reality. Even some largely awaited improvements have been made at the core and language syntax level, to link one of a few. But it's been too many years, and it's been consistently like that: PHP is always slowly improving. It's been slowly improving forever. Right now, to me, the improvement speed that PHP historically had is just not enough anymore, and it is already, objectively behind.

A more proper question would be "Is there any hope for PHP and continually-running processes?".

No, right now, there isn't.

For instance, Flask has an insanely-detailed deployment chapter. In node.js, everything is expected to be out of the box, though I think proxying and monitoring should be a must, rather than a mere recommendation. ↩︎

You'll need to install React through Composer. As an old-timer coming from the dark PEAR days, I can only say that the work that has been put into it is top-notch. If you've tried other attempts to centralise dependency management in PHP, you know what I'm talking about. ↩︎

Node.js has its own limitations for this too. Here's an equivalent sample script. After a hundred concurrent requests it obviously starts struggling and performance gets seriously affected, but the server itself doesn't die in a matter of seconds. ↩︎

Globals have been a long and unending nightmare in PHP. If you want some background, just try to come up with a rightfully architected solution to manage multiple timezones with PHP's original datetime functions. ↩︎

See "Bring it to the foreground" in the original post. ↩︎

I've heard strong arguments stating that every time, every error has a proper exit, and that the perfect programmer should absolutely foresee all that, catch every single possible error condition and manage it accordingly. That reasoning means that no one will never, ever experience an unexpected condition, specially not in a very busy instance, and that if you do, it's your fault because servers are perfect, operating systems are perfect, programming languages are perfect, database engines are perfect, time is infinite, and you're flawed. Send me a ticket to that fantasy world, please. ↩︎

Disclaimer: I've got 10+ years on my back as a PHP developer. I started using it when PHP4 was a little boy and PHP5 only a dream in Zend's mind. I did a lot with it, loved it, cursed it, and saw it grow and evolve, sometimes with great shame. I still use it for some legacy projects, but it's been a while since it's not the language of my choice anymore. Also, I'm not affiliated with any framework or tool mentioned here.

TL;DR: If your project will rely on continually-running processes to function, avoid PHP.

In my opinion, a lot of the hatred that PHP receives misses the utter basic point: PHP is meant to die. It doesn't mean that a perfectly capable (to some extent) programming language will disappear into nothingness, it just means that your PHP code can't run forever. Now, 13 years after the first official release in 2000, that concept still looks valid to me.

A dying programming model

The core PHP feature follows the most simple programming workflow: get input data, process it, display the output, and die. As much as I think sites like PHP Sadness are a fun way to help in detecting language inconsistencies, there's no much they can do against that basic design choice. But there's been some attempts through time, for sure.

In the beginning, dying wasn't that big of a deal for websites. You read something from the database, apply some logic or formatting to it, and display the result in between a sea of HTML tags. PHP is still really hard to beat at that, it's the language core feature, despite the many horrors brought to the world of programming from this approach. Thankfully, most problems solvable by these techniques have already been solved, and the remaining ones are being tackled with more clever or modern stuff, because that tiny functionality is usually just a microscopic part into a bigger, complex project. Alas, PHP insists in dying when it's done. When you try to do the opposite, bad stuff happens.

Not just websites anymore

Sending information from a contact form through email is still a good task for PHP. Nothing wrong with that (well, except that if I keep doing things like these after 10+ years, I probably made some very bad decisions along the way). But that's also an example of something that has been solved a million times, in million different little ways. Virtually any other web-oriented language or framework can do that too.

Let's say we've past that point. Now we're writing a real application, a complex one. As complexity grows, also grows your codebase. Let's say you're a capable enough programmer. You use PHP 5.x and modern programming techniques. Everything is cleanly abstracted into interfaces and classes. You know your way around existing production-quality libraries. Now, you're probably now dependant on your own ORM models, vendor code, custom interfaces, maybe your client API for a third-party RESTful interface, and so on. Everything is written in PHP.

And here is when the nightmare begins: you'll inevitably need to run code in the background. Your application reached a complexity level in which waiting for an HTTP request to do something isn't enough. What's different may be anything: you have to process queued orders, or you have to cache information to speed up the actual HTTP response, you have to check periodically for due payments and proceed depending on the result, you have to constantly update pull-only data from external sources, you have to write some stuff to the database in batches to avoid performance degradation, you have to open and keep alive several network connections, or you have to implement the server-side part of a WebSockets application. They're just examples, there's no end for this list, it depends on what you're building.

Bring it to the foreground

The most infamous implementation for the above thing may sound familiar: No matter how "enterprise" PHP or the framework of your choice claim to be, it's also supposed to be so cheap that you don't have access to ssh, cron or any similar tool in your $5/month shared hosting server.

Which is the easiest solution to that? Bring background tasks execution to the foreground, of course! Run them at random times for 1/nth of the page visits. That means that one visitor, randomly, will patiently wait while a system-wide background task runs at its expense, and only when it's done, the actual request will be processed for him. It's alarming how many "serious" and "mature" PHP frameworks implement this into their internals, how difficult is to find those blocks of code, and how intricate is to untie them.

Why this aberration? Well, special server credentials aren't required; plain, old and feature-less FTP is more than enough. Everything is triggered by the client, and a little delay every once in a while for only one visitor shouldn't be noticed. And that's exactly why it'll inevitably fail, because it works great when testing, it works well when deploying, and it will inelegantly crash under load. You're assuming that the amount of visits you'll get will always be the approximate number you need to run some task each a few minutes. But pseudo-randomness in hundred-nth slices is irrelevant with thousands of requests per minute. Tasks that were supposed to run every once in a while start running several times between blinks of an eye. Weird issues start happening, like the same task running -and failing- twice in the same exact moment, sounding errors when some of the tasks access to shared resources in a locking way, and so on. Keep in mind that a crashing tasks also means a broken page, because, you know, PHP dies and everything else dies with it, including the user's request. Dying doesn't necessarily mean success.

If you're enterprise enough, you can move those background tasks to a cron job. While that's somewhat better, it works until time is not enough as the only control variable. For recurring tasks (and recurring is the core nature of crond), you're always expecting that a single task finishes before it starts again. That may be mostly true, otherwise you may find yourself adjusting crontab intervals as your load increases, or assuming a long enough one that may be safe, but not fast enough for your application. Under scale that may get very, very hairy. In that scenario, you may start thinking in separating system-wide tasks in small batches. And someone must coordinate those batches, someone that doesn't die.

Summon the daemons

Here's where the nightmare gets really ugly. When at the point where the proper solution seems to create a daemon, or a process that doesn't die. Two examples of this would be establishing and maintaining a WebSockets connection, or creating the producer component in a Producer-Consumer implementation.

But PHP will forsake you.

There's several issues that just make PHP the wrong tool for this. Remember, PHP will die, no matter how hard you try. First and foremost, there's the issue of memory leaks. PHP never cared to free memory once it's not used anymore, because everything will be freed at the end -- by dying. In a continually-running process, that will slowly keep increasing the allocated memory (which is, in fact, wasted memory), until reaching PHP's memory_limit value and killing your process without a warning. You did nothing wrong, except expecting the process to live forever. Under load, replace the "slowly" part for "pretty quickly".

There's been improvements in the "don't waste memory" front. Sadly, they're not enough. As things get complex or the load increases, it'll crash. Consider the following snippet of code:

https://gist.github.com/anonymous/5337645

Those are called circular references, objects that have a reference to another object that have a reference to the original one. When you try to run that script, depending on the PHP version you use, it'll slowly eat more and more memory, because at runtime, it fails to recognise when all the objects in a circular reference aren't used anymore, even if you explicitly try to free them with unset().

That's a very basic example, you may have more "nodes" in the circle, which is what happens when your program starts being more complex than the above simple script. We can argue if circular references are good software design or not, but there's one thing that uses them profusely: ORMs. For a variety of good reasons, an object representing a row in a database is always the same object in memory, and it's referenced in all other objects, if required, usually defined by foreign key constraints. That saves a lot of trouble, like propagating changes in a single row throughout all instanced models and their references. Now look at your database schema, count how many foreign keys you have… see how many circular references you may have in your running process? PHP won't be able to tell which ones are being used and which ones aren't, sooner or later memory will be exhausted, being used or not, and PHP will die.

But more importantly, keeping a program running forever was never PHP's top priority, that's why problems like the above one weren't never really solved. Garbage collection was introduced in PHP 5.3, in its first real implementation, as an opt-in feature. That's right, the language tells you to ask "please, could you not leak memory for this one?". That was around mid-2009, a relatively recent change. What happened before that? Absolutely nothing. Nobody cared if PHP leaked memory or not, because it was meant to die as soon as it was done. In PHP terms, leaking memory was just fine.

Not just about memory

There's more to this. If you've used PHP an awful lot , you may have experienced this very weird issue:

Fatal error: Exception thrown without a stack frame in Unknown on line 0

What does that mean? I honestly have no idea. I can't find the line #0 into an unknown PHP file. People seem to be as clueless as me. Others say it's related to PHP's errors and exceptions handling. No one knows for sure, I can only tell you the preconditions to get that error: continually running processes, working with a gigabytes-sized database, under heavy load/processing, in a production environment. That's right, it's something you wouldn't see in your local instance.

I've also experienced sudden process crashes that were untraceable. They just died. PHP is full of cryptic, hard to debug or impossible to reproduce errors. Just look here.

Using the wrong tool

Do you see the pattern? I've inherited projects where PHP was used for daemons or other stuff that's not just regular websites (yes, I'm a hired keyboard), and all of them shared that same problem. No matter how good or clever your idea looked on paper, if you want to keep the processes running forever they will crash, and will do it really fast under load, because of known or unknown reasons. That's nothing you can really control, it's because PHP is meant to die. The basic implementation, the core feature of the language, is to be suicidal, no matter what.

For a moderately new project, I've been using Python, virtualenv, Flask , Supervisor , Gunicorn. I'm amazed how well all those work, how mature each of those components are -- some of them are even considered to be in beta. But more importantly, they handle the not-dying thing really well. How they do it, you may ask? They do nothing, Python just won't commit suicide. That's it, it's part of the language core design. I'm not saying that's the magic bullet to everything. Maybe I spent too much time with PHP and I'm easily impressed.

Conclusion

Yes, you can always find a workaround. You can increase PHP's memory limit to ridiculous amounts and keep processes alive for a little more. You can get creative with cron, shell scripts and other UNIX tools. You can use PHP for the website and another language for daemons. But keep in mind that we're talking about mid-to-high levels of complexity here, it's not only about the language itself but also about all the libraries and tools that you put on top of it, so adding a second programming language because your main one has abandoned you isn't a trivial task. It's also about working with stuff in which a 30-seconds delay may have disastrous results.

Maybe there's people that think that a more services-oriented architecture may help to overcome PHP's limitations. Maybe there's people that can argue that PHP 5.4 or PHP 5.5 are a lot better, and that the language is slowly improving.

Thankfully, I've already moved on.

Discuss on HN or Reddit

Continued here

Some updates, based on people's comments (yeah, the two of you, dear Internet strangers):

Yeah, I'm aware of the irony in smacking PHP from a PHP-based blog platform. That was intentional, like the tabloid-like post title.

No, I don't want PHP to disappear. As I said, it's a prefectly capable language to some extent. My intention was to encourage the use of the right tool for the job, nothing more.

Yes, I'm aware that message queues are a solution for some of the problems stated above (not on $5/month shared hosts, of course). In fact, in combination with cron, that was part of the solution for a particularly ill-designed project. However, one thing is using message queues as the proper solution, and a different one to use it to circumvent the language's limitations.