#timerfd

The service was a success. It was rock solid and we could brag about up-time of months. Maybe years...

For event handling, we used boost::asio. For cost effective I/O handling, asynchronous I/O using minimal number of threads is the way to go. If you are blocked and waiting on I/O, there's absolutely no reason to spawn threads to wait on it. Popular examples of such services are nginx, haproxy, nodejs, etc. All employing very similar principles.

The single threaded processing works until the system call and processing overhead on the CPU core starts to peak and saturate that core. In other words, you can only execute X number of system calls (and CPU bound calculations) per second on a single core. But a lot of times, it is possible (and simpler) to run another copy of your application on a different TCP/UDP port utilizing more CPU cores.

This effective strategy avoids multithreading as much as possible since synchronization overhead is very undesirable. Minimizing threads also keeps the code much simpler and robust. Another way to look at this is to accept the fact that the more locks you have, the less effective your I/O strategy is. True parallelism is only achieved when you actually don't have locks that stall your processing. 

A similar analogy is using shared pointers. You are probably (hopefully) using shared pointers not because they are very good and awesome, but because you have actually failed (or unable to because you are using boost::asio) to establish robust ownership policy across the code.

There are variety of libraries that offer asynchronous I/O. Besides boost::asio, there is also libevent, libev, libuv, etc. However looking back, now I would have not used any of these libraries at all.

This is because with Linux epoll and Linux timer file descriptors, you really have no reason to use these libraries. Rolling out your own poller with your custom code is trivial and it will out-perform them.

In fact, that is what I'm working on these days. Having tight control over the lower level details and the poller (event code) itself, I've also managed to ditch shared pointers. I cannot stress enough how liberating this is. Now I can establish strong ownership policies across my code and make sure the event handling code does not survive any of stored raw pointer life times. I was also able to minimize calls to malloc/free/new/delete and tightly pack my data in a more cache efficient manner. (Sure, you can also do this using allocators for shared pointers.)

Focusing on low level details, I also successfully avoided boost::function or std::function variants. This avoids behind the scenes memory allocations and copying. Basically old school function callbacks may not look syntactically the best at first, but they perform better than these heavy weight alternatives.

More importantly, I can now control these simple callback functions better. This is essentially same thing as deciding which virtual function is called at run time. Basically instead of using object hierarchy and inheritance, I'm using a small family of callbacks to handle each type. This keeps the code base small since I don't have to write a number of unnecessary classes and their implementation.