#python performance

I am really proud.

Yes, it was not 100% by my own work, it was thanks to the help of others...

... but I'm still the one coding it, and it finally sees the light of the day :D

Currently public version:

8013 records in 76.91 seconds

Private version, self optimized:

8013 records in 63.41 seconds

Highly-optimized version with the help of the kind people at codereview.stackexchange.com:

8013 records in 40.90 seconds

So, I've finished optimizing some critical paths in sims3py.

A couple of days ago, the critical paths took 266 seconds to complete on a relatively small .nhd file (47.7 MB, 8013 resources).

Now after optimization, the same critical paths take 78 seconds. Less than one-third the time they took prior to optimization.

Taken together with the earlier optimization, I still see a 4x speedup, so I'm not complaining 😀

I'm giddy with excitement!

Thanks to the magic of the intervaltree package, a crucial codepath that originally ran in 42 seconds...

... now runs in 3 SECONDS FLAT!! Just one-fourteenth (1/14) of the original time taken! That's a 1300% SPEEDUP!!

Who says Python can't be fast? :-D

So here I am, performing code profiling for sims3py PackageParser.

There will be at least two iterations: Before and After. And more if there are performance-impacting changes after the After iteration.

Every iteration, I'll collect the statistics using the (5-2) method. Or, if I have the time and/or inclination, (7-4) method.

(Explanation: "(5-2)" means take 5 observations, and discard the highest and lowest values. "(7-4)" means take 7 observations, and discard the 2 highest and 2 lowest values.)

So, I was trying to find a way to optimize the slowest parts of sims3py. And serendipity found me this:

http://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews

This is the first time ever I heard about "buffer protocol" and "memoryview".

Reading the article and digesting the information, I suddenly realize that these pair of Python features could very likely be the key to a performant Python program!

You see, sims3py has to do a HUGE amount of slicing and dicing. Every invocation of slicing apparently creates a copy of the buffer. And if the copy doesn't neatly fall into the size of the memory page, performance will suffer.

After finding out about these two Python features, I see how I can speedup the operations by preventing memory copy. The article mentions a 300x speedup; but honestly, I'll take it even if it promises only 2x speedup.

We've all heard it before: Python is slow.

When I teach courses on Python for scientific computing, I make this point very early in the course, and tell the students why: it boils down to Python being a dynamically typed, interpreted language, where values are stored not in dense buffers but in scattered objects. And then I talk about how to get around this by using NumPy, SciPy, and related tools for vectorization of operations and calling into compiled code, and go on from there.

But I realized something recently: despite the relative accuracy of the above statements, the words "dynamically-typed-interpreted-buffers-vectorization-compiled" probably mean very little to somebody attending an intro programming seminar. The jargon does little to enlighten people about what's actually going on "under the hood", so to speak.