#planetpython

GridRoyale - A life simulation for exploring social dynamics

Another day, another project :)

This is a project that I wanted to do for years. I finally had the opportunity to do it. Check out the GridRoyale readme on GitHub for more details and a live demo.

GridRoyale is a life simulation. It's a tool for machine learning researchers to explore social dynamics.

It's similar to Game of Life or GridWorld, except I added game mechanics to encourage the players to behave socially. These game mechanics are similar to those in the battle royale genre of computer games, which is why it's called GridRoyale.

Live-coding a music synthesizer

After so much work and waiting, the video of my EuroPython talk is finally released!

This is a fun live-coding session using NumPy and SoundDevice. The goal of this talk is to make the computer produce realistic-sounding instrument sounds, using nothing but math.

Improving Python exception chaining with raise-from

This is going to be a story about an esoteric feature of Python 3, and how I spent the last few months reviving it and bringing it into the limelight.

Back in the yee-haw days of 2003, Raymond Hettinger wrote an email to the python-dev mailing list, sharing an idea for improving the way that Python handles exceptions that are caught and replaced with other exceptions. The goal was to avoid losing information about the first exception while reporting the second one. Showing the full information to the user would make debugging easier, and if you've followed my work before, you know there's nothing I love better than that.

That idea was polished and refined by many discussions on python-dev. A year later, Python core developer Ka-Ping Yee wrote up a comprehensive PEP that was then known as PEP 344, later to be renamed to PEP 3134. That idea was detailed there, with all the loose ends, potential problems and solutions. Guido accepted the PEP, and it was implemented for the infamous Python 3.0, to be used... By no one. For a long time.

If there's one thing I don't miss, it's waiting 10 years for the Python ecosystem to adopt Python 3. But finally, it happened. Almost all the packages on PyPI support Python 3 now, and getting a job writing Python 3 code is no longer a luxury. Only a few days ago, NumPy finally dropped Python 2 support. We live in good times.

When a modern Python developer catches an exception and raises a new one to replace it, they can enjoy seeing the complete information for both tracebacks. This is very helpful for debugging, and is a win for everybody.

Except... For one thing.

Two cases of exception chaining

There was one interesting detail of PEP 3134 that was forgotten: It has to do with the question, "What does it mean when one exception is replaced with another? Why would someone make that switcheroo?"

These can be roughly divided into two cases, and PEP 3134 provided a solution for each case.

The first case is this:

"An exception was raised, we were handling it, and something went wrong in the process of handling it."

The second case is this:

"An exception was raised, and we decided to replace it with a different exception that will make more sense to whoever called this code. Maybe the new exception will make more sense because we're giving a more helpful error message. Or maybe we're using an exception class that's more relevant to the problem domain, and whoever's calling our code could wrap the call with an except clause that's tailored for this failure mode."

That second case is quite a mouthful, isn't it? It didn't help that the first case was defined as the default. The second case ended up falling by the wayside. Most Python developers haven't learned how to tell Python that the second case is what's happening in their code, and to listen when Python is telling them that it's happening in code that they're currently debugging. This resulted in a Catch 22 situation, not that different from the one that slowed down Python 3 adoption in the first place.

Before I tell you what I did to break that Catch 22, I'll bring you into the fold and show you how to make that feature work in your project.

Exception causes, or `raise new from old`

I'm going to show you both sides of this feature: How to tell Python that you're catching an exception to replace it with a friendlier one, and how to understand when Python is telling you that this is what's happening in code that you're debugging.

For the first part, here's a good example from MyPy's codebase:

try: self.connection, _ = self.sock.accept() except socket.timeout as e: raise IPCException('The socket timed out') from e

See the from e bit at the end? That's the bit that tells Python: The IPCException that we're raising is just a friendlier version of the socket.timeout that we just caught.

When we run that code and reach that exception, the traceback is going to look like this:

Traceback (most recent call last): File "foo.py", line 19, in self.connection, _ = self.sock.accept() File "foo.py", line 7, in accept raise socket.timeout socket.timeout The above exception was the direct cause of the following exception: Traceback (most recent call last): File "foo.py", line 21, in raise IPCException('The socket timed out') from e IPCException: The socket timed out

See that message in the middle, about the exception above being the direct cause of the exception below? That's the important bit. That's how you know you have a case of a friendly wrapping of an exception.

If you were dealing with the first case, i.e. an exception handler that has an error in it, the message between the two tracebacks would be:

During handling of the above exception, another exception occurred:

That's it. Now you can tell the two cases apart.

What I did to push this feature

I found that almost no one knows about this feature, which is sad, because I think it's a useful piece of information when debugging. I decided I'll do my part to push the Python community to use this syntax.

I wrote a little script that uses Python's ast module to analyze a codebase and find all instances where this syntax isn't used and should be. The heuristic was simple: If you're doing a raise inside an except then in 99.9% of cases you're wrapping an exception.

I took the output from that script and used it to open PRs to a slew of open-source Python packages. Some of the projects I fixed are: Setuptools, SciPy, Matplotlib, Pandas, PyTest, IPython, MyPy, Pygments and Sphinx. Check out my GitHub history for the full list.

I then added a rule to PyLint, now known as W0707: raise-missing-from. After the PyLint team makes the next release, and the thousands of projects around the world that use PyLint upgrade to that release, they will all get an error when they fail to use raise from in places they should.

Hopefully, in a few years' time, this feature of Python will become more ingrained in the Python community.

What you can do to help

Do you maintain a Python project that already dropped Python 2 support? Install the latest version of PyLint from GitHub. You can do this in a virtualenv if you'd like to keep your system Python clean. Run this to install:

pip install git+https://github.com/PyCQA/pylint

Then, run this line on your repo:

pylint your_project_path | grep W0707

Symlinks and hardlinks, move over, make room for reflinks!

If you've been around Linux for a while, you know about symlinks and hardlinks. You've used them and you know the differences between how each of them behaves. Besides being a useful filesystem tool, they're also a favorite interview question, used to gauge a candidate's familiarity with filesystems.

What you might not know is that there's also a thing called reflink. Right now it's supported only on a handful of filesystems, such as Apple's APFS, used on all Apple devices, XFS which is used on lots of Linux file-sharing servers, Btrfs, OCFS2, and Microsoft's ReFS.

If a symlink is a shortcut to another file, and a hardlink is a first-class pointer to the same inode as another file, what's a reflink, and when is it useful?

A reflink is a tool for doing copy-on-write on the filesystem.

If you've heard the term copy-on-write before, I'm willing to bet that it was in the context of the Linux fork call. Let's talk a bit about that.

Copy-on-write when forking a process

When you fork a process in Linux, the new process has a new copy of the old process's memory. This is essential, because if the new process shared the old process's memory, either process could crash if the other process was making an unexpected change to their shared memory. Therefore, Linux needs to make a copy.

However, Linux is smart, and it knows better than to just make a naive copy. Making a naive copy could be a waste of memory, especially if your process has several gigabytes of memory allocated, and you're forking lots of processes for small tasks. If Linux were to make naive copies, you could find yourself with an out-of-memory crash very quickly.

When you fork a process, Linux uses copy-on-write to create the new process's memory. This means that it holds off on making actual copies of the existing memory pages until the last possible moment; which means, the moment when the two processes start having different ideas on what the content of these memory pages should be. In other words, as soon as one of these processes start writing to these memory pages, Linux makes a copy of it, assigning the original page to the original process, and the new copy to the newly-forked process.

This is a huge boon, because most of the time, the new process will either only be reading the memory, or not even that. So many copy actions are avoided thanks to this technique. The beauty part is that these shenanigans are completely transparent to the process, and to the developer who's writing the logic that this process performs. The new process behaves as if it has its own copy of the parent's memory pages, and the floor is being paved ahead of it as it walks forward, so to speak. It'll never even know that copy-on-write was performed.

Now we're ready to talk about reflinks.

Reflinks are copy-on-write for the filesystem

If you read the section above, you already know 90% of what you need to know to understand and use reflinks.

A reflink is a copy of a file, except that copy isn't really created on the hard-drive until the last possible moment. Like the forking version, this logic is invisible. You could do a reflink of a 10 gigabyte file, and the new "copy" would be created immediately, because the 10 gigabytes wouldn't really be duplicated. They'll only be duplicated once you start modifying one of the copies.

All the while, you could treat the reflink as if it was a completely legitimate copy of your original file.

How do you create reflinks?

On Linux, run the following:

$ cp --reflink old_file new_file

On Mac, there's a different flag for some reason:

$ cp -c old_file new_file

If you're creating reflinks programmatically, you could also use dedicated libraries such as this one for Python.

When are reflinks useful?

Here's an example of where I've used reflinks for a client of mine years back. They had a tool for developers that takes their entire codebase and copies it into a Docker container to run tests on it. (Don't ask.)

That recursive copying took a while, and the developers couldn't change their code in the meanwhile, or checkout any other branches, because then an inconsistent version of their code would be copied into the container. That was pretty annoying for me personally, because I was twiddling my thumbs whenever I started the test process.

I figured, why not use reflinks?

I wrote some Python code that creates reflinks to the code in a temporary folder, and then does a real copy from that temporary folder to the Docker container. The big advantage here is that as soon as the reflinks were created, I could modify the original code as much as I wanted, without affecting the tests.

Fortunately, all the developers were using Macs in that company, so I knew I didn't have to worry about filesystem support.

How can reflinks go wrong?

You might be thinking, "What happens if I create a reflink of a huge file, that's bigger than the amount of space I have available on the harddrive?"

PySnooper: Never use print for debugging again

I just released a new open-source project!

Mike Driscoll interviewed me on his blog

I recently had the pleasure of being interviewed by Mike Driscoll on his blog, The Mouse vs. The Python.

Mike is well-known in the Python world and especially in the wxPython user group. He often posts tutorials for beginners on his blog, and it's happened serveral times that I googled a technical question and found the answer in one of his tutorials.

PythonTurtle makes it into Saudi Arabia’s official state curriculum

I just heard some very exciting news.

Six years ago, when I was just starting out my development career, I made a little program called PythonTurtle. It’s a program that helps children learn how to program in the Python programming language, which is the programming language that I use it my day-to-day work as a web developer. I created PythonTurtle as a side project, because I saw there wasn’t a viable solution for children to learn how to program in Python. I figured there should be a solution, so I spent roughly two months of hard work building and releasing PythonTurtle.

Screenshot from the program:

What’s special about PythonTurtle is that it lets children learn programming in an exciting way that puts emphasis on fun and creativity rather than technical details.

When using the program, an illustrated turtle is displayed on the screen, and the children can program it to move around the screen and draw lines. The more programming concepts the children learn, the more impressive drawings they can create with the program. This gives them motivation to learn and improve their skills without feeling that it’s being forced on them by their schoolteacher.

PythonTurtle is based on an educational program called LOGO that was developed in the eighties; what I made is in fact a modern version, so instead of teaching programming in a didactic language, it taught programming in the Python programming language, which is a real language used in the industry today. The idea is to bring children closer to the techniques used in the real world, and possibly plant the seeds of a career in software development.

Because I was just starting out as software developer back then, I didn’t have the skills that I have today, and developing this software was hard for me. There were technical challenges (specifically modifying the wxPython shell to be able to command an auxiliary process.) These challenges were so hard, that it looked like I wasn’t going to solve them, and at a few points I considered giving up on the project entirely. I was asking myself, why am I even doing this? No one even knew I was working on this program, and no one seemed to care.

But I told myself that I’m creating something big here, and it’s important that I see this through to the end. So I did, and I overcame the technical problems.

I released the program as open-source under the MIT license, which means that every person on Earth could download it and use it free of charge. I decided to release it that way rather than as commercial software because I figured more children could use it if it was free, and that seemed more important to me than making a few bucks. I also liked the idea of contributing back to the open-source community, because so much of the software that I use every day is built on open-source software that was made by volunteers, so I was happy to contribute my share of open-source software.

I released the software for download and I submitted a link to the website to tech forums such as HN and Reddit, and over the next few days, the story blew up, and thousands of people visited the website. I was very happy and proud that people liked my project so much.

Over the six years since I’ve released the program, I’ve gotten many happy emails from teachers and parents who used the program to teach their children to program. It’s always heartwarming to get these emails. They come from all over the world: From the States, from the UK, from Africa, Australia, South America… I would occasionally also get emails from children themselves, and one time even from a 80-year-old man who said that he used my program to learn to program himself. I got more reports of adults enjoying using the program. Looking at the analytics for the website, I saw that PythonTurtle was downloaded almost 100,000 times, which made me very proud.

But last year, I’ve noticed something odd. I was checking how the site is doing on Google Analytics and saw that I’m getting a disproportionally large number of hits from Saudi Arabia.

Specifically, there was a big peak of Saudi visitors around January 2014, and than that peak appeared again in January 2015. I also got more feedback emails from people with Arab-sounding names. I investigated why, and found a Saudi forum where PythonTurtle was mentioned. The text was in Arabic, and I tried translating it to English using Google Translate, but the result was too hard to understand, so I let it go and didn’t investigate further.

Until a couple of days ago, I got an email from a teacher from Saudi Arabia about PythonTurtle. He told me that PythonTurtle is being used in all high-schools in Saudi Arabia! The ministry of education of Saudi Arabia has put PythonTurtle into the official state curriculum! This means that it’s being used by more than 4,000 schools which teach more than 700,000 students!!!