#mypy

this is from November 2021, but it was just linked in the release announcement for urllib3 v2.0.0 a few days ago:

Strictly type-hinted APIs

The entire urllib3 v2.0.0 package now has strict type hints included with every install. This means that auto-complete and type-checking tooling like mypy will natively work with the urllib3 module making it even easier to use with confidence. As a part of adding type hints to our entire API, we also wrote an extensive case study for adding type hints to urllib3 and what we learned (and fixed!) along the way.

talks about how they incrementally added type hints to an existing large-ish codebase, including their test suite itself, and the benefits they got from doing that:

When we originally started this journey our primary goal was to provide a good experience for users who wanted to use type checking tools with urllib3. We weren’t expecting to find and fix many defects.

After all, urllib3 has over 1800 test cases, 100% test line coverage, and is likely some of the most pervasive third-party Python code in the solar system.

Despite all of the above, the inclusion of type checking to our CI has yielded many improvements to our code that likely wouldn't have been discovered otherwise. For this reason we recommend projects evaluate adding type hints and strict type checking to their development workflow instead of relying on testing alone.

[...]

We wanted to add Mypy to our continuous integration to ensure new contributors wouldn’t accidentally break type hints but we also wanted to add Mypy incrementally so it wouldn’t grind all our development to a halt.

If you’ve ever tried running Mypy on a single file in a project without any type hints you’re unlikely to only receive errors from the file you ran Mypy on. Instead you’re likely to receive type errors from files where you imported other APIs, both within the project and third-party modules.

This makes adding type hints seem like a daunting task, either needing to be added all at once or with tons of temporary # type: ignore annotations along the way to ensure Mypy continues to pass in CI.

Our solution to this was to maintain a list of files in our project that we knew had correct type hints. Mypy would be run one by one on every file and the list of issues that were detected by Mypy would be gathered up, de-duplicated, and crucially we’d filter out files that weren’t on the “known-good” list.

This meant that once a file was complete we’d add the path to the known-good list to ensure that future contributions wouldn’t regress on our type hints.

The progression from “idiomatic Python” to “optimized Python” to “C” is a one-way process that gradually loses the advantages that brought us to Python in the first place.

The difficult-to-reverse nature of each step means that once you have prototyped out a reasonably optimized data structure or algorithm, you need to quasi-permanently commit to it in order to squeeze out more straight-line performance of the implementation.

Plus, the process of optimizing Python often destroys its readability, for a few reasons:

1. Optimized Python relies on knowledge of unusual tricks. Things like “use the array module instead of lists”, and “use % instead of .format”.

2. Optimized Python requires you to avoid the things that make Python code nicely organized:

--- method lookups are slow so you should use functions.

--- object attribute accesses are slow so you should use tuples with hard-coded numeric offsets.

--- function calls are slow so you should copy/paste and inline your logic

3. Optimized Python requires very specific knowledge of where it’s going to be running, so you lose the flexibility of how to run it: making your code fast on CPython might make it much slower on PyPy, for example. Native extension modules can make your code faster, but might also make it fail to run inside a browser, or add a ton of work to get it set up on a new operating system.

Maintaining good performance is part of your software’s development lifecycle, not just a thing you do once and stop. So by moving into this increasingly arcane dialect of “fast” python, and then into another programming language entirely with a C rewrite, you end up having to maintain C code anyway. Not to mention the fact that rewriting large amounts of code in C is both ludicrously difficult (particularly if your team primarily knows Python) and also catastrophically dangerous. In recent years, safer tools such as PyO3 have become available, but they still involve switching programming languages and rewriting all your code as soon as you care about speed.

So, for Python to be a truly general-purpose language, we need some way to just write Python, and have it be fast.

[...]

Mypyc takes the annotations meant for Mypy and generates C with them, potentially turning your code into a much more efficient extension module. As part of Mypy itself, it does this with your existing Python type-hints, the kind you’d already use Mypy with to check for correctness, so it doesn’t entail much in the way of additional work.

I’d been curious about this since it was initially released, but I still haven’t had a hard real-world performance problem to really put it through its paces.

So when I learned about the High Throughput Fizzbuzz Challenge via its impressive assembler implementation that achieves 56GiB/s, and I saw even heavily-optimized Python implementations sitting well below the performance of a totally naïve C reference implementation, I thought this would be an interesting miniature experiment to use to at least approximate practical usage.

Cómo Mypy podría simplificar la compilación de Python

El proyecto de comprobación de tipo estático de Mypy para Python está explorando formas en que podría ayudar con la compilación sin esfuerzo de Python en C o lenguaje de máquina.

Es el sueño de todos los programadores profesionales de Python: tomar una aplicación Python existente, ejecutarla a través de un compilador y generar código de alta velocidad y plataforma nativa respetando la naturaleza…

Python 3.5 introduced type hinting which I think is an awesome addition to the language. However, as the PEP 484 outlined there is no type checking happening at the runtime, and the primary rationale of the PEP seems to be adding a standard language for type annotations to enable static analysis. Still, the PEP says that type hinting opens Python potentially for runtime type checking while it emphasizes that Python will remain a dynamic language.

I think type checking at the runtime could work well; I have been programming recently in the PHP land, and I like PHP’s type hinting. However, you can turn on strict types on/off in PHP, which you (still?) don’t have in Python. Nevertheless, type hinting makes development easier even if you don’t have type checking at the runtime thanks to easier code completion in IDEs and making static analysis more powerful.

I think mypy type checker is a very useful tool. You can run it to check whether you have any type related bugs in your code. I think integrating it into your build pipeline is not a bad idea, and you might be able to catch some nasty bugs with it. Below a very short introduction to mypy:

Consider the following:

class A: def __init__(self, i: int): self.i = i def should_return_int(self, string: str)->int: return str(self.i) + string

If you have ran the code on mypy, you would get the following:

typehints.py: note: In member "__init__" of class "A": typehints.py:2: error: The return type of "__init__" must be None typehints.py: note: In member "should_return_int" of class "A": typehints.py:6: error: Incompatible return value type (got "str", expected "int")

Type hinting also makes it easier to deal with abstract classes (and inheritance in general):

import abc class AbstractClass(abc.ABC): @abc.abstractmethod def some_getter_method(self)->str: pass @abc.abstractmethod def some_setter_method(self, name: str): pass class ConcreteClass(AbstractClass): def some_getter_method(self) -> str: return self.name def some_setter_method(self, name: str): self.name = name def expects_abstract_class(argument: AbstractClass)->str: return argument.some_getter_method() concrete_instance = ConcreteClass() concrete_instance.some_setter_method("John Doe") expects_abstract_class(concrete_instance) expects_abstract_class(object())

Running the code on mypy would yield the following warning: