Python namespace packages
What are namespace packages?
Namspace package setup.py
Namespace package demonstration
What are namespace packages?
Namespace packages are an extension of the existing Python import system. They allow:
Large software projects to be broken up into smaller parts.
Grouping functionality under a common project or company name.
Delivery of individual modules without needing all parts of a large project.
Reuse of package names which may not normally be available.
There are two typical use cases which namespace packages help with.
Lets say a ficticous software company called WidgetCo had come up with their own TCP socket module. Its socket module provides extra abilities not in the standard Python version.
How would WidgetCo name the package? They can't call it socket as this is used. Typically it might get named widgetcosocket for example. Its a bit long but it works and doesn't conflict. Now WidgetCo goes on to develop their own web framework, database layer and maybe various business logic modules. These new modules get called widgetcoweb, widgetcodb, widgetcounderpants, widgetcostep2 and widgetcoprofit.
Is there a better way to name the packages? Yes, using namespace packages would allow WidgetCo to define a top level "namespace". The individual modules could then be delivered under this. For example WidgetCo could delivery their modules under the "wc" namespace. The packages would then be available as:
from wc import socket
from wc import web
from wc import db
:
etc
The wc.socket, wc.web, wc.db would be delivered from eggs called wc-socket, wc-web, wc-db etc.
This time the same WidgetCo company has been developing from the begining with wc as the main package. Beneath this the other packages socket, web, db, etc live. On the file system there is a single checkout of the company's source code. After a period of time this code checkout has become quite large. It is now difficult to develop individual parts due to the checkout size and large amount of code. The companies products need some parts developing faster then others.
Is there a way to break the code up into logical packages, which preserve the imports and allows them to be individually developed? Yes, namespace packages allow you to break up a large project into logical parts. Dependancies and version numbers for the new packages can be used more effectively, to build WidgetCo's products on specific package versions. Logical parts can be tested and developed more easily. This means development can proceed more easily with greater flexibilty. Which in turn increases productivity and the WidgetCo's bottom line.
In a previous article I showed how to create a private egg repository and how to make and egg package. I'm going to build on that reusing the setup.py. I've created three packages on PyPI for this article:
wc-socket : http://pypi.python.org/pypi/wc-socket
wc-db : http://pypi.python.org/pypi/wc-db
wc-web : http://pypi.python.org/pypi/wc-web
You can download the source for these and decompress the archive to disk. wc-db and wc-web depend on wc-socket. You could also install the eggs using easy_install and they will be download and installed into your python.
I tend to favour the following file system layout for an egg package. To make a package for the "wc" namespace I create the following folders and files:
<wc package name socket | db | web>
|
+-- lib
| |
| +-- <wc>
| |
| +-- __init__.py : special namespace code goes in this file.
| |
| +-- <socket | db | web>
| |
| +-- package contents files
|
+-- scripts : command line tool(s)
|
+-- setup.py : distutils file configured for the namespace package.
The wc package __init__.py needs to contain the following code:
# See http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
try:
__import__('pkg_resources').declare_namespace(__name__)
except ImportError:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
No other code should be put in this file as it can cause problems. The setuptools namespace documentation covers this in more detail. The package inside wc web, db or socket can contain any files you would normally put there.
Namspace package setup.py
To make a package you need a setup.py and to make it aware of your namespace package you need to add "namespace_package" keyword to the setup() function e.g.:
setup(
:
namespace_packages = ['wc'],
:
)
The following is a the wc-socket setup.py file I use. The files used for wc-web and wc-db are almost exactly the same save the description and dependance part. If you get the code for these you can see this in more detail on this.
This is demonstrating how to create namespace packages.
This should eggify and in theory upload to pypi without problems.
This is released under the BSD license.
Oisin Mulvihill
2010-05-31
"""
try:
from setuptools import setup, find_packages
except ImportError:
from ez_setup import use_setuptools
use_setuptools()
from setuptools import setup, find_packages
Name='wc-socket'
ProjecUrl=""
Version='1.0.0'
Author='Oisin Mulvihill'
AuthorEmail='oisinmulvihill at gmail dot com'
Maintainer=' Oisin Mulvihill'
Summary='WidgetCo socket: This is a fake socket module use to teach namespace packages .'
License='BSD License'
ShortDescription=Summary
Description=Summary
needed = [
]
# Include everything under package dir. I needed to add a __init__.py
# to each directory inside package. I did this using the following
# handy command:
#
# find lib/wc -type d -exec touch {}//__init__.py \;
#
# If new directories are added then I'll need to rerun this command.
#
EagerResources = [
'wc',
]
ProjectScripts = [
## 'scripts/runweb',
]
PackageData = {
'': ['*.*'],
}
# Make exe versions of the scripts:
EntryPoints = {
}
setup(
# url=ProjecUrl,
name=Name,
zip_safe=False,
version=Version,
author=Author,
author_email=AuthorEmail,
description=ShortDescription,
long_description=Description,
license=License,
scripts=ProjectScripts,
install_requires=needed,
include_package_data=True,
packages=find_packages('lib'),
package_data=PackageData,
package_dir = {'': 'lib'},
eager_resources = EagerResources,
entry_points = EntryPoints,
namespace_packages = ['wc'],
)
Namespace package demonstration
Recover wc-socket, wc-db and wc-web archives. Decompress them and then set the package up in development mode. Alternatively run the command:
(p2)oisin@banter [tmp]> easy_install wc-socket wc-web wc-db
Processing wc-socket
Running setup.py -q bdist_egg --dist-dir /home/oisin/src/CMDev/src/tmp/wc-socket/egg-dist-tmp-bL0ti7
Adding wc-socket 1.0.0 to easy-install.pth file
Installed /home/oisin/src/CMDev/src/tmp/p2/lib/python2.6/site-packages/wc_socket-1.0.0-py2.6.egg
Processing dependencies for wc-socket==1.0.0
Finished processing dependencies for wc-socket==1.0.0
Processing wc-web
Running setup.py -q bdist_egg --dist-dir /home/oisin/src/CMDev/src/tmp/wc-web/egg-dist-tmp-M6uT43
Adding wc-web 1.0.0 to easy-install.pth file
Installing runweb script to /home/oisin/src/CMDev/src/moneyio/tmp/p2/bin
Installing runweb script to /home/oisin/src/CMDev/src/moneyio/tmp/p2/bin
Installed /home/oisin/src/CMDev/src/tmp/p2/lib/python2.6/site-packages/wc_web-1.0.0-py2.6.egg
Processing dependencies for wc-web==1.0.0
Finished processing dependencies for wc-web==1.0.0
Processing wc-db
Running setup.py -q bdist_egg --dist-dir /home/oisin/src/CMDev/src/tmp/wc-db/egg-dist-tmp-yvuSX6
Adding wc-db 1.0.0 to easy-install.pth file
Installing rundb script to /home/oisin/src/CMDev/src/tmp/p2/bin
Installing runweb script to /home/oisin/src/CMDev/src/tmp/p2/bin
Installed /home/oisin/src/CMDev/src/tmp/p2/lib/python2.6/site-packages/wc_db-1.0.0-py2.6.egg
Processing dependencies for wc-db==1.0.0
Finished processing dependencies for wc-db==1.0.0
(p2)oisin@banter [tmp]>
The packages wc-web and wc-db depend on wc-socket. The setup.py of web and db explicity list wc-socket and a particular version string as a dependancy.
The web and db modules also provide basic command line programs runweb and rundb:
(p2)oisin@banter [tmp]> runweb
wc-socket:
----------
>>> wc.web.hello.main: hello world
(p2)oisin@banter [tmp]> rundb
wc-socket:
----------
>>> wc.db.hello.main: hello world
The socket module provides a write method that both rundb and runweb call. This is used to test the namespace package set up. This could be done manually as follows:
(p2)oisin@banter [tmp]> python
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from wc import socket
>>> from wc import web
>>> from wc import db
>>>
>>>
(p2)oisin@banter [tmp]>
Developing namespace packages is quite straight forward. For large scale projects is allows individual parts to developed individually and delivered under a common namespace. For small utilities it may not be worth the effort, however if your project grows its worth keeping in mind. Go forth and claim those name spaces! :)
