Polis Realized | Economics and Politics

I'm working on a program where the main bottleneck is network io. I originally implemented concurrent.futures.ThreadedPool to speed up the process. It turns out there is a better asynchronous io module for my use case, though gevent. Profile results:

Results for concurrent.futures.ThreadPool:

5319963 function calls (5318875 primitive calls) in 359.541 seconds Ordered by: internal time List reduced from 872 to 50 due to restriction <50> ncalls tottime percall cumtime percall filename:lineno(function) 31 349.876 11.286 349.876 11.286 {method 'acquire' of '_thread.lock' objects} 2652 3.293 0.001 3.293 0.001 {method 'recv' of '_socket.socket' objects} 310270 0.790 0.000 0.790 0.000 {method 'timetuple' of 'datetime.date' objects} 25 0.661 0.026 0.661 0.026 {method 'recv_into' of '_socket.socket' objects}

Results for gevent fork (https://github.com/fantix/gevent):

While writing some code today for the XBRL project, I thought, "Writing the code to test this code is going to be a headache." Then, I realized I had been writing test code for years and deleting it when I was done with it. This motivated me to figure out one of python's testing libraries. 

I picked pytest because it looked simple. It is simple. After writing my first test, I ran into a problem with the customizing my PYTHONPATH variable on a project specific basis. I have the file structure:

projectdirectory\

normalmain.py

.modulea

.moduleb\

..test.py

..code_to_test.py (which imports modeula)

When I run pytest moduleb/test.py I get an error than modulea cannot be found! Since I normal program execution means running normalmain.py I want to keep the top level directory on my path.

I found a solution and also stumbled across a better way (for my usecase) to implement virtualenv than autowrapper in the process. Oh-my-zsh includes a plugin virtualenvwrapper which will automatically calls workon <virtualenv> when you enter a directory with a .venv file with the name of an existing virtualenv in it. This isn't anything to write home about, since autowrapper will do the same thing. But, when you to another directory that's not a subdirectory of the .venv directory, the Oh-my-zsh plugin will also call deactivate <virtualenv>. 

To implement the solution:

1. Create a .venv file with the project's virtualenv name in the project's top level directory. 

2. In the project's virtualenv directory, add the following to postactivate:

export OLD_PYTHONPATH=$PYTHONPATH

export $PYTHONPATH=<project top level directory>:$PYTHONPATH

3. In the project's vritualenv directory, add the following to the predeactivate:

export PYTHONPATH=$OLD_PYTHONPATH

Saving and investing is neither fun nor a hobby for most people, but its an important personal and social responsibility for every citizen. Economic activity is essential for putting food on your plate, a roof over your head, hospitals with well trained medical professionals and sterile rooms in your city, and pretty much everything else you've ever enjoyed, experienced, or taken for granted.

It's your responsibility as an sovereign citizen in a country with free, transparent markets and the luxury of legal recourse for the capitalization of forms of almost all possible economic activity to contribute to its capital base and scrutinize the expected outcomes of particular applications of your capital to productive activity.

As a practical matter, participating in capital markets can be a positive or dangerous to your financial health because of the variety of quality and shoddy brokers available. I put together a google doc comparing some brokers along features important to me recently because I want to open a second brokerage account. Check out of the google sheet here. 

I've discovered a more mature python project with XBRL support for SEC filings, Arelle.

The GUI makes little sense to me but the command line program seems to work well. I'll have more notes on functionality after I try using the program more. 

More importantly, Arelle also has a built-in Sphinx interpreter, which I've found zero documentation for besides some example code but allegedly does all the heavy lifting in extracting and aggregating data from XBRL documents and the syntax looks easy.

Also important is that the key XBRL word to look for is concept. XBRL explanations look like a foreign language but XBRL concepts correspond to actual line items of financial statements like net income. Next up is figuring out where the official list of XBRL concepts exists.

Good financial data for US public companies in a machine readable format was hard to come by until very recently.  I had a mess of programs that downloaded data from ADVFN and finance.google.com but the results had lots of errors and missing data points. The data was not worth the time I put into the projects.

I discovered XBRL recently and have only a very superficial understanding of how it works, but it looks perfect for getting financial data from SEC filings. Plus, the SEC requires companies submit financial statements in XBRL and the SEC hosts the filings on a publicly accessible FTP!

Programming is a staple of economics and research but the learning curve for incorporating programming environment best practices is steep. Figuring out best practices and then implementing them is time consuming and it doesn’t get done until the damage of not following best practices is already done.

Avoid the headaches and panic. Follow the outline below for setting up a python programming environment, starting a project, and maintaining the project. 

Installing Python 3

Make sure Python 3 will not break your code. 

Install Python 3.

Update PATH with path to the Python 3 bin.

If you’ve got several versions of python installed you can use pip<versionumber>, e.g. pip3.4, to specify a interpreter.

Install setuptools.

Install pip.

Install virtualenv.

Install virtualenvwrapper.

Add WORKON_HOME=~/Envs to your shell start up file.

Find / -name virtualenvwrapper.sh

Add source <pathfromstep9> to your shell start up file.

Add PIP_RESPECT_VIRTUALENV=true to your shell start up file.

Add PIP_VIRTUALENV_BASE=$WORKON_HOME to your shell start up file.

Add VIRTUALENVWRAPPER_VIRTUALENV_ARGS='--no-site-packages' to your shell start up file. 

Add VIRTUALENVWRAPPER_PYTHON=<pathtopythoninterpreter>. Use find if necessary.

Add VIRTUALENVWRAPPER_VIRTUALENV=<pathtovirtualenv>. Use find if necessary.

Install autoenv.

Find / -name autoenv/activate.sh

Add source <pathfromstep15> to your shell start up file.

Starting a Project 0. mkdir <project> 1. mkvirtualenv environmentname or skip if you want to use an environment you’ve already created. 2. touch project/.env 3. echo “workon <environmentname>” > <project>/.env. This is also a great place to reminders like echo “When is the last time you stretched?” 4. cd project 5. git init 6. touch .gitignore 7. echo “.env > .gitignore” 8. Create a repo on github or bitbucket. Pick github if you want to share your code. Pick bitbucket if you want to keep your code private. 9a. If you’re using github: git remote add origin https://github.com/username/myproject.git 9b. If you’re using bitbucket: git remote add origin https://[email protected]/username/myproject.git

Working On A Project 0. Do some work on the project. 1. Update .gitignore 2. git add * 3. git commit * 4. git push -u origin --all