Code of Feihong

This short script makes it to easy to start a SOCKS proxy and then configure the entire system to use it. This means that GUI programs like your web browsers will use the proxy without any additional configuration on your part. You can kill the SOCKS proxy by pressing ENTER, which will also turn off the system-wide proxying.

""" Script to extract audio from video files and convert to mp3. On Ubuntu, install ffmpeg along with libavcodec-extra-53 to get MP3 codec. sudo apt-get install ffmpeg libavcodec-extra-53 """ import sys import os.path import subprocess if __name__ == '__main__': ext = sys.argv[1] # Show a menu of all files in the current directory. files = [f for f in os.listdir(u'.') if f.endswith('.' + ext)] if not len(files): print 'No files with extension %s' % ext sys.exit(0) for i, f in enumerate(files, 1): print '%d. %s' % (i, f) choice = int(raw_input('Choose a file: ')) input_file = files[choice - 1] output_file = os.path.splitext(input_file)[0] + '.mp3' cmd = [ 'ffmpeg', '-i', input_file, '-vn', # no video output output_file ] print ' '.join(cmd) subprocess.call(cmd) print 'Converted:\n%s\n' % input_file

Using arrow:

import arrow dt = arrow.get(2014, 7, 3, 10, 45) dt.to('US/Central')

Using dateutil.tz:

import datetime from dateutil import tz dt = datetime.datetime(2014, 7, 3, 10, 45) dt = dt.replace(tzinfo=tz.gettz('US/Central')) dt.astimezone(tz.gettz('UTC'))

Using pytz:

import datetime import pytz dt = datetime.datetime(2014, 7, 3, 10, 45) dt = pytz.timezone('US/Central').localize(dt) dt.astimezone(pytz.utc)

Although you rarely need it, setting up a reverse proxy for your Django development server can be a lifesaver (usually when you need to run code that only works in https).

Make it so that your /etc/nginx/sites-enabled/default points to a file with contents like this:

server { listen 80; server_name dev.mysite.com; access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; location / { proxy_pass http://127.0.0.1:8000; proxy_set_header X-Real-IP $remote_addr; proxy_read_timeout 600; } } server { listen 443 default_server ssl; server_name dev.mysite.com; ssl_certificate /etc/nginx/ssl/mysite-ssl.crt; ssl_certificate_key /etc/nginx/ssl/mysite-ssl.key; ssl_ciphers ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:!kEDH; access_log /var/log/nginx/ssl_access.log; error_log /var/log/nginx/ssl_error.log; location / { proxy_pass http://127.0.0.1:8000; proxy_set_header X-Real-IP $remote_addr; proxy_read_timeout 600; } }

Some notes:

Make sure the line "127.0.0.0.1 dev.mysite.com" is in your /etc/hosts file.

This config assumes your Django dev server is running on port 8000.

This config assumes that you've put your SSL certificate files in /etc/nginx/ssl.

Note the high value for proxy_read_timeout (10 minutes). This should give you ample time to debug something in pdb while nginx waits for a response from your dev server.

I recently had a problem where I was unable to update procps. I let it go for a while because I don't know what this package is and everything that I needed was working fine. But then one day I wanted to install rubygems and I got an error from procps so I decided to finally fix it.

I recently did this, although it's possible that I should not have, because it totally screwed up QGIS (which I resolved by reinstalling it). It's possible that the best way to install GDAL 1.9 is to go through UbuntuGIS. However, this is what I did, and it worked, so I'll document it.

sudo apt-get install build-essential python-all-dev cd /tmp wget http://download.osgeo.org/gdal/gdal-1.9.2.tar.gz tar xvfz gdal-1.9.2.tar.gz cd gdal-1.9.2 ./configure --with-python make sudo make install

After running make install, you may need to modify LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

If you've done everything correctly, you should be able to run ogrinfo --version and see this:

I needed something like Python's functools.partial function. This is a succinct way to implement it in CoffeeScript:

partial = (fn, part_args...) -> return (args...) -> fn.apply(null, [part_args..., args...])

Yes I know you can do this with shell scripts but I wanted to write it in Python, OK?

Sometimes you get a layer that just has way, way too many vertices for your needs. QGIS can help you reduce the number of vertices while preserving the general shape of the geometries. Here's how:

Make sure the layer is in a format that QGIS can edit. You may need to convert the layer to ESRI Shapefile before you can proceed.

Click on the Edit button (has a pencil icon). You should see the vertices now.

Zoom in on a region where the vertices are very tightly packed together.

From the menu, select Vector -> Geometry Tools -> Simplify geometries.

In the modal dialog that pops up, just click OK. The default tolerance of 0.0001 degrees is usually fine.

Another modal asks about making sure your layer's attribute tables are closed. If that's true, click Yes.

A result dialog appears, informing you how many vertices were in the original dataset and how many there are now.

After going through these steps, you should be able to verify, visually, that the number of vertices is much smaller. In the dataset I was just working with, I went from 33,389 vertices to just 4,847 (a reduction of 85%).

If the geometries are still too complex then repeat the process with a higher simplify tolerance.

This isn't super hard to figure out, but since I need to puzzle it out every time I'm going to record it here.

ogr2ogr -f "GeoJSON" -t_srs EPSG:4326 output.geojson input.shp

The above command reprojects from the source SRS of the shapefile to EPSG:4326 (i.e. latitude/longitude). It also converts the data to the GeoJSON format. Note that the output file name comes first in the argument list.

You don't want to just apt-get install qgis since you'll get a version that might be a bit too old for your needs. First, open sources.list for editing:

sudo gedit /etc/apt/sources.list

Add the following lines:

deb http://qgis.org/debian precise main deb-src http://qgis.org/debian precise main

Save sources.list and then run:

# Add qgis.org repository public key to apt keyring. gpg --keyserver keyserver.ubuntu.com --recv 47765B75 gpg --export --armor 47765B75 | sudo apt-key add - sudo apt-get update sudo apt-get install qgis python-qgis

Here's the code:

import os import os.path as op import subprocess image_files = [f for f in os.listdir('.') if op.splitext(f)[1].lower() == '.jpg' and not f.startswith('re_')] for image_file in image_files: cmd = ['convert', image_file, '-resize', '1024x1024>', 're_' + image_file] print ' '.join(cmd) subprocess.call(cmd)

Note that the resizing does not change the aspect ratio of the image, it just forces the resized image to fit within a 1024x1024 box. Meaning the new image will be no taller than 1024px and no wider than 1024px. Resized images all have "re_" prepended to their file names. 

Before you can run the script, you must have imagemagick installed. On Ubuntu, this is trivial:

Yes, I know that you basically just download it and then start using it, but here's a little script that does all of that for you without you needing to really think about it too much.

cd /tmp wget http://downloads.sourceforge.net/qooxdoo/qooxdoo-3.0-sdk.zip unzip qooxdoo-3.0-sdk.zip sudo mv qooxdoo-3.0-sdk /opt/ firefox /opt/qooxdoo-3.0-sdk/documentation/manual/index.html

Also add the following line to your ~/.bashrc file:

function qx_create_application() { python2 /opt/qooxdoo-3.0-sdk/tool/bin/create-application.py --name=$1 }

I suspected that context managers inside coroutines should have their __exit__ methods invoked when a coroutine generator is closed. But I wanted to be sure, so I wrote a couple simple programs to test this theory.

def coroutine(fout): with fout: while True: value = (yield) fout.write(value) if __name__ == '__main__': fout = open('test.txt', 'w') gen = coroutine(fout) print 'fout is closed:', fout.closed gen.next() # prime the generator gen.send('hello\n') gen.send('world\n') gen.send('radical\n') gen.close() print 'fout is closed:', fout.closed

This works exactly as expected. The output is:

fout is closed: False fout is closed: True

But what about context managers that you create on your own?

from contextlib import contextmanager @contextmanager def tag(name): print "<%s>" % name try: yield finally: print "</%s>" % name def coroutine(name): with tag(name): while True: value = (yield) print value if __name__ == '__main__': # Normal usage: with tag('h1'): print 'foo ' print 'bar ' print 'basilisk ' # Using with a coroutine: gen = coroutine('h2') gen.next() # prime the generator gen.send('hello') gen.send('world') gen.send('gorgonzola') gen.close()

Again, this works as expected. The output is:

<h1> foo bar basilisk </h1> <h2> hello world gorgonzola </h2>

There's a binary package for Python that I wanted to play with, but I couldn't easily figure out how to build it. Then I noticed that there was a brew formula for it, so I finally decided to try out a brewed Python. 

Installing Python 3 via Homebrew is simple:

brew install python3

One thing that I liked right away about brewed Python is that IPython was so easy to install. For example, I wasn't forced to use easy_install to get the right version of readline, and anyway I always forget to do that.

Viewing the source and looking for an RSS link does not work for FaceBook pages. But rest assured, every FB page does in fact have an associated RSS feed. You basically have to find the profile ID for the page, then use that to make the URL. All FB RSS URLs look like this:

https://www.facebook.com/feeds/page.php?id=PROFILE_ID&format=rss20

It's quite easy to create a bookmarklet that takes you directly to the RSS feed. First, drag the following link into your bookmark toolbar:

FaceBook page RSS

Then open the edit dialog for the bookmark and paste the following code into the URL field:

javascript:(function() { var html = document.body.innerHTML; var groups = html.match(/"profile_id":(\d+),/); if (groups) { var profileId = groups[1]; var rssUrl = 'https://www.facebook.com/feeds/page.php?id=' + profileId + '&format=rss20'; window.location.href = rssUrl; } })();

It's best when a dataset just gives you coordinates in WSG84 (SRID: 4326), but it's not the end of the world if they use a different coordinate system. To transform their coordinates to latitude and longitude, you just need to know the SRID of their coordinate system.

Unfortunately, your contact person for the dataset might not know the SRID. But if she knows the name of the coordinate system then there's still a way for you to obtain the SRID. The following is how I do it.

1. Get some reference points in their coordinate system. These are points for which you know the equivalent latitude and longitude coordinates. This is usually trivial, because most datasets have an address field. To get the latitude and longitude, you can use Google Maps: search for an address, and, on the result page, right-click the red pin and select "What's here?". The latitude and longitude will appear in the search bar.

2. Do a search on spatialreference.org. Entering all the keywords sometimes gives you nothing, because the actual name in their database will be spelled slightly differently than you expect, and their search engine only does exact matches. Pick the most important keyword and search on that first. You may get too many results, but from there you can start winnowing them down.

3. Write a program to try out all possible SRIDs. You might end up with several possible SRIDs. You are unlikely to be able to guess the correct one by reading their descriptions. Instead, write a simple program using GeoDjango to determine the most likely SRID. Here is an example:

from django.contrib.gis.geos import Point # Reference point. Note that the x coordinate comes first. COORDINATES = (397162, 140182) # Reference address: 3100 - 3299 BLOCK OF 14TH STREET NW, dc # Reference latitude, longitude from Google Maps: 38.929527,-77.032734 # Candidate SRIDs from spatialreference.org. At least one of them is right. SRIDS = [2248, 2804, 2893, 3559, 3583, 6620, 6654, 7058, 26785, 26985, 102285, 102685] for srid in SRIDS: pt = Point(*COORDINATES, srid=srid) try: pt.transform(4326) print '{}: {}, {}'.format(srid, pt.y, pt.x) except Exception: pass

The output will look something like this:

2248: 38.0078526925, -80.1766510551 2804: 38.9295130132, -77.0327305535 2893: 38.0078526925, -80.1766510551 3559: 38.9295130132, -77.0327305535 3583: 42.255458399, -71.7461411817 ERROR 6: EPSG PCS/GCS code 6620 not found in EPSG support files. Is this a valid EPSG coordinate system? ERROR 6: EPSG PCS/GCS code 6654 not found in EPSG support files. Is this a valid EPSG coordinate system? ERROR 6: EPSG PCS/GCS code 7058 not found in EPSG support files. Is this a valid EPSG coordinate system? 26785: 38.2098999912, -78.4016791905 26985: 38.9295130132, -77.0327305535 102285: 38.9295130132, -77.0327305535 102685: 38.0078526924, -80.1766510563