Kevin Says "Code!"

X11 forwarding is one of the neatest tricks that X puts in your toolbelt: you can run a graphical application on a remote machine (say, a server off in a closet or in the cloud somewhere) and have it magically appear on your desktop.

Even better, you shouldn't have to open firewall ports or find, install, and configure some new software package; the SSH utilities you already have are capable of X11 forwarding, which allows the remote application to connect back to your local X server just by adding the -X option, like so:

$ ssh -X myserver

But wait! On my Ubuntu 11.10 systems, this is silently ignored---I connect to the remote system all right, but graphical applications fail with errors about not being able to find a display. Typing

$ echo $DISPLAY

and getting a blank line confirms that no X display has been set up for us. What gives?

If you Google around, you will find lots of advice pointing you to /etc/ssh/ssh_config (client configuration) and /etc/ssh/sshd-config (server configuration). This seems reasonable, but the defaults actually work perfectly well, and uncommenting the various X11-related lines you will find won't fix the problem.

Instead, connect to the remote server, and install xauth with

$ sudo apt-get install xauth

Lenny only has Python 2.5.x, which is ancient (as ancient as Lenny is), and there's no official backport. Trying to ./configure && make && checkinstall make install gets you a half-functional Python install; you need to install a few packages before building. But what are they? Python doesn't seem to want to tell you. Give this a go:

apt-get install zlib1g-dev libbz2-dev libsqlite3-dev libgdbm-dev libncursesw5-dev libreadline-dev libz-dev tk-dev libdb-dev

The new Pandora looks shiny, but has never worked in my browser (Chromium under 64-bit Ubuntu 11.04). All I get are errors in the JavaScript console about media URLs. I even went so far as to contact Pandora support (who are apparently using Assistly these days) and, as polite as they were, they didn't come to a resolution. So here it is! While I was digging around in the jPlayer mailing list archives (jPlayer being the JavaScript media-playing library that Pandora's redesign uses) I found a suggestion that the issue might be related to codec availability. It's weird that that would produce path errors instead of, say, "unsupported media type" errors or some such, but stranger things have happened. It turns out that they're right! Just a simple

apt-get install chromium-codecs-ffmpeg-extra

Converting from a timestamp to a datetime object is easy with datetime.fromtimestamp(), but it feels like I spend ten minutes swearing any time I need to go the other direction. Here are a few solutions from this StackOverflow post.

When you need more control than Python's built-in urllib provides, the libcurl library (the brains behind the curl tool) is a nice option. Python bindings are available in PyPI as pycurl. I just spend a little while tearing my hair out, so here are a few things I've figured out.

First, the basic routine (roughly the equivalent of urlopen(x).read()) goes as follows. (This is heavily based on this work by David McLaughlin.)

class Response(object): def __init__(self): self.chunks = [] def add_chunk(self, chunk): self.chunks.append(chunk) @property def content(self): return ''.join(self.chunks) def read(url): response = Response() curl = pycurl.Curl() curl.setopt(curl.URL, url) curl.setopt(curl.WRITEFUNCTION, response.add_chunk) curl.perform() return response.content

With setopt(), you can add any of the options for curl_easy_setopt(), just without the CURL_ prefix.

The first thing that I wanted to do was to to use an X509 client certificate for authentication. This is as easy as adding curl.setopt(curl.SSLCERT, 'some-file-name.crt').

With the curl command-line tool, the -uc and -us options turn off peer certificate validation (do we trust the issuer of the server's certificate?) and host verification (does the name on the cert match who you're talking to?). These are very useful for self-signed certificates, such as you might use in development environments. The equivalent pycurl options are:

curl.setopt(curl.SSL_VERIFYPEER, 0) curl.setopt(curl.SSL_VERIFYHOST, 0)

The last thing that I needed was to be able to post serialized JSON to the server (although this should work with any data you have). This works as follows (where data is the output for json.dumps(), for me):

curl.setopt(curl.POST, 1) curl.setopt(curl.POSTFIELDS, data)

However, I was intermittently getting weird SSL errors on the server side:

WARNING:root:Read error on 36: [Errno 1] _ssl.c:1350: error:0D06407A:asn1 encoding routines:a2d_ASN1_OBJECT:first num too large

The same requests would work just fine using the command-line tool, so I knew it had to be something I was doing wrong. ASN1 is a type of transfer encoding, so that was a first hint, but it took me a little bit longer than I'd have liked to figure out what I was doing wrong. Thankfully, the fix is quite simple.

At first I thought that urlencoding the data I was POSTing with urllib.quote_plus() would fix this; but libcurl does that automatically, it turns out, unless you tell it that the data has already been urlencoded. This reduced the frequency of the issue (perhaps half of requests, from about 90%) but did not eliminate it.

Ultimately I went to the source of curl to see what they were doing differently. The only discrepancy was that I was not manually setting the length of the data to be POSTed. Aha: a blinding flash of the obvious! pycurl is a very thin layer over libcurl; when you do not provide a length, libcurl uses strlen() to find the length of the data instead of len() on the Python side. When you're posting arbitrary binary data, this is clearly a problem! Adding the following line fixed the issue:

curl.setopt(curl.POSTFIELDSIZE, len(data))

(Edit: One last tip! Once you perform() the request, you can get the HTTP result code by calling curl.getinfo(curl.RESPONSE_CODE).)

I run into this one every so often, and always have to look around until I find a good answer.

str = '2007-03-04T21:08:12' import datetime def parse_datetime(str): return datetime.datetime.strptime(str, '%Y-%m-%dT%H:%M:%S') print parse_datetime(str)

If you don't mind the extra dependency on python-dateutil (which is available from PyPI), Wes Winham suggests an even cleaner alternative that works with pretty much whatever you can throw at it:

str = '2007-03-04T21:08:12' import dateutil.parser print dateutil.parser.parse(str)

I recently got a chance to play around with Tornado, which is pretty neat (although that certainly isn't news). One thing that I tried to do pretty quickly and had a hard time with was Basic authentication (you know, the little "so-and-so requires a username and password" pop-up). Paulo Suzart posted a working example over on gist, but it was a bit light on context, and Dhanan Jaynene's request interceptors are a bit overkill for this purpose (although very useful for more complex behavior!). Let's start with the "hello world" example from the Tornado site, and I'll show you what I've cooked up. I'm only an hour or two into exploring Tornado, like I hinted at above, so if you see any room for improvement, I'd love to hear from you. (I've made all the code I've written very verbose in the hopes that'll help people understand and customize it without much trial-and-error. Feel free to tighten things up.)

import tornado.ioloop import tornado.web class MainHandler(tornado.web.RequestHandler): def get(self): self.write("Hello, world") application = tornado.web.Application([ (r"/", MainHandler), ]) if __name__ == "__main__": application.listen(8888) tornado.ioloop.IOLoop.instance().start()

Now, what I really wanted was to be able to add a decorator to any RequestHandler that (without any other modifications) would perform all the back-and-forth with the browser, so that I could write something like this:

@require_basic_auth class MainHandler(tornado.web.RequestHandler): def get(self): self.write("Hello, world")

If you're not familiar with Python decorators, in the simplest usage, this will effectively lead to the function require_basic_auth being invoked with MainHandler (the class) as its argument. Whatever that function returns will replace the definition of MainHandler. Nifty, huh? Let's see how we can use this. We could mess with the get(), put(), etc. methods on RequestHandler, but it turns out that there's a method that Tornado invokes above them called _execute(). If we replace that one, we'll be able to do all of this in one fell swoop.

def require_basic_auth(handler_class): # Should return the new _execute function, one which enforces # authentication and only calls the inner handler's _execute() if # it's present. def wrap_execute(handler_execute): # I've pulled this out just for clarity, but you could stick # it in _execute if you wanted. It returns True iff # credentials were provided. (The end of this function might # be a good place to see if you like their username and # password.) def require_basic_auth(handler, kwargs): auth_header = handler.request.headers.get('Authorization') if auth_header is None or not auth_header.startswith('Basic '): # If the browser didn't send us authorization headers, # send back a response letting it know that we'd like # a username and password (the "Basic" authentication # method). Without this, even if you visit put a # username and password in the URL, the browser won't # send it. The "realm" option in the header is the # name that appears in the dialog that pops up in your # browser. handler.set_status(401) handler.set_header('WWW-Authenticate', 'Basic realm=Restricted') handler._transforms = [] handler.finish() return False # The information that the browser sends us is # base64-encoded, and in the format "username:password". # Keep in mind that either username or password could # still be unset, and that you should check to make sure # they reflect valid credentials! auth_decoded = base64.decodestring(auth_header[6:]) username, password = auth_decoded.split(':', 2) kwargs['basicauth_user'], kwargs['basicauth_pass'] = username, password return True # Since we're going to attach this to a RequestHandler class, # the first argument will wind up being a reference to an # instance of that class. def _execute(self, transforms, *args, **kwargs): if not require_basic_auth(self, kwargs): return False return handler_execute(self, transforms, *args, **kwargs) return _execute handler_class._execute = wrap_execute(handler_class._execute) return handler_class

It's pretty simple, isn't it? There is one subtlety to keep track of, and it has to do with kwargs (the keyword argument dictionary). If you'll notice, require_basic_auth uses it without the stars. This lets us pass a reference to the dictionary object, and not send the key/value pairs in the dictionary to require_basic_auth as arguments. Then, when we modify the dictionary, the username and password get passed into handler_execute() as arguments. Naturally, we have to change our RequestHandler to take them:

@require_basic_auth class MainHandler(tornado.web.RequestHandler): def get(self, basicauth_user, basicauth_pass): self.write('Hi there, {0}! Your password is {1}.' \ .format(basicauth_user, basicauth_pass)) def post(self, **kwargs): basicauth_user = kwargs['basicauth_user'] basicauth_pass = kwargs['basicauth_pass'] self.write('Hi there, {0}! Your password is {1}.' \ .format(basicauth_user, basicauth_pass))

Either method works---you can take them as normal arguments, or as keyword arguments again. P.S.: If we tighten up the decorator, we wind up with this: