YummyCoding

I just spent an hour or two ripping my hear out metaphorically because I could not for the life of me figure out why some data was not correctly being saved into our database.

We have a usage database that pulls the data from an Amazon Web Services ELB traffic log, collates the data, and then saves it to a database. (Why aren't we using a log analyzer like Kibana, you ask? Because we need to link up the data to related records in our main database, and because we need to do a lot of transformations on the data that are sufficiently complicated that it made more sense just to roll our own tool).

Anyway, this is what a line form the ELB log might look like (taken from the "Access Logs for Your Classic Load Balancer" page of AWS documentation):

2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.000086 0.001048 0.001337 200 200 0 57 "GET /product/1234/ HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2

I would extract the URL from each line entry, then compare it with expected regex patterns and create the correct entry in the database. Something like this:

PATH_PATTERNS = ( r'^/paper/(?P<paper_id>\d+)$': 'paper', r'^/journal/(?P<journal_code>[-\w]+)$': 'journal', )

In the example url above, it would match to the first pattern, and create an entry in the log database pointing to a paper record with the id 1234.

I was using python's [csv.DictReader](https://docs.python.org/3.5/library/csv.html) to break apart the line which then allowed me to grab the request, which looked like:

"GET /paper/1234 HTTP/1.1"

Here's the code I used to determine the requested URL:

request_method = request.split()[0] url = ' '.join(request.split()[1:]).rstrip('HTTP/1.1')

Now, looking back I can definitely see better ways to write this code anyway. Assuming the format of METHOD{space}URL{space}HTTP/1.1, this is much cleaner code:

request_method, url, _ = request.split()

But for the sake of argument, let's assume that when I wrote this, the way I parsed it was correct — that the first token should be the request method, and the remainder of the string should be the URL, and we should strip away the HTTP/1.1 at the end.

Does anyone see the problem?

No?

The cardinal rule of programming:

You can never trust customer input

In this case, although Amazon is scrupulous in how it outputs its log data, it is still just reporting the content it gets from another source — in this case, the request made by a user agent to our server that created the log entry in the first place.

Turns out even though HTTP 1.1 is 20 years old there are still some user agents out there that use HTTP 1.0!

Here is the request that was confounding my app:

"GET /paper/1234 HTTP/1.0"

After processing, the url, which should have been /paper/1234 was, instead /paper/1234 HTTP/1.0 which of course did not match any of the expected patterns. So none of the records for this user agent were being recognized or saved to the database!

A simple fix, fortunately. (Again, just in case I'm not using the simpler cleaner code.):

import re ... request_method = request.split()[0] url = ' '.join(request.split()[1:]) url = re.sub(r'HTTP/[.\d]+$', '', url)

And here, I could also write it as

request_method = request.split()[0] url = ' '.join(request.split()[1:-1])

but I wanted to explicit about exactly what I was excluding here. Writing out the re.sub makes it clearer that I'm taking out the protocol information.

So I'm currently whipping up some testing for my Django-based website. It's technically not unit testing since that entails testing only the smallest parts of applications and not testing the interactions between, e.g. databases, mail servers, etc. Since I'm testing all that, it's not unit testing.

Nonetheless I am making use of the unittest module in Python and it's working great.

Just now, however, I ran across a snag.

Namely: how to deal with delayed tasks (specifically using Celery).

In a non-delayed task situation, this is what the code would look like:

# views from .urls import something_time_consuming def some_view(request, pk, template_name='foo.html'): # start of view code something_time_consuming(input, otherinput=var) # rest of view code

By the way the testing for that would look something like this:

from .views import some_view class TestMyViews(TestCase): def setUp(): self.factory = RequestFactory() def test_post_some_view(self): request = self.factory.post('some-url') record = Record.objects.create(…data…) # code to check expected state before running view result = some_view(request, record.pk) # code to check expected state after running view

Let's imagine the function something_time_consuming is a function that takes a long time relative to a web request — say, it sends out 1000+ emails. Depending on the complexity of the emails, that might take a few seconds to complete and that's too long to wait for a web server to respond.

However, you can use celery, a task queue to avoid this. Rather than executing something_time_consuming on the spot, you can tell the system "please run something_time_consuming later" and then continue serving the web request. Then, the system at some point (generally very soon after) gets instructed to run something_time_consuming.

With celery and delayed tasks set up, the code now looks like this

# views from .tasks import do_something_time_consuming def some_view(request, pk, template_name='foo.html'): # start of view code do_something_time_consuming.delay(input, otherinput=var) # rest of view code # tasks from celery import shared_task from .utils import something_time_consuming @shared_task def do_something_time_consuming(*args, **kwargs): something_time_consuming(*args, **kwargs)

So what happens here instead is the view is called, and the do_something_time_consuming which has been wrapped by the shared_task decorator and so has some special celery functions added including .delay which means "wait a bit and then run do_something_time_consuming, sending it the arguments I received.

So when the request is called, .delay(input, otherinput=var) causes celery to add a delayed task to its queue with the function to be called do_something_time_consuming and the arguments input and otherinput. Then celery has the server run that task with those arguments.

Okay, this is all well and good.

What about testing?

So this is where it gets tricky. Because normally when you call delay, the action gets shot out to the queue and then run by a different process, so you can't even see what is happening at all let alone test it.

The solution is actually pretty simple: use mock.

Mock is a module that is built into Python 3 (available under unittest.mock) and available as the mock package for Python 2.7.

It is used pretty much exclusively for testing and I'm not sure how it works, only that it is magic. What it does is it mocks aspects of your application — replacing a function, method, or even a class with a mocked-up function, method, class, etc.

Here's how the test code above would be rewritten:

from .views import some_view class TestMyViews(TestCase): def setUp(): self.factory = RequestFactory() @mock.patch.object(do_something_time_consuming, 'delay', side_effect=do_something_time_consuming) def test_post_some_view(self): request = self.factory.post('some-url') record = Record.objects.create(…data…) # code to check expected state before running view result = some_view(request, record.pk) # code to check expected state after running view

Basically here what we're saying is when the delay function is called, instead of performing the given action of sending it all to the queue, go ahead and just run the function without any delay at all.

The advantage of doing testing this way is it takes the entire queue logic out of the picture. So we don't have to worry during the testing about whether our task queue works correctly or not (although it's not a bad idea to write what's called an "integration" test to test that further down the road).

There are times where you're messing with text and you're like

"OMG I do not care, please just do whatever you want with the text, it can have weird symbols or question marks in it for all I care!"

And then, even then, you'll see something horrible like this

UnicodeDecodeError: 'ascii' codec can't decode byte 0xcb in position 0: ordinal not in range(128)

For times when you don't care what the text actually contains, how it's encoded, or what it will look like when you're done with it, you can use the following helpful function:

def unmessup(input): for enc in ['utf-8', 'latin-1', 'unicode_escape']: try: return input.decode(enc) except UnicodeError: pass except AttributeError: # if input doesn't have the .decode attribute # it is either a Python unicode string or not a string at all return input

This function goes through a variety of common encodings in order from most to least likely. Odds are good, very good, that you're just looking at a non-unicode string that is utf-8 encoded. So we try that first. If that fails, Latin 1 (which corresponds to ISO 8859-1) is the most likely alternative. At this point even if it's wrong one of the two encodings has probably returned something.

Still, if neither of these work (possible), then we turn to the fallback encoding unicode_escape which is basically the hail mary of text encodings. It basically says, "look, whatever characters there are here, just put them — as entered — right into the resultant string, without trying to decode them or alter them in any fashion whatsoever.

The result will be horrific. For instance, the word Beijing written in Chinese: 北京, when encoded using Big5, becomes '\xa5_\xa8\xca'. When decoded using Big5, it looks like u'\u5317\u4eac', but when using the unicode_escape codec, it becomes: u'\xa5_\xa8\xca'. Notice that none of the codes have changed. The only difference is that the string is now prepended with u, which means "Sure, buddy. This is unicode. Don't worry about it." And when you print it out, it looks like this:

¥_¨Ê

That's not anywhere close to the original. So it's wrong. Which is absolutely why UnicodeErrors exist in the first place — to keep you from writing a travel book that encourges you to visit China's amazing capital city, Yen Underscore Umlaut E-Circumflex.

These days, most browsers have an autofill function, that helpfully will fill in forms with default values (e.g. your name, email, address, or other contact info.

Usually browsers get this right — they can recognize that a form is expecting a first name, or a zip code, or a phone number.

But sometimes, it gets it wrong — and that's where the autocomplete attribute can come in useful. It tells the browser exactly what field it's expecting. For example:

<input type="text" name="title" autocomplete="honorific-prefix">

In this example, it might autocomplete this field with "Mr.", "Mrs.", "Dr.", etc.

Here's another example:

<input type="text" name="badge_label" autocomplete="nickname">

This is all well and good. However, there's a serious problem, which is that, possibly to account for broader, less country-specific postal addressing rules, they opted not to include terms like locality or city, or state/province/region, as acceptable values for autocompletion:

<input type="text" name="ship_to_c" autocomplete="city">

The above example is incorrect according to WHATWG.

Instead, you are supposed to use what are called administrative levels (also called administrative divisions).

The problem with this is that these levels are actually different from country to country.

For example, in the United States, administrative level 1 is the state, while administrative level 2 is the city, like so:

Philadelphia, PA ^ ^ | |-- Administrative Level 1 | |---- Administrative Level 2

Meanwhile, in the UK, the city (or, technically, the Post Town) is Administrative Level 1.

Which means that if you're making a form to be used exclusively in the UK, it would look like this:

<input type="text" name="address1" autocomplete="address-line1"><input type="text" name="address1" autocomplete="address-line2"><input type="text" name="city" autocomplete="address-level1">

Whereas for a US address it would need to look like:

<input type="text" name="address1" autocomplete="address-line1"><input type="text" name="address1" autocomplete="address-line2"><input type="text" name="city" autocomplete="address-level2"><input type="text" name="state" autocomplete="address-level1">

For forms which are intended to be used across countries, this means that you would have to be consistently adjusting the form for each country, which also means you'd need to keep track of the postal and or legal regulations of every country in the world, just to track whether a city is considered administrative level 1, level 2, or even 3 or 4 (yes, that can happen).

Indeed, the standards themselves make it clear that it is expected of developers to dynamically flip and change the form to make it conform to all of these possible variations:

Authors are encouraged to provide forms that are presented in a way that matches the country's conventions (hiding, showing, and rearranging fields accordingly as the user changes the country).

I've been making web forms since the early, early days of the web. And the city field has always been called the city field, and the state field has always been called the state field (or, sometimes, State/Province). Although occasionally individuals from some countries have to make judicial use of the 2nd or 3rd address line in order to complete their address, and sometimes have to stretch the use of the "State/Province" field for their own purposes, I haven't run into any problems with this.

So #aws is down which means I can't really get much done right now (although fortunately our websites are responding correctly).

We've all been there — you've just written a beautiful bash script that takes in a list of files and fiddles with them beautifully.

It works perfectly…right up until the moment some nimrod introduces a file with spaces or quotes in it! Suddenly the script, at best, sputters to a halt or, at worst, starts creating a slew of fake files or replacing existing ones.

Sure, for certain commands you can use a null delimiter, using flags like --null or -print0 or -0 (would be kind of nice if this were standardized!). But not all commands do this. And even if all the commands you are using do support null one way or another, what a pain in the neck it is to go through all the different commands and fix them!

Turns out there's an easier solution — just fiddle with the filenames until they are correctly escaped!

In other words, turning this horrible and unsafe filename:

Final Document "Use This".docx

To this horrible but safe filename:

Final\ Document\ \"Use\ This\".docx

If you're doing a one liner, just pip it to this set of commands:

<command>|sed -E -e 's/ /\\ /g'|sed -E -e 's/([^\\])"/\1\\"/g'

The first sed escapes all spaces, the second one escapes all quotes UNLESS they've already been escaped (it happens). By the way, this means that if you have a filename that actualy has a backslash preceding a quote, then you're kind of screwed. Sorry!

If you're fiddling with variables or would like to put this in a centralized location in a bash script, here's a function:

function escape_filename() { echo "${1}"|sed -E -e 's/ /\\ /g'|sed -E -e 's/([^\\])"/\1\\"/g' }

Note the use of "${1}". It preserves whatever line breaks existed in the original filename listing.

Had literally no idea you could do this.

In other words, let's say you have a class like this:

class Widget(object): sockets = ['PRIMARY', 'SECONDARY', 'BACKUP'] default_socket = 'PRIMARY'

What happens if you rename the first socket to "INITIAL". You have to change the code in two places:

class Widget(object): sockets = ['INITIAL', 'SECONDARY', 'BACKUP'] default_socket = 'INITIAL'

or if the default socket is always supposed to be the first one in the list, you can actually refer to the sockets variable in the class definition:

class Widget(object): sockets = ['INITIAL', 'SECONDARY', 'BACKUP'] default_socket = sockets[0]

This whole time I've either been resiging myself to repeating code or resorting to something like:

class Widget(object): sockets = ['INITIAL', 'SECONDARY', 'BACKUP'] def __init__(self): self.default_socket = self.sockets[0]

You can also use loops or any other kind of code you'd like. For example, if you're setting up a bunch of sockets:

class Widget(object): sockets = ['PRIMARY', 'SECONDARY', 'BACKUP'] for x in range(12): sockets.append("SOCKET_{0}".format(x + 4)) default_socket = sockets[0]

This whole time I had thought the only code you put immediately under a class was either a class method or a property. And once you stop to think of it, of course you can put any code you like under a class definition! For one thing, there's nothing preventing you from doing so and secondly, all you're doing when you put code under that class (or a function for that matter) is that you're saying that the variable only exists under that class (or function) and isn't available globally.

Sometimes when your AWS instances are terminated by AWS autoscaling, you want to know why. AWS has a feature where it will put those instances in a Terminating:Wait mode so that they no longer respond to requests but are still available, so you can ssh in and diagnose the problems.

If you follow the steps in the AWS tutorial Analyzing an Instance Before Termination you're bound to run into a few problems.

Problem #1 "Invalid choice"

If you try to run the given command:

aws autoscaling put-lifecycle-hook --lifecycle-hook-name WaitForDiagnostics --auto-scaling-group-name group-name --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING --notification-target-arn arn:aws:sns:us-east-1:999999999999:topic-name --role-arn arn:aws:iam::999999999999:role/role-name

You may see the following error:

aws: error: argument operation: Invalid choice, valid choices are…

This happens when you have an older version of the AWS CLI (command line interface). Frustratingly, the latest package available via yum is out-of-date.

You have to install it, not with yum, but with pip (the Python package installer). I'm assuming you already have pip installed (if not, this step is left as an exercise to the reader).

sudo pip install --upgrade awscli

will install the latest version.

The above command should now run as expected.

Problem #2 "Unable to publish test message"

Ha ha, just kidding. Of course there are more errors. If you get this one:

A client error (ValidationError) occurred when calling the PutLifecycleHook operation: Unable to publish test message to notification target arn:aws:sns:us-east-1:999999999999:topic-name using IAM role arn:aws:iam::999999999999:role/role-name. Please check your target and role configuration and try to put lifecycle hook again.

there are (likely) two problems:

The IAM role doesn't have permissions for the needed SNS actions. Most people familiar with IAM and such probably already figured this and adjusted accordingly, for the record, here's the policy you need for that ( A simpler way to get this is just to add the managed police AutoScalingNotificationAccessRole):

{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Resource": "*", "Action": [ "sqs:SendMessage", "sqs:GetQueueUrl", "sns:Publish" ] } ] }

The IAM role doesn't have a trust relationship with autoscaling.amazonaws.com. Honestly, I wasn't even aware of trust relationships before this and I'm not even sure what they are or what they do. But without the right trust relationship, you cannot post messages based on actions taking place within autoscaling — or at least, not for the purposes of this fix. It wasn't until I looked at a sample IAM role created exclusively for Auto Scaling notifications that I figured this out. To solve this, go to the IAM Roles Page, choose your role, then click on the "Trust Relationships" tab and then click on "Edit Trust Relationship". In the section labeled "Statement" you want to add the following (make sure to put a comma between it and the previous statement):

{ "Effect": "Allow", "Principal": { "Service": "autoscaling.amazonaws.com" }, "Action": "sts:AssumeRole" }

Once I did this, the put-lifecycle-hook finally worked.

If at a later point you decide you don't want instances put into the Terminating:Wait state, you can first use the command

aws autoscaling describe-lifecycle-hooks --auto-scaling-group-name group-name

to see what you named your hook and then

aws autoscaling delete-lifecycle-hook --auto-scaling-group-name group-name --lifecycle-hook-name hook-name

An elegant solution, all credit to Tim Brigham:

Create /etc/cron.d/reboot with the contents below. Note - It's been a while, you should check syntax.

@reboot run-parts /etc/cron.reboot

Create a folder /etc/cron.reboot

From there it's a matter of dropping files in to the cron.reboot folder.

In copy-paste command line form:

So I was running some slow commands in bash and wanted to know when they were done.

The command for making a beep in bash is

echo -e "\a"

so I used that for making the beep sound. For each \a, you get one beep, so \a\a means two beeps.

Initially I just used something like:

cmd && echo -e "\a\a"

However, that only worked in the command executed successfully -- the && means "if cmd before && is successful, then run cmd after &&" — which often didn't happen. So I switched it to

cmd; echo -e "\a\a"

Then I wondered if I could combine these techniques so that it would beep twice when successful, but more times when failing. There's another operator (maybe not the correct term?) in bash ||, which behaves like && only when the command is unsuccessul, and I wrote this:

cmd || echo -e "\a\a\a"; echo -e "\a\a"

This command translates to:

if cmd is unsuccessful, then run echo -e "\a\a\a" (beep 3 times). Then, no matter what happens, beep twice

In practice, this means if the command fails, it beeps three times, then two times, for a total of five times. If the command succeeds, it beeps twice.

This is a really useful way for me to keep track of long-running tasks in the background, as well as getting heads up whether they succeeded or not.

I've written up a gist so that you don't have to append || echo -e "\a\a\a"; echo -e "\a\a" at the end of every command you run:

https://gist.github.com/JordanReiter/b4cbef6163d4f69fa0bd: SUCCESS_BEEPS=2 ERROR_BEEPS=5 prog=$1 shift args=$@

function countbeep() { head -c $1 < /dev/zero | tr '\0' "\a" }

if $prog $args; then countbeep $SUCCESS_BEEPS else countbeep $ERROR_BEEPS fi

querySelector returns the first matching element for one or more CSS selectors (a comma delimited string) while querySelectorAll returns all matching elements. It returns either a single element or an array of elements so unlike jQuery, there's no chaining.

So as of Django 1.8 the old

urlpatterns = patterns('', url(r'...', view, name='view_name'), ... )

format is deprecated in favor of

urlpatterns = [ url(r'...', view, name='view_name'), ... ]

As of Django 1.10, patterns() will be removed entirely.

Granted, support for 1.7 is technically over, and support for the LTS version 1.4 is over soon, but if you're writing a reusable app for Django I'd consider it common courtesy to write it so that it's compatible, both with versions where urlpatterns has to be a list and where it has to be created using patterns.

Fotunately, it's an easy fix:

try: from django.conf.urls import url, patterns except ImportError: from django.conf.urls import url patterns = None urlpatterns = [ # enter patterns here url(…) ] if patterns: urlpatterns = patterns('', *urlpatterns)