#effective bash

I wrote the following script the other day because A) text is always superior to a screenshot, B) C-r is a thing, and C) I can't seem to find where Amazon actually publicly documents their AZs.

I wanted to share because I think it shows a couple of techniques which I don't see very often in scripts as well as yet again proving that it's good to know the tools you use.

#!/usr/bin/env bash { cat availability-zones || { for region in $( cat regions || { aws ec2 describe-regions --output=text --query \ 'Regions[].RegionName' | tee regions } ) do aws --region="$region" ec2 describe-availability-zones | jq '.AvailabilityZones[]|{ZoneName,RegionName}' | tee -a availability-zones done } } | jq -Ss 'reduce .[] as $item ( {}; .[$item.RegionName].azs += [$item.ZoneName] )'

cat availability-zones || … | tee -a availability-zones

This is the first bit that I think is both clever and useful. I use it twice in this script. Once here and the second one with the regions output. The idea is to use a file as a cache with cat's exit code being used to figure out whether the cache needs to be populated or not. To re-query the regions you just rm availability-zones regions and rerun the command. If you're happy with the regions you found but you want to re-query the zones you can just rm availability-zones and be good to go from there.

The key is to use cat on the one hand and tee on the other. tee is so fantastically useful and everyone should be aware of it. I'm an especially huge fan lately of the tee <<<"charnock" stephen idiom to both populate a file with a string while also printing it my screen. Very useful in my CD pipelines.

The next bit that I think is cool is the use of --query on my aws commands.

… aws ec2 describe-regions --output=text --query 'Regions[].RegionName' | …

Unfortunately you then see me immediately turn around and fall back to jq like a coward because JMESPath just does not compute for me. And really more so because jq is general whereas JMESPath only helps me in awscli.

Nevertheless I'm sure it's more efficient (for some definition of the word) to encode my JSON response processing in --query if I can rather than starting up the separate process (although I don't think I can measure jq's startup time.).

I also really like the use of jq's -S and -s flags here.

… | jq -Ss 'reduce .[] as $item ( {}; .[$item.RegionName].azs += [$item.ZoneName] )' …

If you're unfamiliar with them -S means sort keys and -s means slurp all input into an array before processing it. -s is getting me the ability to suck all the responses together into a single array so I can reduce it and -S is making it so that the humans I'm pasting this text to can easily find what they're looking for.

Just for kicks here's a fully parallelized version of the same code:

#!/usr/bin/env bash { cat availability-zones || { parallel aws --region='{}' ec2 describe-availability-zones ::: $( cat regions || { aws ec2 describe-regions --output=text \ --query 'Regions[].RegionName' | tee regions } ) | jq '.AvailabilityZones[]|{ZoneName,RegionName}' | tee -a availability-zones } } | jq -Ss 'reduce .[] as $item ( {}; .[$item.RegionName].azs += [$item.ZoneName] )'

Just because, you know, 😍 parallel 😍.

If you're anything like me you may have thought of the source Bash builtin as something that is capable of running code in the active Bash shell with full access to the environment from a given file.

What I didn't know, though, was that so long as the filename doesn't contain a slash, it's actually searched for on PATH. The other neat thing is that you can pass arguments to the script being run.

What this lets you do is place a script that needs to run in the context of a running shell somewhere on PATH even if it needs arguments. I use this for getting role sessions for the aws cli like:

source get_role_session read_only 123456

. (a period)

. filename [arguments]

I absolutely love hub. Almost as much as I love magit! It's an amazing addition to your git tooling if your primary git host is GitHub. hub pull-request alone is worth the entire tool.

One of the features I've recently been exploring is the api sub-command. Essentially it allows you to use the token you've generated for the tool to interact directly with the api. I often find orgs that I want to archive onto my filesystem for later browsing. Here's a command tha does that handily.

hub api --paginate orgs/"$organization"/repos | jq -r '.[].name' | parallel hub clone ~/git/"$organization"'/{}'

Another thing I often want to do is to sync the state of all my repositories up with GitHub. Here's a script that does that nicely.

find ~/git -type d -name '.git' -print0 | parallel -0 'cd {}/.. && pwd && hub sync'

One thing to remember is that when you're executing with parallel you likely need to refer to hub rather than git, even if you've aliased git to hub just because of the nature of the shell parallel is launching.

$ PS1='\n\$ ' PS2='> ' bash --norc

I often want to set up my shell for a quick demo to someone, whether that's in IRC or a gist or whatever. It took me an embarassingly long amount of time to figure out how to invoke bash in this cleanroom manner.

That could easily be made into a shell function if you wanted.

demo_shell() { PS1=$'\n\$ ' PS2='> ' bash --norc }

I've struggled for years to come up with mnemonic for # and % in paramater expansion. % is easy because you can think of it as modulo or 'without the remainder' so it masks at the end. I alwas just sort of thought of # as 'the other one' but that wasn't very satisfying.

Now I read # as 'mask' and % as 'modulo' and I think it finally makes sense. Mask is for the beginning and modulo is for the end. ## is longest mask and %% is longest modulo.

$ x=charnock $ cat <<EOF > masked x=${x#char} > longest masked x=#{##*o} > modulo x=${x%ock} > longest modulo x=${x%%n*} > EOF masked x=nock longest masked x=#{##*o} modulo x=charn longest modulo x=char