#effective jq

I wrote the following script the other day because A) text is always superior to a screenshot, B) C-r is a thing, and C) I can't seem to find where Amazon actually publicly documents their AZs.

I wanted to share because I think it shows a couple of techniques which I don't see very often in scripts as well as yet again proving that it's good to know the tools you use.

#!/usr/bin/env bash { cat availability-zones || { for region in $( cat regions || { aws ec2 describe-regions --output=text --query \ 'Regions[].RegionName' | tee regions } ) do aws --region="$region" ec2 describe-availability-zones | jq '.AvailabilityZones[]|{ZoneName,RegionName}' | tee -a availability-zones done } } | jq -Ss 'reduce .[] as $item ( {}; .[$item.RegionName].azs += [$item.ZoneName] )'

cat availability-zones || … | tee -a availability-zones

This is the first bit that I think is both clever and useful. I use it twice in this script. Once here and the second one with the regions output. The idea is to use a file as a cache with cat's exit code being used to figure out whether the cache needs to be populated or not. To re-query the regions you just rm availability-zones regions and rerun the command. If you're happy with the regions you found but you want to re-query the zones you can just rm availability-zones and be good to go from there.

The key is to use cat on the one hand and tee on the other. tee is so fantastically useful and everyone should be aware of it. I'm an especially huge fan lately of the tee <<<"charnock" stephen idiom to both populate a file with a string while also printing it my screen. Very useful in my CD pipelines.

The next bit that I think is cool is the use of --query on my aws commands.

… aws ec2 describe-regions --output=text --query 'Regions[].RegionName' | …

Unfortunately you then see me immediately turn around and fall back to jq like a coward because JMESPath just does not compute for me. And really more so because jq is general whereas JMESPath only helps me in awscli.

Nevertheless I'm sure it's more efficient (for some definition of the word) to encode my JSON response processing in --query if I can rather than starting up the separate process (although I don't think I can measure jq's startup time.).

I also really like the use of jq's -S and -s flags here.

… | jq -Ss 'reduce .[] as $item ( {}; .[$item.RegionName].azs += [$item.ZoneName] )' …

If you're unfamiliar with them -S means sort keys and -s means slurp all input into an array before processing it. -s is getting me the ability to suck all the responses together into a single array so I can reduce it and -S is making it so that the humans I'm pasting this text to can easily find what they're looking for.

Just for kicks here's a fully parallelized version of the same code:

#!/usr/bin/env bash { cat availability-zones || { parallel aws --region='{}' ec2 describe-availability-zones ::: $( cat regions || { aws ec2 describe-regions --output=text \ --query 'Regions[].RegionName' | tee regions } ) | jq '.AvailabilityZones[]|{ZoneName,RegionName}' | tee -a availability-zones } } | jq -Ss 'reduce .[] as $item ( {}; .[$item.RegionName].azs += [$item.ZoneName] )'

Just because, you know, 😍 parallel 😍.