Screechy Threat’s Tech Blog #1: Zippity doo dah
The goal was to get a bunch of data on S3 securely and efficiently. The zip command line tool had a convenient way of securing and compressing by including a password in the bash command line. Furthermore, AWS’s Lambda allowed us to process the data we need and not worry about a local server limitations that come with doing it locally (i.e. disconnecting from the internet, limited bandwidth, going from AWS dbs to local computer speeds). It was so simple of course there needed to be a problem.
Using a node script we were spawning a child process for the zip command. The spawn command was “spawn('zip', ['-P', password, PATH + '.zip', '-j', '-r', filePath]);”. Simple enough; zip this file path into a file at PATH.zip with a password. But the child process kept reporting onError an error of “{ [Error: spawn ENOENT] code: 'ENOENT', errno: 'ENOENT', syscall: 'spawn' }”. The files we were trying to zip were in the /tmp folder. AWS Lambda gives access to the /tmp folder with a limited amount of space of 512MB, which we were definitely under. So that wasn’t part of the issue.
An “ENOENT” error would seem to mean that it can’t find the path. However when the script was run locally it worked fine. So we knew that the script was pointing to the /tmp folder and that locally it could access it. We also tried to do other commands in the /tmp folder, like creating folders, writing files and listing files there, and we were not getting back any errors on these. Therefore this error must mean something else.
The next logical step is for this error to mean that it could not find the zip command. This seemed very suspicious because most if not all linux and windows based machines have the zip command pre-installed. But, just to be sure, we got the instance ami that AWS Lambda runs from http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html, and we started up our own instance in us-east-1 to see if it has the zip command. And, or course, we found that it did. So we opened a ticket with the Amazon support team.
After the support team passed it over to the Lambda team, we finally got the answer. This error was actually saying that it could not access to zip command. They replied, “The issue is that the zip binaries, while available on the AMI we use in the lambda execution environment, is not available to the container that the function is run in… If you were to attempt this using gzip or tar, you likely wouldn't run into issues.” This explained why zip was accessible from the instance but not while running Lambda. Gzip and tar don’t provide the same cross platform and simplicity of securing files that zip does with its password feature. Therefore we switched over to S3’s signed urls, which was just as simple and didn’t require a hassle to implement. Plus, it works on Lambda.