BOTHSIDES OF THE CL

Oh FTP. It's the protocol that just keeps giving. Today's story involves ports and firewall and routers. I was adding TLS support to paradise:

https://github.com/andrewarrow/paradise_ftp/commit/a376f5bb5784507fdd14462591eb229a5e000a03

Did you know that firewalls speak FTP? They can actually listen to the ftp commands going back and forth, read them, and do stuff based on the commands. For example, if the FTP server says I'm about to listen on port 50001, the firewall reads that and opens port 50001 just in the nick of time. Get it? Port 50001 is closed 99.999% of the time but for the 20 ms when it needs to be open right before an FTP client is going to connect to it, it opens it.

So, I didn't know this. I thought port 50001 was already open because everytime I tried to connect to it, it worked. Silly me.  So when I added TLS support to FTP guess who could no longer read the encrypted message to open that port? Yup. It went something like this:

"OPEN PORT (|||50001|)" <--- oooo, I can read that and know what to do says firewall "KEJRHHEF&F^^E^EFGHEG@" <--- oooo, I have no idea what that encrypted text says

So, hours, HOURS spent debuging why I could not connect to that port THROUGH NORMAL TCP NOT EVEN TLS because once I sent the AUTH ftp command and turned on TLS, the next commands were now encrypted.

Okay but I also mentioned routers. I have a "ubee" wifi cable modem at home. It has a box checked for "FTP" that means you know, don't allow passive connections to work. 

So this made this even more complicated I was doing this work 50% at home thru ubee and 50% remotely on a laptop with an LTE iphone connection that does NOT have that ubee ftp box thing. So even when I removed TLS support completey from the app I still saw that exact same behavior of not being able to connect to port 50001 for a totally different reason, but only sometimes.

Years ago when they wrote the ftp standard they made a small mistake. Before uploading a file they really should have required the client to tell the server the total number of bytes of the file first. But they didn’t. In ftp you can send 10 bytes and that MIGHT be the complete file or 10 minutes later the client might append 10 more bytes to make a 20 byte file. And it doesn’t even have to be just 10 minute later. It could be 10 years later. There is a STOR (store) command and a APPE (append) command. There’s no statute of limitations on calling APPE. Programs like pure-ftpd and paradise_ftp do the best they can in this crazy world of never ending files.

Say you have a major FTP site with 1000′s of users uploading files of all different sizes and once a file is uploaded it needs to be copied to S3. When do you send it to S3? After the last APPE command right?

In order to avoid lots of “this file is done, copying to S3, oops, just kidding, 1 more append call, now it’s done copying to s3, 50% done with copy, oh, oops, just kidding, couple more bytes just came in for that file...” I made a thread run every 5 minutes checking for potentially done files. After the “last append” the program places that file’s full path in a hash with it’s total size. The thread (safely) removes it from that hash and enqueues it for copying to S3. If more bytes come in later, c'est la vie, 5 minutes later we do it again. But at least that 5 minutes solves most cases.

Then we noticed another problem: as the bytes are coming in from the client we carefully coded an upper limit of 20 MB. i.e. I never wanted to accidently read in a full 2 GB file into memory. I read 20 MB chucks and write those chucks to disk as they come in. But what if a client is “chatty” and sends just 4k over and over. 4k is well under the 20 MB upper limit but 4k over and over and over is a little too chatty. I needed my program to buffer each 4k conversation and wait until it has a full 20 MB or so before writing to disk.

So I made another thread. Every 60 seconds it flushes its buffers and 100 4k chatty calls get collapsed into one 400k post. Do you see the race condition I just created? What if thread 1 tells the server the file is done, but thread 2 has yet to drain the buffer?

(I didn’t noticed this race condition, another programmer pointed it out to me. I was ready to push this to production. oops.)

But I feel vindicated with my solution:

q['/full/path/to/large_tree.jpg'] = [[1,2,3,4,5],[6,7,8,9,10], total, [11,12], total]

I now have just 1 thread reading (safety, remember safe theads people, use mutexs) from the hash but inside the hash are two different flavors: bytes and totals. The above example sends:

1,2,3,4,5,6,7,8,9,10 to the server with the command write these bytes to disk.

Then it thinks the file is done at 10 bytes and starts the copy to S3.

Then bytes 11,12 get writen to disk. Then another copy to S3 for this 12 byte file.

Before with my 2 threads it might have tried to copy a 12 byte file to S3 but not all 12 bytes were actually there yet.