My Universe

Let’s talk about checksums. Checksums are important and helpful. However for some reason they tend to be difficult for normal, or lazy, person to use.  I am no checksum expert, but hopefully I can help make them a little bit easier for normal people to understand and use. 

Checksums are in short a way of displaying an entire file as a single generally unique numerical fingerprint. My own definition is a bit vague, but I don’t want to get bogged down in the details of a hashing algorithm since that is not the point of my discussion and far from what the end user needs to worry about. Let me just attempt to use an analogy to sum up quickly.  A checksum works kinda like an ISBN does for books. It is generally substantially smaller than the file it originates from, often a set length like 512 bits. It is unique to that specific file’s contents at that moment, if something in the file is changed, then the checksum is different. However if a file is duplicated or the file name is changed, then it will still have the same checksum as the original. If you want to know more about the specifics of how checksum works, then go read about it here.

Okay, now that that is out of the way, lets talk about why you should care about a checksum. Files on a computer are almost always in danger of corruption, loss, or modification. And, I am not talking about someone accidentally “sudo rm -rf”ing your hard drive. Whenever you copy a file from one place to another, whether it be through email, a thumb-drive, or a file sharing website, your file is passed along bit by bit across a series of mediums that are far from reliable. Often times a file may look fine, but then turn out to be corrupted due to a simple transmission error. This is fine for small files, where you can always just re-copy that mp3 to your thumb drive again. However, what if you are backing up hundreds of Gigabytes of footage from a day on set? or If you are downloading a large .iso file to burn to a CD? You would hate to find out 30 minutes into an installation that your .iso is bad or that all your backups from the shoot were corrupted. A checksum allows you a way to quickly verify that a file matches the file on the other device. This way you could quickly tell if a file is correct or not, even if you can’t actually verify by simply opening the file.

There is also a security element to a checksum as well. We tend to download a lot of files over the internet. Unfortunately, we often don’t/can’t get them from an official mirror from the developer. But by checking a files checksum with the checksum the developer has posted on their website, then you can be sure that you are getting the proper file and not one modified by a malicious third party.

Excited about checksums yet? HELLO!! ANYONE STILL LISTENING? no, well I will just pretend you are. I feel better that way.

^That was comedy. I have been told that makes you more popular. So moving on. Now that we all know generally what and why a checksum is, lets talk about using one. Well, in theory using a checksum is simple. You just need two checksums. One to be generated before and one afterwards. Then you compare the two. If they match your file hasn’t been altered.

Most modern operating systems contain methods for calculating the popular types or checksums such as md5 or SHA1.

A word of note, the following commands and discussion will rely heavily on command line utilities, as that is what I work a lot with and they are often already installed on your computer. Most operating systems unfortunately don’t include graphical methods for checking checksums. There are plenty of third party programs for calculating and comparing checksums but I will not go into those here. I will go ahead and recommend TeraCopy for copying files on Windows. It can perform basic CRC checks and a host of other features.

Two of the more popular checksum types are md5 and SHA variants. md5 is based on 128 bit and is no longer considered reliable for security purposes, but is still generally adequate for normal file copy operations. For security purposes, you are better of with a stronger variant of the SHA algorithms, such as SHA256 or SHA512. We will use SHA512 in the examples, but the methods are pretty much the same, just change the algorithm to the one of your choice. 

To generate a checksum of a file on a *nix based machine, such as OS X or Linux, open up your terminal and call the sha512sum utility on a file.

$ sha512sum myfile.txt

The command line will return the checksum, in this case as a 512 bit number (Since we used the sha512sum algorithm), and the file name on which you ran the algorithm.

To write that number to a file just redirect the command’s output to a text file like so:

$ sha512sum myfile.txt > myfilesum.txt

Now you will have a basic text file that contains the checksum followed by two spaces and then the file name you got the shecksum from.

Likewise, it is fairly straightforward to generate a checksum on windows as well. You didn't think I would leave my Windows brothern out did you? Anyways... Go ahead and open up your PowerShell on Windows. Then use the "Get-FileHash" commandlet (Available on PowerShell4 and newer) like so:

Get-FileHash -Algorithm SHA512 -Path myfile.txt

Well, hold on. By default PowerShell seems to format the output into a tabular format and truncate the hash number since it is so long. That doesn't get us much. We have a couple options to manipulate the output for us to use in PowerShell though. Which you use is your preference depending on what specifically you are wanting to do. In this case I will just pipe the output to the Formal-List commandlet. This will show us the whole value and will even word wrap by default. The code looks like this:

Get-FileHash -Algorithm SHA512 -Path myfile.txt | Format-List Hash,Path

This gives us a nicely formatted list with the Hash first and the path to the file we just hashed second

Now if we would like to write this output to a file we can just use the "Out-File" commandlet tacked on the end like so:

Get-FileHash -Algorithm SHA512 -Path myfile.txt | Format-List Hash,Path | Out-File myfilehash.txt -width 140

The "width" parameter specifies the screen size and thus where PowerShell will insert the line return. By setting the width to something larger around 140-200 your SHA512 hash shouldn't be broken into two lines.

Okay, now we know how to generate a SHA512 hash on both Windows and *nix based systems. But what do we do with them now? How are they useful? Well, If we are sending the file to someone, we can now just also send them the checksum. Either through the file we made in the previous commands or simply sending them the number through any other means. Then they can calculate the hash themselves and compare it to the one you sent them.

If we have downloaded the file and want to verify it, we will need a hash calculated by the original person or before the file was moved. If you are downloading the file most developers will post a hash of some sort on the downloads page for you to use. All you need to do is compare the two checksums.

I am fairly lazy. Thus I tend to avoid the whole "let's compare two really long numbers manually" thing. So lets compare these two checksums automatically.

On *nix we are lucky! The sha512sum command I introduced you to earlier already has a function to compare the checksum of a file by using the file we created from the command earlier. All we need to do is have the "myfilehash.txt" file in the same directory as the "myfile.txt" file that we want to check. Then we run the command with the "-c" flag set and point the command to the hash file like so:

$ sha512sum -c myfilehash.txt

This time the sha512sum command will look in the myfilehash.txt file, and use the checksums and file names listed to go find the file and compare the calculated checksums. It will output the file name and its status as either ok or failed. This can be useful to check a whole bunch of files at once as it will check every file listed in the hash file and give a status for each file.

For my Windows friends, we are a bit out of luck. The Get-FileHash commandlet I mentioned earlier doesn't have a check function built in. At this point you are solidly in the scripting/extra module loading domain of PowerShell. I wouldn't blame you if you bailed out now and just downloaded a third party "checksum checker" app that has proper buttons and what not. I hear there are plenty of good ones out there. However, at this point I at least am committed to doing this easily in PowerShell. Luckily though it isn't hard or super complicated.

So I put together a rather simple script that takes as arguments, an algorithm, file to hash, and the checksum to compare it too. You can get the script from my github. The basic gist of the script is that it loads the input arguments as the appropriate variables and then uses a basic if/else statement to compare them. I won't go into explaining PowerShell scripting and how it works here. Instead you can just look at my script for yourself to see how it works.

Note my script is not signed as of yet. Maybe at some point latter on I will get around to signing it. But for now you will need to set your execution policy to unrestricted or just copy and paste the contents of my script into your own script (Which is what I would recommend for security reasons).

You will notice I have also written a bash version of this scripts as well. This script isn't as useful since bash's sha512sum already has a checker built in. However it does come in handy if you don't have a properly formatted file for the checker to use since you can just copy and paste the checksum into the command line.

I do want to acknowledge that these are simple scripts, and there is a whole lot more that can be done with checksums. Plenty of people have much better and more complete scripts than I gave you here. I will probably keep improving the scripts since there are some other things I want to do with them in the future. But for now, they work and are fairly straightforward to explain. Which was my goal. I wanted to make them practical and simple for normal people to start using checksums. You shouldn't use them for every file, as that would be rather time consuming, but for large files, important files, or ones downloaded from an untrusted source, you should definitely be checking the checksum.

So I am done now. finally. Hopefully you found that helpful, or at least interesting to read. LinuxDanish out!

Oh, one more thing! If you want to try it out yourself, here are the checksums for my two scripts:

PowerShell Script SHA512 checksum: 224EEBE98B845208AE2D709D3EC1FA17AE1473ED365DC13D280B300A17D235584CE973732607CAE813F203847E220D02E09C190B73A7DDEEEB637B2AAA2FA185