Back up data to Google Drive
Backing up and restoring data are essential but painful. We want to automate them, so they are simple and easy to perform. Ideally, the only human interaction required is pushing the button.
At Fongmun, we always aim to automate everything that can be automated, e.g. reading logs, backing up data, and restoring data. One good side effect of the principle is that automation artefacts serve as documentation. For example, we use the reading-logs script for reading logs and look at its source code to find out where the logs are.
We've chosen to backup our data to Google Drive because of the free 15GB space. Let's look at how we store our data on Google Drive.
Here's the overview of the backing-up task:
Copy the data file from Redis and dump the data from Postgres to multiple files
Zip all data files into a single ZIP file
Upload the zip file to Google Drive
Remove previous backed-up ZIP files that we don't need
The only complicated part in the above task is how to talk to Google Drive. We've learned a few things from building the script.
First of all, we upload a ZIP file using Ruby with the google-api-client gem from our server. The credential downloaded from the Google Developers Console is enough for consuming Google Drive API as the service account. So, our first lesson is:
Service accounts can consume Google Drive API without user action.
Service account can be created using Google Developers Console. You can create a project, go to "Credential", and create a new OAuth Client ID. And here comes the second lesson:
A service account is totally separated from its owning Google account. The service account has its own email address.
Since service account is separated, our third lesson is:
If the service account creates a file, its owning Google account won't be able to see it.
It isn't over yet. While trying to debug the script, we've discovered the fourth lesson, which is:
The service account's Google Drive cannot be accessed through web UI.
Be able to see the backed-up files is useful. We want to download them and inspect them. To overcome the inconvenience, we create a folder with the Google account and share it with the service account (using the service account's email). It allows us to see backed-up files through our Google account and upload backed-up files through our service account.
With the explained setting, we have a great and free backing up system!.
PS. The Ruby code for upload a file to Google Drive:
client = Google::APIClient.new( :application_name => 'Example Ruby application', :application_version => '1.0.0' ) key = Google::APIClient::KeyUtils.load_from_pkcs12('your_private_key_file.p12', 'notasecret') client.authorization = Google::APIClient::JWTAsserter.new( '[email protected]', 'https://www.googleapis.com/auth/drive', key ).authorize() client.authorization.fetch_access_token! drive = client.discovered_api('drive', 'v2') file = drive.files.insert.request_schema.new({ :title => 'test.txt', :mimeType => 'text/plain' }) client.execute({ :api_method => drive.files.insert, :body_object => file, :media => Google::APIClient::UploadIO.new('local_path_to_file', 'text/plain'), :parameters => { :uploadType => 'multipart', :alt => 'json' } }).data