aoakley.com
It's easy to build up a huge archive of family photographs and holiday snaps. If your storage failed, you'd be gutted, but with even basic smartphone cameras having 8 or more megapixels, the storage space can be huge, and thus expensive and difficult to back up. My family photo archive, just for the last 10 years, takes up 25 gigs. Fine for a removable hard drive or network attached storage, but difficult to manage over the internet.
This is a backup solution for "what if all my devices were destroyed in a house fire?"
I want my backups on another continent, somewhere very far away from my house. But I don't want to pay much, ideally nothing, and I'm prepared to compromise; this isn't my only backup. And I want a solution that can be automated on my Linux workstation.
My solution is to reduce the resolution of my photos when I make offshore backups, and then store them in zip files on Google Drive.
Here's where the compromise kicks in. I resample the photos down to one megapixel, that's no more than 1200 pixels high or wide. These backups are still alright for social media and standard 6x4 inch (10x15cm) photo prints, but not for enlargements nor publications. This is a "backup of last resort" - this isn't my only backup, I've got copies of them on a PC, a NAS box and a removable hard drive already. This solution is suitable for family snaps that have sentimental value, but not suitable for a design company where photos might have a legal or commercial value.
A typical 8-megapixel photo (3200x2400 pixels) at 90% quality is around 1 megabyte in storage size. A 1-megapixel photo (1200x900 pixels), at 75% quality, is around 75-150 kilobytes in size; about nine times smaller. On the web or glancing at a 6x4 inkjet print, you wouldn't notice the difference.
The next question is, how do we do this in bulk for an existing large archive, and how do we automate it?
This solution requires novice familiarity with the Linux command line. If you don't know how, why or when to use sudo then you may need more help than I can provide here. The Ubuntu forums and Fedora forums are good places to get up-to-speed.
To misquote Mrs Beaton, "first create your Google Drive" - go to http://drive.google.com and create/claim your account. At the time of writing (2014-09), you'll get 15GB of free storage and another 100GB is only two US dollars per month. I managed to shrink down 25GB of high-res 8-megapixel family photos to just 3GB of medium-resolution 1-megapixel photos, so I don't need to pay for more storage (yet). GDrive's paid plans are cheaper than Amazon's basic cloud storage, and cheaper than Glacier if you factor in retrieval costs (and the fact that Glacier is a real pain to use).
I have written one script and adapted another:
As with anything that's easy in the long term, preparation is key. In this example, my high resolution photos are archived in folders under my Pictures/Archive folder. I name my folders by year, month and subject, but so long as you have at least one folder below ~/Pictures/Archive , and no deeper folders, you'll be fine. For example, here's my directory structure:
~/Pictures/Archive/ ~/Pictures/Archive/2005-06 Wedding/ ~/Pictures/Archive/2006-08 Honeymoon New York/ ~/Pictures/Archive/2006-05 Annabel Birth/
Note that if there were further subfolders below, for example, 2005-06 Wedding, such as 2005-06 Wedding/Official or 2005-06 Wedding/Bridesmaid Camera then those subfolders would NOT get backed-up. To make backup automation easier, I need a flat subfolder structure, only one deep under the archive folder. Keeping it simple, makes it easy. Make more folders at the top level (e.g. 2005-06 Wedding Official and 2005-06 Wedding Bridesmaid Camera) if you need to categorise things more finely.
I then chain the snapshrink and gdrive-upload scripts together with zip. I use the find command to list the subdirectories, and then pipe those to xargs to fire off the snapshrink script against each subdirectory. Next, since you can't really compress .jpg files any more than they already are, I use zip -0 to simply store the photos inside one zip file per subfolder, with no further compression.
find ~/Pictures/Archive/ -maxdepth 1 -type d -printf "%f\n" \
| xargs -i snapshrink -a -v "{}" ~/Pictures/Med-Res/
find ~/Pictures/Med-Res/ -maxdepth 1 -type d -printf "%f\n" \
| xargs -i zip -0rv "{}.zip" "{}"
cd ~/Pictures/Med-Res ; gdrive-upload *.zip
This creates:
~/Pictures/Med-Res/ ~/Pictures/Med-Res/2005-06 Wedding/ ~/Pictures/Med-Res/2006-08 Honeymoon New York/ ~/Pictures/Med-Res/2006-05 Annabel Birth/ ~/Pictures/Med-Res/2005-07 Wedding.zip ~/Pictures/Med-Res/2006-08 Honeymoon New York.zip ~/Pictures/Med-Res/2006-05 Annabel Birth.zip
...and then it uploads the .zip files to Google Drive. It prompts for your Google username and password. You can use your GMail login, or your Google Plus login, or whatever you use to log in to Google Drive, but the point is you must have already visited http://drive.google.com and created/claimed your storage.
Problems with the upload script are usually caused by an incorrect username/password, but you may also need to turn off two-factor authentication if you've set your account to require it. Discuss the Google Drive upload script over at GitHub - Deanet .
Whenever you need to recover a backup, you can download whichever folder you need as a .zip file from Google Drive (another benefit of my solution is that you don't have to download all your backups in one go).
Weaknesses? Well, it is a compromise; the gain in ease and cost, is offset by loss in quality. Also, resampling the photos with snapshrink/imagemagick is very slow on ARM appliances such as most NAS boxes or the Raspberry Pi; you really want to do this on a Linux PC that has a proper desktop processor, or a reasonably fast laptop.
The reduced resolution folders also make backing up to optical storage, such as DVD-R, much more manageable. Optical storage is likely to outlast magnetic and electronic media such as USB sticks, memory cards and hard drives, if stored away from light. I plan to put a DVD backup in a sealed card envelope, in my elderly parents' loft.
You could also write a cron job to automatically shrink, zip and upload new photo archive folders. You'd have to work out a way for the job to recognise which folders had already been backed up (grep a hidden file, if not found then perform the backup, don't forget to append the folder name to the hidden file before finishing?).
Public Domain - Andrew Oakley - 2014-09-12
