I have been photographing earwigs recently, and this reminded me of another group of hexapods, the members of which often have big, pincer-like cerci, the diplurans. “I’ll write a post about them”, I thought, “now, let’s see what kind of pictures I have.” I started looking and located a bunch of shots of diplurans that I took in 2006 in Ghana, but when I tried to open the files I received a Photoshop error telling me that the files were in an unrecognizable format. This usually means that the file is corrupt. Other files opened, but were partially garbled.
“No problem, I’ll just get the backup files.” Same error. Second backup, same error, third backup, same error. I began to worry. I am pretty good about keeping all my files backed-up, on multiple, physically separated drives, but the system has clearly failed me.
My normal file-saving procedure while in the field is as follows:
1. Copy files from CF/SD cards to the hard drive on my laptop
2. Catalog image files in iViewPro, rename them, discard crappy shots
3. Add keywords and captions to the files
4. Copy the annotated files to two separate, portable hard drives
5. Back home, copy the files from the laptop to a separate, stationary hard drive (HD1)
6. Copy the content of this drive to two additional, separate drives (HD2 & HD3), occasionally I add a fourth drive (HD4) to the lineup.
After this is done I feel that the files are safe, and I reuse the portable hard drives for another project. Until a few years ago I would have also burned DVDs with an extra copy of the files, but as the files got bigger, and more numerous, I abandoned the practice, opting instead for adding another hard drive to the backup.
What must have happened in the case of my corrupt dipluran shots was either a copying error during the process of transferring files from the laptop to HD1 drive, or file corruption on the HD1 following the transfer. Regardless, the damaged files were then duplicated to drives HD2-HD4. So, it seems, all is lost.
Luckily, not. Back in 2006 I still burned DVDs with file backups, using the original files from the laptop as the source. I pulled them out, and sure enough, uncorrupted, original files were there. Alas, going back to using DVDs as a backup is simply not an option. During my recent trip I shot 352GB of images, which translates to 42 DVDs (DVD+R, 8.5GB each), or about 14 hours of burning and swapping DVDs (assuming 20 minutes per disk.) What are my options then?
For one, I will be copying my files from the camera’s card simultaneously to at least two separate drives. You can do it directly in the Finder (or Explorer), or use a dedicated program, such as Photo Mechanic (this program also allows for renaming files while copying them.) Second, I will no longer reuse the portable drives that hold the files copied from the cards in the field. These drives are becoming so cheap that they are now a very sensible alternative to large, external, independently powered hard drives (I may even consider keeping the CF/SD cards with the original shots and not reusing them; some photographers already do it.) And third, I will add storage space to my Dropbox account. There is no way I can keep all my photos there, but at least I can safely store the most important shots. The experience of losing some of my original files was a reminder that no single backup method is 100% safe, but hopefully a combination of several methods will reduce the chances of a disaster similar to what I saw today.
As for the diplurans, I think I will need to write another post about them later on. These are very cool animals, close relatives of insects, and they deserve a proper writeup. They have an interesting reproductive behavior and a sophisticated maternal care, and give us a glimpse of what the early chapters of the insect evolution might have looked like.
The Smaller Majority 
First, thanks much for sharing your photos and expertise. This has become one of my favorite bio blogs.
I do computer support, and the worst part is having to tell someone they’ve lost years worth of work due to failed or no backups. So I’ve become somewhat paranoid about my own stuff, and try to instill paranoia into everyone else, too. I have three types of backup: Time Machine for the working disk (system/apps/current data); Chronosync to two drives set up with encrypted disk images, which rotate between home and office; Crashplan, which backs up to servers a couple of thousand miles away. TM counts as more of a way to revert to old versions than a real backup, because a thief would probably steal the TM disk as well as the computer (Fire ditto.) Chronosync is excellent and highly configurable, but does have a learning curve. Crashplan is great and inexpensive. Their main flaw is that they don’t duplicate backups to different data centers, but last I checked (year or so ago) the other online backup services didn’t either.
For photos, I use Ingestamatic, which is kind of a poor man’s photomechanic. It can add metadata as it imports (as side files), and it can make up to two backup sets to different disks as well as the final copy for the image program to use. I never delete anything from the backups, or touch them in any way if I can help it. That lets me be more ruthless with the working set than I might otherwise be, and it also protects the backups from possibly buggy software (or my careless use of non-buggy software.)
Dropbox is great for sharing files, but it isn’t really set up well for backups. For one, they don’t copy all of the mac file metadata, which can be important for some things. There are also quirks–last week I absentmindedly unpacked a zipped application file that was on Dropbox, and it had a fit–what should have been about 200MB grew and grew to 1.5 gig, filling up my quota and messing up some automatic syncs. But the real problem, is that Dropbox makes it easy to absentmindedly twiddle with what you think is your backup… they are also expensive compared to online backup services.
Redundancy is good. Not only multiple copies of all files, but copies made in different ways. All software has bugs, even backup software, so use more than one backup app. Tidbits.com has a great story about restoring their server from a catastrophe, in which two different backups failed, but in different ways, so they were able to get almost everything back. (Tidbits are also the originator of every Friday 13th being International Verify Your Backups Day.)
I don’t get to go into the field, so I don’t have direct experience. But I’d be inclined to carry at least two different hard drive devices, such as a Hyperdrive and a laptop, plus a small backup drive for the laptop as a Time Machine device (then your System is recoverable, too), and not erase the cards until back home and everything’s merged with the normal backup/archiving system. I might also shoot raw+jpeg and copy or move the jpgs to a separate folder for Crashplan to deal with. Crashplan isn’t likely to be able to backup gigabytes of raws on the road, but might have some hope of getting through jpgs if there’s a halfway decent connection from time to time. If I didn’t want to carry a laptop, I’d find another type of photo backup device similar to Hyperdrive, but from another company. Don’t forget to keep the devices and cards distributed as well as possible so the loss of one coat/pack won’t lose everything. Smaller rather than larger cards is probably also good, so there are fewer eggs in any one basket.
The Tao of Backup is an excellent brief backup tutorial: http://www.taobackup.com/. It was an ad for a long defunct backup product, but it’s still the best overview I’ve seen of -all- the things you need to protect against.
Thanks again for the great photos, even the corrupted ones!
Thanks for the info, very useful. I will need to beef up my backup strategy, that last episode was quite unsettling (on the other hand, even the best backup wouldn’t have helped if I didn’t have the original, uncorrupted source files.)
Piotr, I have had the same rare corruption problem and found I didn’t have a clean original file to recover from. Luckily it was just a file here and there. But they were also good shots. Why a bad shot didn’t get corrupted can only be answered by the photo gods. However, have you had issues where the card was corrupted during time of capture? I lost an entire shoot (albeit a short one) to a corrupted card. PhotoRescue and a couple other rescue programs failed. I tossed the card.
Photorec is a utility that could recover your corrupted photos.
Hi Piotr,
When I was an undergraduate a long time ago I experienced another form of data corruption related to diplurans – the zoology course I took was using a specimen of Heterojapyx as an example of the order Dermaptera!!!
I wonder if they ever changed that. I told the lecturer at the time and he didn’t really seem to care….