|
Building a
Comprehensive Photo Storage and Backup System
AUG 10, 2016
SCOTT DAVENPORT
www.petapixel.com
Storage
and backup. It’s a mundane reality of photography. I’d much rather spend time
with my camera or be teaching others about photography. But, in the digital
age, you have to have a storage and backup strategy for your photos. Storage
and backup may be mundane, but they are unquestionably critical.
Two
or three years back, I wrote about outgrowing yet another hard
disk. Hard disk updates are a bitter pill. It’s not just updating my
main storage drive, but also my backups. The ecosystem has to grow, not just a
part of it. As the adage goes, “If it’s not stored in three places, it doesn’t
exist.”
Earlier
this year I moved my photo library to a new
iMac 5K from my aging MacPro. It was a much needed boost in
performance. I also looked forward to putting those snappy Thunderbolt ports to
good use. My storage capacity needs were also growing. RAW files from my newer
Sony camera are larger than my older Nikon system. I’m creating more video
content than ever before, too. Photography training courses, YouTube tutorials,
personal projects. If you want to chew up storage fast, make videos.
In
2016, it was time to get ahead of the curve.
How Much Storage Do I Need?
There’s
two things to know when building your storage system. What are your storage
needs and what is your budget? Let’s talk needs first.
I
reviewed my photo library and measured its growth over the last 12 months. I’m
out shooting every week, usually twice. Yet, I’m not a prolific shooter. My
landscape photography is much more targeted. I’ll leave a location with 100
photos on average. Only a subset of those go through processing with ON1 Photo
or Photoshop. That’s important to know—once a photo round trips from Lightroom
as a PSD it can easily reach 1GB in size.
Another
factor is video content—I’m making a lot more video in 2016. I produce at least
2 photography videos a week on my YouTube channel. I also produce longer form
photography video courses. In the first quarter of 2016, I released two courses
and had a third in development. I tallied up the numbers for the first few
months of 2016 and ran the projections. I estimate growth between 1.5TB and 2TB
a year. Steady growth, but nothing horrifying. An 8TB system will meet my needs
for several years. Larger capacity would of course be better.
Then
there’s budget. Remember, you want three copies of your photos. Drives and
array systems will fail. Your budget needs to cover not just the disks to hold
your masters. It also needs to cover the costs for backup drives, both offsite
and onsite. I already own some older, slower USB disks. Totaled up, they have
more than 8TB of storage. That satisfies my needs for several years, so I can
repurpose those as offsite storage disks. Offsite storage does not need to be
fast. It needs to exist and be offsite should anything happen to the entire
studio
I
need to buy two redundant 8TB systems to last me for several years. Photos are
important to me, professionally and personally, but I’m not independently
wealthy. I decided on a budget of $2000 US. Amortized, that’s a few hundred
dollars a year for my photo business—well worth the expense.
What Storage Should I Buy?
This
is an interesting question with a few dimensions. Newer computers have internal
SATA drives and USB3 or Thunderbolt connections. Thunderbolt v2 is the fastest
of the lot. eSATA has a slight edge over USB3.0. USB3.1 leapfrogs eSATA, but is
still behind Thunderbolt in raw transfer speeds. The speeds of USB3.x and
Thunderbolt are quite good. External drives are easy to add, so I chose to go
that route.
Next,
decide if you want a RAID system or not. There’s lots to choose from—Drobo,
G-Tech, Synology, etc. I went with a RAID system to maximize speed (more on
that in a moment). I chose the G-Tech G-RAID Studio system. How did I arrive at
choosing G-Tech? I took a serious look at both the Drobo and G-Tech products. I
wanted a Thunderbolt connection which helped me home in on a few products. For
the capacity I needed, the G-RAID Studio was a better fit than the Drobo 5D for
me and at a better price point.
Also,
I looked at customer service reports from users of both Drobo and G-Tech. No
matter the vendor, you will find both positive and negative reviews. Some
stories are quite… passionate, shall we say. Nobody can please everyone. What I
look for is how a company responds to and cares for their customers when
equipment breaks. I accept that drives will fail. What I need to know is my
vendor will be there to take care of me and get me back up and running at full
capacity quickly. I liked what I read about G-Tech, and some of my photographer
heroes I respect use their products, too. That means something. And, to date, I
haven’t had to call G-Tech’s support line. Touch wood.
I
mentioned a RAID system for speed—let’s get back to that. When setup as RAID0,
data in the RAID array gets striped across several drives. More drives fetching
data simultaneously means faster response and snappier performance. The speed
comes at the cost of data protection. RAID0 offers no redundancy. If your array
has 3 or more drives, you can use a RAID5 setup and gain protection from a
single drive failure. But, the array itself is still a single point of failure.
Some
consider the proprietary RAID algorithms in such arrays as a disadvantage. A
potential failure mode is the array chassis itself breaks, but the drives are
fine. Without the proprietary controller, you can’t read the data on the drives.
The actual data is “clouded” by the RAID algorithm. RAID0 splits data across
drives. RAID5 intercedes parity bits among the data. No controller, no data.
The
solution? You buy two RAID arrays. RAID0 offers no data protection in a single
array. RAID5 offers protection against a single drive failure, but not an array
failure. You need at least two of them for redundancy. Offsite backup becomes
your third tier of data security.
How Do I Setup My Storage
I’ll
share my setup with you. It works for me and I think it’s a good balance of
high performance and low complexity.
My
system has several disks:
·
Macintosh HD: A 3TB Apple Fusion drive internal to my iMac. “Fusion” is
Apple-speak for a hybrid drive. A part of the drive is solid state and the rest
is spinning rust. This is where my OS and applications live, including my
Lightroom catalog. I want the speed advantage of the SSD part of the Fusion
drive for snappy Lightroom performance. Other than my catalog, no vital data
gets stored on this drive.
·
iMac Time Capsule: A simple USB 3.0 external disk for backups of the internal
drive. Access speed is not a primary concern for this disk.
·
SDP Master: The first of my two G-Tech arrays. All my vital data is on
my “SDP Master” G-Tech array. Photos, videos, documents, the lot. It’s contents
are crucial and need to get backed up. Twice. This array connects via
Thunderbolt to my iMac, so performance is great.
·
SDP Clone 1: The second of my two G-Tech arrays. It’s my onsite backup
and also connected to my iMac via Thunderbolt.
·
·
SDP Offsite: (Not pictured) Another simple USB disk to backup the
contents of “SDP Master”. It is only in the studio for periodic backups and
remains offsite most of the time.
The
contents of the internal iMac drive as functional—it’s and OS with Apps. The
exception is my Lightroom catalog. I want the speed benefit of the Fusion drive
there. So, I have configured Lightroom to make a backup of the catalog every
day and store it on my G-Tech disks.
To
set this up, go to Lightroom > Catalog
Preferences. In the General tab, choose the frequency to backup the
catalog. The next time you exit Lightroom, you are prompted to backup your
catalog:
Choosing
where Lightroom backs up your catalog.
Use
the Choose button to set the location of your catalog
backups. Every day my Lightroom Catalog gets backed up to the G-Tech array.
That’s how I funnel my catalog into the rest of my backup flow.
How Do I Back Up My Photos
I’m
a Mac user and love the simplicity of Time Machine. Yet, Time Machine is only
suited to backing up your OS disk. And it’s great for that. It’s my opinion
that it’s worth it to get specialized software to run your backups for the rest
of your data.
My
criteria for backup software is:
·
It’s Fast: It needs to be fast. I’ll take initial slowness for the
first backup. After that, later backups need to be fast.
·
It’s Smart: Daily backups need to copy only changed files from one
disk to another. Most of my data doesn’t change from day to day. The software
needs to be intelligent, scan for what’s different, and only back up the
changes.
·
It’s Automatic: I want backups to run without me having to start them. I
also want control over how often the backups fire up.
·
It’s Not Mirroring or Cloning: A problem with drive clones and mirrors is it doesn’t
protect against human mistakes. If I accidentally delete a batch of photos, the
next mirror or clone duplicates my mistake. Poof… my onsite backup is gone too.
ChronoSync from Econ Technologies met all my
needs. To my knowledge, it’s a Mac only application – a top rated one at that.
I started using ChronoSync when I “graduated” to external disks. Creating a
backup schedule is pretty easy in ChronoSync, and they have thorough
documentation. Here’s the job “document” I use to copy “SDP Master” to “SDP
Clone 1”:
Setting
up a synchronization job in ChronoSync
The
Setup page reads left to right by default. ChronoSync has more options than
most of us will ever need. An option I turn on is Synchronize deletions. I also select Move to trash in the Destination Target setup
area. I know there will be temporary files as I work on projects and I will
delete them. When I delete something from “SDP Master”, ChronoSync moves the
file into the trash on “SDP Clone 1”. This is my safety net to prevent accidental
deletions.
Look
closely and notice “SDP Clone 1” is fuller than “SDP Master”. I’ve deleted
files and they are gone from “SDP Master.” There is still a copy of them in the
trash bin on “SDP Clone 1”. From time to time, I’ll go into the trash and clean
up files I know for sure I no longer need.
Once
your document is setup, run a Trial Sync and
then Analyze the results. There are a few OS X
temporary files and folders that you can excluded from the synchronization. The
ChronoSync documentation has all the details.
You
can run synchronization jobs by hand with the Synchronize button, or scheduled
to kick off automatically:
My
evening synchronization to save off the day’s work
I
also have a ChronoSync job setup to copy “SDP Master” to “SDP Offsite”. I don’t
schedule this one since my offsite disk isn’t usually attached to my iMac. I
fire it off by hand when I’m refreshing my offsite backup. Ideally, I do that
every 2 weeks but in practice it’s more like every month. That’s my risk
tolerance. One month’s worth of work is gone if disaster were to strike at the
studio.
Why
don’t I use a cloud-based backup? I have too much data. My upstream internet
connection is good, but not great. I’ve read the stories of others that have
tried cloud services and the problems they’ve had. Large numbers of photos.
Questions about RAW file storage. The pace new work gets created. It doesn’t
add up for me. It might for you, so think about it.
Am I Done with Storage?
You’re
never done. There’s no finish line. In a couple of years, I may upgrade camera
bodies again. Megapixel counts keep climbing and that means larger RAW files.
I’ll keep creating. I may graduate to 4K video… or 8K video when that comes to
pass.
The
G-Tech drives in a few years and grow the ecosystem again. The difference is
for my next upgrade I have a good framework to work within. It’s never the end.
Unless and until I stop creating photos and videos, which is not likely. Or,
when I reach my end date and universe reclaims my energy, which I hope is a
long time from now.
About
the author: Scott Davenport is a photo
educator and landscape photographer based in San Diego, California. You can
find his work and educational materials on his website, Instagram, Twitter,
and YouTube.
|
Another Special Announcement - Tune in to my radio interview, on Rider University's station, www.1077thebronc.com I discuss my recent book, above on "Your Career Is Calling", hosted by Wanda Ellett.
In addition to this blog, Netiquette IQ
has a website with great
assets which are being added to on a regular basis. I have authored the
premiere book on Netiquette, “Netiquette IQ - A Comprehensive Guide to
Improve,
Enhance and Add Power to Your Email". My new book, “You’re Hired! Super
Charge
Your Email Skills in 60 Minutes. . . And Get That Job!” has just been
published and will be followed by a trilogy of books on Netiquette for
young people. You can view my
profile, reviews of the book and content excerpts at:www.amazon.com/author/paulbabicki
In addition to this blog, I maintain a radio show on BlogtalkRadio and an online newsletter via paper.li.I have established Netiquette discussion groups with Linkedin and Yahoo. I am also a member of the International Business Etiquette and Protocol Group and Minding Manners among others. I regularly consult for the Gerson Lehrman Group, a worldwide network of subject matter experts and I have been contributing to the blogs Everything Email and emailmonday . My work has appeared in numerous publications and I have presented to groups such as The Breakfast Club of NJ and PSG of Mercer County, NJ.
I am the president of Tabula Rosa Systems,
a “best of breed” reseller of products for communications, email,
network management software, security products and professional
services. Also, I am the president of Netiquette IQ. We are currently developing an email IQ rating system, Netiquette IQ, which promotes the fundamentals outlined in my book.
Over the past twenty-five years, I have enjoyed a dynamic and successful career and have attained an extensive background in IT and electronic communications by selling and marketing within the information technology marketplace.Anyone who would like to review the book and have it posted on my blog or website, please contact me paul@netiquetteiq.com.
=============================================================
