Page 1 of 2

Thoughts on image storage and backups

PostPosted: Sun May 24, 2009 11:20 pm
by DaveB
How many of you feel in control of your image management, storage, and backups?

I've written some notes on this, and would appreciate your feedback. Do you disagree with any of my statements in that article? Some of the issues (e.g. filenaming) might not match the way you currently do things, so please try to keep an open mind when reading it.

Thanks!

Re: Thoughts on image storage and backups

PostPosted: Mon May 25, 2009 1:01 am
by DanW
Great list of tips.

A couple of things popped to mind. As you said "RAID is not backup. It's just increasing the reliability of the storage device.". This is very true. I've had a RAID controller die on me - I suspect from a bad power supply. Took 4 hard drives with it. I managed, after a lot of effort, to recover most of the important stuff. From then on, I always made a backup separate to the RAID.

"You should always have at least 3 copies of your data." Totally agree. For important jobs I have a) a spare hard drive in the computer solely for backups, b) an external hard drive that mirrors this and c) during the job, burn a DVD every day. I've found having a new disk for every day a lifesaver when I mistakenly delete a file from the main working drive. Although this deletion might propagate through the other hard drives, the file will still be recoverable from last night's disk. I guess this fits in with your "More than one stage of backup can be good" tip. At only a few cents per disk, it's very cheap insurance.

Cheers

Re: Thoughts on image storage and backups

PostPosted: Mon May 25, 2009 9:51 am
by Reschsmooth
Good post. I know that I need to implement a better back-up system at home. We have thousands of images which are only stored on 1 HDD, another few thousand on two HDDs and only a relative handful stored on 2 HDDs and external media. Good reminder to buy some DVD-Rs or CD-Rs! Any suggestions on preferred type?

Re: Thoughts on image storage and backups

PostPosted: Mon May 25, 2009 10:19 am
by biggerry
Backups need to be operated simply and regularly.


I cannot agree more! if a backup solution is complex or time consuming, it WILL fall by the wayside.

Keeping data (backed up) in different physical locations is also an important point, but not particularly feasible for most small operations, however even running off a dozen DVD's every 6 months and dropping them off at ya mums is not a bad idea!

Is it worthwhile, if you plan on including further detail in the article, to consider some example situations or some recommended scripts/software to perform backups? maybe ones more suitable to photographers if there is such one?
(I note, at the end, that you have mentioned you will discuss tools in the future)

Also, organisation of data is particularly important imo - poorly thought out data storage structures make it a nightmare to efficiently and reliably backup.

anyway ta for the good read - I look forward to the next installment.. :up:

Re: Thoughts on image storage and backups

PostPosted: Mon May 25, 2009 11:59 am
by DaveB
Reschsmooth wrote:I know that I need to implement a better back-up system at home. We have thousands of images which are only stored on 1 HDD, another few thousand on two HDDs and only a relative handful stored on 2 HDDs and external media. Good reminder to buy some DVD-Rs or CD-Rs! Any suggestions on preferred type?

My recommendation would be to buy an external HDD to put the backup onto. I generally only use DVD-R or CD-R these days for shipping files to clients. Not for long-term storage.

See the point in the article about "No backup media will last forever". DVD-Rs will need to be checked and re-checked regularly to make sure they still work (and then make new copies from one of our other backups when you find the disk has failed). You did make at least 2 DVDs and store them in different locations didn't you? ;)
If you want to go down the archival optical disk path, check out these.

Checking a 500GB hard drive is much easier and faster than checking 100+ DVDs, and they keep getting cheaper.

biggerry wrote:Keeping data (backed up) in different physical locations is also an important point, but not particularly feasible for most small operations, however even running off a dozen DVD's every 6 months and dropping them off at ya mums is not a bad idea!

That's only one DVD's worth of data every month (I presume when you mention a dozen DVDs that half of those are duplicates?). Not a very active shooter obviously (or one using a low-res camera). Ok, maybe I'm exaggerating slightly. :roll: As you say, it is an option.

Is it worthwhile, if you plan on including further detail in the article, to consider some example situations or some recommended scripts/software to perform backups? maybe ones more suitable to photographers if there is such one?
(I note, at the end, that you have mentioned you will discuss tools in the future)

This first article was too long as it was, and I wanted to keep it generic. I am planning to include lots of examples as we go on. The design decisions we make about how to set up our filing system and backups are predicated on some of the points I raised. Better to get them out in the open first so we can refer back to them as we get to examples.

In fact if you're a Mac user you may be very interested: this is partly a build-up to the tools I've put together for OS X. The Windows users amongst you will not be so lucky (sorry!) but hopefully there'll be useful knowledge you can gain by following along. At least this intro article was platform-agnostic.

Re: Thoughts on image storage and backups

PostPosted: Mon May 25, 2009 4:38 pm
by ATJ
Dave, it looks good to me.

I have heard the DVD+R are more reliable than DVD-R. Apparently it has to do with DVD+R having "permanent" tracks. I only use DVD+R for archives.

I know you said that filenaming conventions may be different, but you mention not changing the filenames in your processing. I prefer to leave my source images as DSC_1234 and create more meaningful names for the JPEGs, such as S_mestus_Bare09.jpg (which tells me what, where and when). All the JPEGs (for a particular purpose) and up in the same directory/folder and Lightroom makes the names unique. I write the original filename to the EXIF of the JPEG so I can always track back to the original file.

By the way, here's How I backup my photographs.

Re: Thoughts on image storage and backups

PostPosted: Mon May 25, 2009 5:26 pm
by biggerry
In fact if you're a Mac user you may be very interested


fortunately not :twisted: :wink: however I will be disappointed if Rsync (http://www.samba.org/rsync/) does not get a mention, this is available on mac and is free and is australian made and ...could there be more? yes.... its has to be one of the quickest means to keep data up to date over different locations. :up: I use it on linux servers for going between boxes and between external drives.

Not a very active shooter obviously

each to their own :? the main point is offsite backup :up:


By the way, here's How I backup my photographs.


Nice read Andrew, can suggest at your second level backup to the linux box maybe trying a rsync script on the linux box scdeduled to run at set intervals rather than the xcopy? just a thought - xcopy is set to be deprecated (apparently, although it is still shipped with blister). :wink:

Re: Thoughts on image storage and backups

PostPosted: Mon May 25, 2009 6:26 pm
by ATJ
biggerry wrote:Nice read Andrew, can suggest at your second level backup to the linux box maybe trying a rsync script on the linux box scdeduled to run at set intervals rather than the xcopy? just a thought - xcopy is set to be deprecated (apparently, although it is still shipped with blister). :wink:

That would mean giving access to my Windows box from my Linux box, which I don't want to do. As it is, the backup batch file I currently run on the laptop requires me to enter my Samba password (i.e. so isn't automatically connected either). Additionally, being a laptop, it isn't always connected to the network so a scheduled copy could mean I miss copies.

Re: Thoughts on image storage and backups

PostPosted: Mon May 25, 2009 6:30 pm
by muzz
Hi Dave, I'm a Mac user and it was a timely read for me - just backed up 4 DVDs worth of photos - I'm both non-prolific and until recently a JPEG only shooter, so that represented quite a timespan for me. My normal backups include daily backups from my main computer to a photos only external HD in their original format, another to a different HD using the Backup program that comes with a MobileMe account (this is an incremental backup in proprietary form) and also another TimeMachine backup to a third HD. Periodically I backup to DVDs and make 2 copies and take one off site. I have just organised an old G4 tower at work to make regular transfers from one of the external HDDs but have yet to make the first transfer. I also use a DigiMate III to store images straight off the CF card from the camera but this is not always done immediately. But I still feel vulnerable!

One of the problems with external HDDs I think is the price and quality of enclosures as well as the HDD itself, so I'm wondering if anyone has been using "naked" drives with the hot-swappable docks such as these or these.

One problem I must look at solving is image naming - I've always left the images named out of the camera but when I upgrade my camera (as I did from a G5 to a 350D a few years back), the image/file numbers started to be duplicated. I think I'll have to rename the whole library all at once in Aperture (maybe concatenate the camera model and the filename or may be date and filename) and re-organise my library from there.

Re: Thoughts on image storage and backups

PostPosted: Mon May 25, 2009 11:29 pm
by Mr Darcy
I am a recent Mac convert, and backups are one area I need to address. ATM just using Time capsule. It will do as first line of defence but not a patch on what I did in Win$

I look forward to more details.

I am also interested in naming conventions. This os something I have NEVER nailed. Sigh.

Re: Thoughts on image storage and backups

PostPosted: Tue May 26, 2009 1:21 am
by surenj
Reschsmooth wrote: Good reminder to buy some DVD-Rs or CD-Rs! Any suggestions on preferred type?


Taiyo Yuden springs to mind. MSY stocks them for good prices.
http://www.supermediastore.com/taiyo-yuden-dvd-r-media.html

Re: Thoughts on image storage and backups

PostPosted: Tue May 26, 2009 8:44 am
by DaveB
Andrew, I expect the "chapter" on filenaming to be up this evening - I'll post here when it is.

biggerry wrote:I will be disappointed if Rsync (http://www.samba.org/rsync/) does not get a mention, this is available on mac and is free and is australian made and ...could there be more? yes.... its has to be one of the quickest means to keep data up to date over different locations. :up: I use it on linux servers for going between boxes and between external drives.

Rsync is indeed a wonderful tool. It's shipped by default as part of the Mac OS X, and my tools use it to do the grunt-work of copying files.
I notice Andrew uses Microsoft's SyncToy, which is also worth a mention for Windows users (go to http://microsoft.com/prophoto/downloads to find it). But rsync for the PC is also available.

Mr Darcy wrote:I am a recent Mac convert, and backups are one area I need to address. ATM just using Time capsule. It will do as first line of defence but not a patch on what I did in Win$

Time Machine is wonderful for taking care of the operating system and all your normal files. I've used it to restore a complete working machine after a crash. But it's sub-optimal when it comes to handling large files that change including: photos, Lightroom and Aperture databases, VMware/Parallels virtual disks, etc. It also only keeps one backup drive (although that backup drive contains many versions, your data's only on 2 drives). My own setup uses Time Machine for the OS/etc and tells TM to ignore the other bits so my tools can take care of them. I regularly put images of my TM drive onto off-site storage to take care of the redundancy issue. We'll get to all that.

Re: Thoughts on image storage and backups

PostPosted: Tue May 26, 2009 9:57 am
by ATJ
muzz wrote:One problem I must look at solving is image naming - I've always left the images named out of the camera but when I upgrade my camera (as I did from a G5 to a 350D a few years back), the image/file numbers started to be duplicated. I think I'll have to rename the whole library all at once in Aperture (maybe concatenate the camera model and the filename or may be date and filename) and re-organise my library from there.

The same thing happens when you go around the clock on a single camera which did with my D70 and have already done with my D300. However, I don't see it as a huge problem as the dates of the files are so far apart it is easy to work out if it was first time, second time, etc.

Re: Thoughts on image storage and backups

PostPosted: Tue May 26, 2009 10:07 am
by gstark
ATJ wrote:
muzz wrote:One problem I must look at solving is image naming - I've always left the images named out of the camera but when I upgrade my camera (as I did from a G5 to a 350D a few years back), the image/file numbers started to be duplicated. I think I'll have to rename the whole library all at once in Aperture (maybe concatenate the camera model and the filename or may be date and filename) and re-organise my library from there.

The same thing happens when you go around the clock on a single camera which did with my D70 and have already done with my D300. However, I don't see it as a huge problem as the dates of the files are so far apart it is easy to work out if it was first time, second time, etc.


It's actually very easy to use a date based folder naming convention. Try YYYYMMDD. It's self sorting, and easy to comprehend. As I offload files from my CF card, I just do it by folder, then rename the folder for the date.

Re: Thoughts on image storage and backups

PostPosted: Tue May 26, 2009 10:27 am
by ATJ
gstark wrote:
ATJ wrote:
muzz wrote:One problem I must look at solving is image naming - I've always left the images named out of the camera but when I upgrade my camera (as I did from a G5 to a 350D a few years back), the image/file numbers started to be duplicated. I think I'll have to rename the whole library all at once in Aperture (maybe concatenate the camera model and the filename or may be date and filename) and re-organise my library from there.

The same thing happens when you go around the clock on a single camera which did with my D70 and have already done with my D300. However, I don't see it as a huge problem as the dates of the files are so far apart it is easy to work out if it was first time, second time, etc.


It's actually very easy to use a date based folder naming convention. Try YYYYMMDD. It's self sorting, and easy to comprehend. As I offload files from my CF card, I just do it by folder, then rename the folder for the date.

I use YYYYMM for folders, but the files themselves still have dates on them if it came to the pinch and you needed to work out which DSC_0345.NEF you were looking at. Further, the NEF has EXIF embedded if you were really stuck.

Re: Thoughts on image storage and backups

PostPosted: Tue May 26, 2009 10:34 am
by aim54x
I do something similar to you Gary. I copy the folder onto my computer then name is XXXX_DD(MONTH)YYYY where XXXX are the event details (location usually). Not the best way but i find that I can search for the keywords or locations easily using this system.

I do a mirror backup of all my data onto an external HDD, I used to burn them off onto DVD/CD but this is proving to be difficult due to the volume of data I accumulate, often shooting more than a single DVD can hold....

Re: Thoughts on image storage and backups

PostPosted: Tue May 26, 2009 11:41 am
by darklightphotography
I had a Time Machine drive on my Mac fail last week, so my paranoia level has risen significantly. I now back up every card onto DVD and onto a network drive as well as my working drive before the card gets formatted. I have the network drive set so that I can only read and add files, this way I can leave it connected and not worry about accidental deletions.

I also upload all my full res jpegs to private galleries on Smugmug so in the event of total disaster I'll have something left.

When I shot film, I used a cardboard box to store negs and prints, so I'm way ahead of the game now.

For naming, I use 2 systems, one for my own work and one for client work. Once I've culled rejects I rename all my images to DLxxxxx and store in subject based folders for most things. I think I have about 87000 numbers to go before I run out. For client work I assign a shoot number X and rename images to XXXX-YYY. These are stored in folders by year. All of this renaming is done automatically with Bibble.

Re: Thoughts on image storage and backups

PostPosted: Tue May 26, 2009 10:14 pm
by DaveB
Thanks for the feedback folks!

ATJ wrote:I use YYYYMM for folders, but the files themselves still have dates on them if it came to the pinch and you needed to work out which DSC_0345.NEF you were looking at. Further, the NEF has EXIF embedded if you were really stuck.

What you've got is obviously working for you now, but I recommend against it. If you modify your workflow to make the filenames unique across your entire system you will in fact give yourself lots of flexibility for future reorganisations.

Incidentally, I've seen situations where the file timestamp gets lost and you only have the filename and the metadata to work with. Mainly when files have been restored off optical media.

Anyway, the post about filenaming is now up on my blog. Next scheduled installment will talk about folder structures.

Re: Thoughts on image storage and backups

PostPosted: Wed May 27, 2009 1:01 pm
by muzz
Hi Dave,

are you able to give advice on first-aid/rehabilitation of a current collection of images that require re-organisation and renaming. For most of us this will be in at least the tens of thousands of images. My cataloging program has always been iPhoto but I now also have Aperture 2 which seems similar to Lightroom as far as renaming goes.

I'm thinking it may be better to export the entire library and reimport to Aperture to change names. There is some ability in iPhoto to batch change but I'm not sure how flexible it is.

Cheers, Muzz

Re: Thoughts on image storage and backups

PostPosted: Wed May 27, 2009 1:22 pm
by DaveB
In my own environment (given that I "live and breath" Lightroom - I use it for my own work and I teach it) I would open up the iPhoto database (in Finder right-click on it and Show Package Contents) and copy the contents out to a normal file tree somewhere else. Then you can see what iPhoto has been doing with your files. It tends to make extra copies of the files when you tweak photos, so you may want to throw away everything but the "Original"s.
Then I'd import that tree of files into Lightroom and massage further. Within LR you can select files from throughout the tree (i.e. even if they're in different folders) and do things like rename, apply keywords/etc, move to a new folder, etc.

10,000 images will take a while to sort through, but you should be able to have most of it sorted within a couple of weeks (assuming you also have a job/life and can't spend all your time working on the files ;)).
Of course any tweaks you've applied to the photos within iPhoto won't transfer to the Adobe Camera Raw settings that Lightroom uses, so you'd have to re-do those. You might be able to also import the derived iPhoto images into Lightroom and keep them with the originals to show you what you're aiming for.

All of that's off the top of my head, and is fairly nuts-and-bolts. If you're using Aperture you have the advantage of being able to import iPhoto libraries although I haven't done this myself. If you go down that path I would suggest first getting used to Aperture with a small set of images and getting a feel for where your files end up on the disk. Then you can be more in-control of the import process.
Even if you bring the files into Aperture you'll probably still have a lot of work (weeks?) ahead of you organising the files within Aperture.

Re: Thoughts on image storage and backups

PostPosted: Wed May 27, 2009 1:44 pm
by muzz
Geez Dave, that was quick! Thanks for that info. I have imported different images and events from iPhoto into Aperture in the recent past and it seems pretty painless. What I realise I may have been doing in the past when backing up is only backing up the altered versions, using the Export command in iPhoto. I think I'll plan to import everything into Aperture event by event which will give me a way of ensuring that things are working smoothly as I go and compare it to the results from the folder tree method, and change from there if necessary.

So can I clarify that you suggest copying into a folder structure straight off the card and then importing/renaming, or are you using one of the applications to import directly into it and renaming on the fly - typically I have used iPhoto to import straight off the camera and used a separate backup off the CF card directly into a folder structure on a DigiMate III. I'm thinking that importing directly into Aperture will replace the iPhoto past practice, but am not sure if instead I should be simply dragging and dropping into a folder tree and importing from there, keeping both sets.

Something tells me you have more info in store on your webpage so I'll be keeping an eye out.

Thanks again for the info, and the webpages too.

Cheers, Muzz.

Re: Thoughts on image storage and backups

PostPosted: Wed May 27, 2009 2:20 pm
by DaveB
muzz wrote:So can I clarify that you suggest copying into a folder structure straight off the card and then importing/renaming, or are you using one of the applications to import directly into it and renaming on the fly - typically I have used iPhoto to import straight off the camera and used a separate backup off the CF card directly into a folder structure on a DigiMate III. I'm thinking that importing directly into Aperture will replace the iPhoto past practice, but am not sure if instead I should be simply dragging and dropping into a folder tree and importing from there, keeping both sets.

The import function in Aperture will read directly from your CF cards and rename the files as they're being copied to your drive/library. Some import programs provide a function where they place a duplicate copy of the files on another drive as an instant backup - I'm not sure if the Aperture importer has that. But you can make backups within Aperture: you should investigate the Vault feature.

If reading directly from the card streamlines the process and you can still maintain backup copies of your images, then making a 2-step process of copying the files to somewhere on your hard drive and then re-copying the files during the Aperture import could be seen as a backwards move. You'd also then have two sets of images you'd need to manage and back up of course...

Re: Thoughts on image storage and backups

PostPosted: Wed May 27, 2009 3:51 pm
by muzz
Thanks Dave. I'm going to have a good look at things when I get home tonight. I may wait for your article on folder organisation before proceeding further.

Cheers, Murray.

Re: Thoughts on image storage and backups

PostPosted: Thu May 28, 2009 11:08 am
by tntman
This is how my images are currently stored.

1st Copy - Stored on local PC on 2x 1TB HDD configured in RAID 0 - Note, next project is to convert it to RAID1
2nd Copy - Automatically synced to external Netgear Duo NAS device - Currently 2TB storage.

For archives I burn the images onto DVD discs at 4x and run a verify. Store the discs away.

For onsite shoots, I have a Epson P5000 backup device that has been very reliable.

Re: Thoughts on image storage and backups

PostPosted: Thu May 28, 2009 11:53 am
by DaveB
Ouch! The internal drive in my laptop just self-destructed.

Am currently restoring from backups onto a new drive (posting from a different machine obviously). Fingers crossed! ;)

Re: Thoughts on image storage and backups

PostPosted: Fri May 29, 2009 4:01 pm
by DaveB
My laptop's back up and running. Took about a day to recover the machine, and I did lose some files (but only about 1/2 a day's worth, so the backups did their job). Scary coincidence this happening while writing about it...

Anyway, the next post is up, talking about grouping your files.

Re: Thoughts on image storage and backups

PostPosted: Fri May 29, 2009 4:53 pm
by muzz
DaveB wrote:Scary coincidence this happening while writing about it...

Dave you should make your next article on winning Lotto - couldn't do any harm! :wink: Glad to hear that things didn't work out too badly. By the way, didn't TimeMachine keep any of the files with it's hourly backup?

I've read your post on folder organisation and I'm going to bite the bullet this weekend and get started. I think I may need to reread it once or twice and make sure I know what I'm doing. Certainly iPhoto's structure is a date based structure so it may well be easier to use that existing structure to begin with as you suggested but obviously need to strip out the thumbnails and other non-image files in there.

Thanks again for providing this resource.

Re: Thoughts on image storage and backups

PostPosted: Fri May 29, 2009 5:24 pm
by DaveB
muzz wrote:By the way, didn't TimeMachine keep any of the files with it's hourly backup?

After a panic that the Time Machine disk may have been corrupted in the crash (in the end, it wasn't) and worrying about how long it'd been since I imaged that disk, I restored the machine directly from the Time Machine backups.
That got most of it back, except for a few system config folders (the postfix email and apache2 web stuff was getting upset until I created some new folders for their logs/etc) and except for the areas I specifically exclude from TM. TM tries to make hourly snapshots, but doesn't cope with large files that keep changing (e.g. VMware virtual disks, large .lrcat files). Every hour it would try to make a new copy of each file that's changed, no matter how large, and even if other software is in middle of modifying it. This slows the machine down as well as chewing up all the space on the TM disk. Thus I use other software for backing up the files in most of my sets, and it doesn't run hourly at the moment. TM takes care of the OS, applications, email, Documents, etc. I had to copy the affected sets (the ones that had been on the failed disk) from their backups.

Also it turns out that TM backups must have started failing a couple of hours before the machine froze: while I was still creating work and saving new files. The most recent TM snapshot I could restore from wasn't quite as late as I would have hoped for.

Re: Thoughts on image storage and backups

PostPosted: Fri May 29, 2009 5:47 pm
by Murray Foote
DaveB wrote:... Scary coincidence this happening while writing about it....

Actually i remember when I was president of the Canberra Photographic Society in 1986, in my speech at the AGM, referring to the year receding like a lens dropping out of a helicopter.

Curiously enough the next year i got at least one return journey in a helicopter. At one point I was over the sea between Tasmania and Maatsuyker island strapped in the passenger seat of a helicopter with no door and while changing a lens, I dropped it ...

and watched as it bounced on the open door lip of the helicopter probably a thousand feet above the ocean and ...

...

... bounced back into the helicopter, seemingly undamaged.

Regards,
Murray

Re: Thoughts on image storage and backups

PostPosted: Mon Jun 01, 2009 1:42 am
by Murray Foote
A very useful summary, Dave. I have to admit I currently only have two copies of my data.

There's only one thing that occurs to me. You mention the desirability of having backups of all files, not just data files. Perhaps it might be worthwhile to suggest image backups as well as file backups, to increase the chance you can roll back your system when a serious problem occurs without having to reinstall all your software.

Regards,
Murray

Re: Thoughts on image storage and backups

PostPosted: Mon Jun 01, 2009 9:09 am
by hark40
There was a Lightroom blog post recently (just trying to find it, and of course can't!!!) where the author there was proposing that there was no need for you to backup your Lightroom catalogue, nor your photos. This caused some reaction from his readers, but he had a very valid reason for stating it. The reason? They should already be backed up as part of your normal usage of the computer. In that context, the blogger was 100% correct.

Essentially backups should be performed as often as they need to be based on the answer to the simple question - How much data/work are you prepared to loose if the system goes pear shaped? RAID 1 as mentioned is not a backup (power spike is the killer) and it is amazing how many people think that it is a backup mechanism. Storing the data on another HDD (or two) is a reasonable solution. A link off the blog post - bugger I wish I could find it - showed the setup for a professional photographer where they have a custom case complete with Mac and large LCD screen, along with a case full of removable HDDs. They use RAID, but only as a means of duplicating the data onto multiple HDDs for backup purposes. Those multiple HDDs are their backup - its all done as part of the process of taking and processing photos and nothing beyond that is required to be done. That is probably the ideal backup solution.

For us mere mortals, its a different story. (And for those that want to know, I store my LR catalogue on my workstation, the photo files on the server. The catalogue gets copied to the server, the server gets backed up on a semi-regular basis to an external HDD, plus the RAW images get backed up onto two different DVDs when I've taken another 4G of photos. Just have to implement an off-site backup process and I should be complete.)

I sort photos by date & "shoot", with sometimes multiple folders for the same day if I end up shooting multiple things. Fortunately, I don't shoot that much (bugger) for that to be a real problem. Each image is renamed uniquely upon import into Lightroom. It works for me.

The thing is, naming of files and sorting of files/photos tends to be a rather personal thing. There are many correct ways, but there is no one correct way. However, there are probably a number of wrong ways of doing it.

When it boils down to it, there are some interesting thoughts/ideas there DaveB.

Mark P.

Re: Thoughts on image storage and backups

PostPosted: Mon Jun 01, 2009 10:02 am
by ATJ
hark40 wrote:There was a Lightroom blog post recently (just trying to find it, and of course can't!!!) where the author there was proposing that there was no need for you to backup your Lightroom catalogue, nor your photos. This caused some reaction from his readers, but he had a very valid reason for stating it. The reason? They should already be backed up as part of your normal usage of the computer. In that context, the blogger was 100% correct.

I agree about the image files themselves, but not the Lightroom catalog.

I don't backup every single file on my hard disk. Just the important ones. I won't backup executable files, for example (at least not as part of a regular backup. For my regular backups, I only backup files that are likely to change and I would need to get back in the event of a failure.

Before I started using Lightroom, I was already backing up all my image files - and have done for years. I had to specifically add the Lightroom catalog to my routine. If I hadn't, it wouldn't get backed up.

Of course, as I add any program that creates files I want to keep in the event of a failure I will add it to my routine - which perhaps is the point the blogger was trying to make. However, I think the original point is somewhat misleading.

Re: Thoughts on image storage and backups

PostPosted: Mon Jun 01, 2009 8:46 pm
by Murray Foote
I think the converse question is probably more interesting. I wonder how many are exposing themselves by thinking Lightroom is backing up their files and not just the catalogue?

Re: Thoughts on image storage and backups

PostPosted: Tue Jun 02, 2009 12:40 am
by DaveB
Thanks for the feedback so far!

The next article won't be up for another day or so, but does talk about some of these issues. I've got a rant about RAID, but I've moved that out to a separate post that might be up later in the week.

Re: Thoughts on image storage and backups

PostPosted: Wed Jun 03, 2009 11:59 pm
by Alex
I just bought a 1TB external hard drive for image back up. However, I am using Nero back up software to manage automatic back up. I do not like this software as it writes back up in its own format and I cannot access files on that drive unless I go through a restore process using the Nero software. Is there a more user friendly application that backs up files automatically in their native format so I can access those files directly from that hard drive any time without using special software?
Thanks for any advice.

Alex

Re: Thoughts on image storage and backups

PostPosted: Thu Jun 04, 2009 8:51 am
by ATJ
Alex wrote:However, I am using Nero back up software to manage automatic back up. I do not like this software as it writes back up in its own format and I cannot access files on that drive unless I go through a restore process using the Nero software. Is there a more user friendly application that backs up files automatically in their native format so I can access those files directly from that hard drive any time without using special software?

Alex, are you doing compression as part of the backup?

I use Nero to backup to DVDs and found that with compression on, the only way to access the files was by restoring with Nero. With compression turned off, the files are written to the DVD as is. I even run a compare (using CDCheck) to compare the files on the DVD to the hard disk to ensure it was copied correctly. The only minor problem I have is that one to two files per DVD get split across two DVDs and so I can check those (although I believe a simple copy filea+fileb filec would combine them.

Re: Thoughts on image storage and backups

PostPosted: Thu Jun 04, 2009 8:58 am
by gstark
Alex wrote: Is there a more user friendly application that backs up files automatically in their native format so I can access those files directly from that hard drive any time without using special software?



For the PC, try Syncback. I think that's it's name. Gives you options to syncronise files on two drives, exclude folders ... much better and less top heavy then MS's SyncToy.

Re: Thoughts on image storage and backups

PostPosted: Thu Jun 04, 2009 12:16 pm
by Alex
Thanks, Gary. Will check it out.

Regards,
Alex

Re: Thoughts on image storage and backups

PostPosted: Thu Jun 04, 2009 12:44 pm
by DaveB
Latest blog article is now online!

Re: Thoughts on image storage and backups

PostPosted: Sat Jun 06, 2009 10:04 pm
by DaveB
Latest post: RAID is not enough.

Re: Thoughts on image storage and backups

PostPosted: Sat Jun 06, 2009 11:04 pm
by surenj
Thanks Gary for the info on Syncbackup! This program absolutely rocks for freeware!

Dave, your articles are much much appreciated.

Re: Thoughts on image storage and backups

PostPosted: Sun Jun 07, 2009 2:06 am
by Murray Foote
Likewise, Dave, great articles.

Dave on Image Backups wrote:Of course any operation like this requires some "down time" to have the system idle (or shut down) while the copy is being made. The hassle involved in making a backup this way usually means that they happen infrequently, so it's not a perfect solution. But if your boot drive fails it can be nice to be able to boot off the backup drive and know you have a complete working system.

Acronis True image 2009 allows you to set up automated backups which can include image backups and it can run them when you turn off your computer at night. You can schedule different jobs at different frequencies so you might back up your data every night and your image evry month. The only catch with this is that if it encounters a problem your computer will still be on in the morning.

Logically, the time for an image backup is when your image changes - eg you install some new software. I should remember that more often because i automate my file backups but not my image backup. An image backup is much faster than a file backup, though.

Acronis can also create "rescue" disks that allow you to bootup when your C Drive fails. Then, if you need to replace the drive and you have an image backup, you can put that image on the new drive (restoring all your data and software).

Regards,
Murray

Re: Thoughts on image storage and backups

PostPosted: Sun Jun 07, 2009 11:39 am
by DaveB
Murray Foote wrote:Acronis True image 2009 allows you to set up automated backups which can include image backups and it can run them when you turn off your computer at night.

Cool. These aren't bootable image backups are they?
The bootable images that I used to make of the Windows laptop here were made by booting into Knoppix (a Linux variant on a CD) and using 'dd' to copy the entire 160 GB disk to a matching drive in an external USB box. Took about 2 hours, so we'd run it overnight. Actually we made a shell script to run 'dd' and put that on a USB stick, as it's very easy with dd's arguments to accidentally copy the other way and screw up the internal drive :evil:. We were able to (and had to several times) swap the internal disk with the USB disk and boot up with exactly what we had at the last backup.

Acronis can also create "rescue" disks that allow you to bootup when your C Drive fails. Then, if you need to replace the drive and you have an image backup, you can put that image on the new drive (restoring all your data and software).

This would seem to be essential, not just something it "can" do. Having a backup you can restore from directly is very important, rather than having to reinstall the OS from original media, apply patches/etc, then the applications required to restore from a backup, THEN start restoring.
If it "can" do it, you should do it, and you should check every now and then to make sure the restore process actually works.

Apple's Time Machine backups can be restored directly from the OS install media, but even so it's worth doing it every now and then (to a spare USB/Firewire drive for example) to check that everything in the process works the way you expect.

Do I sound paranoid? ;) That's just the experience showing through... BTW, earlier this year we had to reinstall several machines from backups onto different hardware (the original machines were in for repair). The OS X machines just kept on running on the new hardware (e.g. MacBook Pro -> MacBook, MacBook -> iMac) but Windows wasn't happy at all when the machine changed. Microsoft Support's answer was to deauth the original machine, but it took a long time to get through to them that the original machine was temporarily out of commission. Eventually we got them to free things up so we could let the new machine continue for the weeks it was going to be needed for, but it was a pain. So much so that when the original machine came back we couldn't face up to going through the whole rigmarole again to get the machines back.
Score one for OS X...
The Adobe Creative Suite apps also lock down to your hardware, but they were easier to get sorted (partly due to the fact that they lock down to any two of your machines).

Re: Thoughts on image storage and backups

PostPosted: Sun Jun 07, 2009 7:00 pm
by Murray Foote
DaveB wrote:
Murray Foote wrote:Acronis True image 2009 allows you to set up automated backups which can include image backups and it can run them when you turn off your computer at night.

Cool. These aren't bootable image backups are they?

No, I don't think so because they are compressed, about 50% in my case. You'd need to boot up off a rescue disk and then retrieve the image.

I know you can restore an image backup to a different disk but I don't know if there might be any Windows hardware authentication problems because I've nevr needed to do it.

Regards,
Murray

Re: Thoughts on image storage and backups

PostPosted: Sun Jun 07, 2009 11:22 pm
by Alex
DaveB wrote:Latest post: RAID is not enough.


Hi Dave,

Just read your blog. Thanks heaps for posting it.

Cheers
Alex

Re: Thoughts on image storage and backups

PostPosted: Thu Nov 05, 2009 12:04 am
by Murray Foote
DaveB wrote:
Murray Foote wrote:Acronis True image 2009 allows you to set up automated backups which can include image backups and it can run them when you turn off your computer at night.

Cool. These aren't bootable image backups are they?

It sounds as though it can be with Acronis 2010:
Windows 7 Ultimate users can boot from a backup image
(.tib file) of their system partition. This will allow testing the ability to boot a backed up system without actual restoration. If the operating system boots from the .tib file, then it will boot when the need arises for a full system recovery.

It also allows for incremental backups evry 5 minutes, which perhaps only makes sense under dual or quad core and Win7 64 bit.

Re: Thoughts on image storage and backups

PostPosted: Sat Jan 02, 2010 7:59 pm
by Mr Darcy
I have finally picked up a couple of external drives care of Big W's post XMas sales
(1.5 Tb USB2 @AUD148) so I am trying to get backups happening a bit more rigorously than just relying on TimeMachine.

From what Dave, and others have written, the answer seems to lie in rsync. After much scratching around, I found it via terminal, but cannot get it to do anything useful. I have tried many variations on the theme of
Code: Select all
rsync -avz . /Volumes/1T5Backup01

which seems to be what the documentation suggests for a simple local backup (Fancy will come after I get this to work!)
but cannot get any of them to do much better than
Code: Select all
building file list ... rsync: link_stat "/home/*" failed: Input/output error (5)
done


Can some one please teach this old dog a new trick or two. Pretty Please.

Of course, once I get this working, I need to get cron to drive it. That, I am sure will be another frustrating day or two

Re: Thoughts on image storage and backups

PostPosted: Sun Jan 03, 2010 12:38 am
by Murray Foote
No idea but good luck. I notice (from Wikipedia) that the utility actually emanates from a thesis at the ANU about 15 years ago.

Re: Thoughts on image storage and backups

PostPosted: Sun Jan 03, 2010 6:38 am
by gstark
Greg,

Does rsync require you to be using the same drives on both your source and target volumes? It's been a while since I had a play with it.

Are you looking to do a full backup of the drive? If so, have a look at SuperDuper. It's free, and it will very painlessly back up your system to another volume. I've used it several times when wanting to migrate to a larger HDD in my MacBook.

Re: Thoughts on image storage and backups

PostPosted: Sun Jan 03, 2010 9:34 am
by Mr Darcy
From what I can make out, it doesn't care. The documentation indicates it will back up both locally and across the internet. At the moment, I am just trying local (baby steps!) but the idea of a remote backup to another site is appealing. Since I stopped working on location, offsite backups have been in the too hard basket.

What I want to set up is a regular incremental backup with occasional full backups, similar to what I achieved with XCOPY and AT in the DOS (Windows) environment I have posted my DOS BAT file here somewhere in the past.

I will look into SuperDuper, but your use of it indicates it is more of a migration tool than a backup tool. Similar to Ghost in the DOS world.