how do you store your files?

[OzzyJim]

Gday,

I have been thinking about different ways of storing and backing up all the photo's that I have on my pc. At the moment I have all my photo's stored on external hard drives and thinking of just creating a copy of that drive to another for backup, the trouble is that with a very large number of files this can take a long time over USB 2.0 connection. I have been looking into different methods and like a mirroring raid solution as it seems to be a type of automatic backup. Looking further into this there are numerous types of ways you can set up a RAID array (IDE, SATA, SCSI, Fibre Channel etc).

My question is how do you do your backups and store files for easy access later?. Also how is this accomplished in a large studio setup, I imagine that a large commercial studio would have a striped / mirror combo over a SCSI or fibre channel connection for sheer speed and reliability. I have been looking into the SCSI striped / mirror combo for my own use but it is very expensive, speed is not as important for me as the backup but I really hate to wait when loading large PSD's etc.

James.

[iconoclast]

Hello OzzyJim,

I would recommend against SCSI, once upon a time it was the defacto RAID option but nowadays most SATA motherboards have raid, its all built in. Also SATA drives are so d@mn cheap!

The backup methodologies and systems vary on size, skill and budget.

Personally I don't have any redundancy...haha.. I could lose everything at any time. The only thing I have going for me is that I have my OS installed on a different partition But then again this isnt my lively hood, and I'm broke!

I'll try to describe a few different ways of doing things....

1. workstations have space on the server. the worker bees may keep some files locally but basically keep their files on the server. the IT peeps back up the server in several ways.... a RAID array..... a backup computer in another part of the building or off site and/or physically removable backup options like tapes or drives.

2. at home you might have a few computers and a server, connected by a fast giga lan. on your workstation you could have 1 drive for your OS and apps. and then 2+ drives in a RAID array and then have a scheduled back up occuring every week or so to your server. On your server you could have the same sort of set up. Then also a syncronizing external hard drive (or optical disks or tape drives but I think the external hard drive is the best option), plug it in every week or so and let it do its magic.

3. Something thats basically just as redundant as #2 is having.... in your workstation 1 hard drive for the OS, 2+ drives for a RAID array and then back up with a syncing external drive.

The external back up solutions are important because there's one thing that you can never recover your data from: fire! You should keep your external backups (drives, optical disks, tapes etc) inside a reliable fireproof safe. If you wake up at night you shouldn't be running into your study to fetch your external drive.

With regards to speed/performance....... As you've noted its not really speedy to work on files that are located on a USB drive or network. But SATA RAID arrays offer great performance, and you don't really need the drives mirrored either, just having your OS on its own disk is going to give you a good boost anyways. Though a 4 disk mirrored RAID array would be swee haha.

It's not clear from your post but do you back up your files onto your external drives? Or do they reside only on the external drives and not on the internal drives? If the former I'd recommend rigging something up internally and syncing them to your external drives due to the higher fail rate of external drives.

[pommie]

I have two 250gig drives running all the time, one for image data and the other for the OS (Linux) and everything else.
I also have one 250gig drive that I only plug in when doing backups of my images, the rest of the time it just sits in the case unplugged, yeah I know if there is a fire I lose the lot, but if my house burns down I think that would be last thing I will be worried about
And transfering data from one sata2 hdd to another is wayyyyyy faster than via usb.

Cheers David

[OzzyJim]

Gday,

I have just been looking up some info before I checked this thread and although I knew SCSI would be expensive, I did not think it would be Soooo expensive. A SCSI Raid setup with 2x 300GB 15k rpm hard drives and controller was going to cost a little over $1,500 Aus while 2x 1TB Hard drives to use in a raid array with my pc would cost a bit over $350 Aus with a removable HDD caddy. The difference in speed is not much either:-

Ultra-320 SCSI: 320 MB/s or 2560 Mbit/s
SATAII: 300MB/s or 2400 Mbit/s
Versus the USB2 method: 60 MB/s or 480 Mbit/s!!!!!

I figure the removable HDD caddy was a good idea so I can pull the disk when not in use and store it elsewhere or in a fire-proof safe.

I do not really like software for backups as I'd probably foret to run it and don't know that I'd always trust it - What happens if it crashed - could it destroy any of the files??. At the moment I do not backup all my files to DVD due to the time it takes to burn , verify then label and store. I appreciate your input so far and am also looking into a UPS with voltage protection, although I have had a pc power supply fail and kill every component inside a pc before I'm sure that is very rare.

I store my images on a USB2 HDD at the moment, a slow portable drive powered by the laptop as I figured this was safer from voltage nasties (spikes etc) than a normal external drive. This also had a advantage that no data would be lost in the event of a power failure as I use a laptop at the moment but when I can get the UPS I will use a desktop pc.

James.

[zemlin]

I don't consider mirrored drives a backup. It is insurance against a failed disk and improves data transfer but it doesn't offer any protection from the most common failure - human error.

I am certainly guilty of inadequate photo backup - I just plopped a 750GB disk in my box for data storage 'cuz I was running out of room and it wasn't much more expensive than a smaller disk. My intent is to backup old photos to DVD disks. 2008 will require a dozen DVDs - yes, that will take a while to burn, but it's a one-time thing. Before I do that, I need to setup a batch to export small JPGs from all my NEF files to make it easier to find things.

[jforkner]

I use a Drobo unit (Data Robotics, Inc.). It's basically an external RAID array with hot-swappable slots for up to four drives---although they have a unit, now, that accepts up to eight. It works great for me---all my system files are on an internal 10K rpm drive; all my data files are on the Drobo. Although it doesn't, technically, provide redundancy; it spreads the files over multiple drives and I'm comfortable with that. I'd suggest watching the product demo video on their website.

I also have a separate external drive that I use to backup "really important" files.

Jack

[iconoclast]

Hi OzzyJim,

Those numbers you list for SATA, SCSI and USB are just theoretical maximum bandwidth figures and should not be interpreted as real world performance. And it refers only to the technology that connect a device to the computer not the device itself. And in the case of hard drives the connection technology is not the bottleneck.
There is a huge amount of variables that effect real world performance of a hard drives for more indepth info or just an overview check out StorageReview.com

A drive caddy, holding 1 of the hard drives of a mirrored raid array is not a bad idea but that drive will always be in the computer while the computer is on, basically. It's not something you can store in a safe and then plug in and expect files to be copied over; thats not how a raid array works. The caddy option would be helpful if you needed to transport your files in a pinch, like a fire, tsunami or something. However it may get tiring to get it from the safe, plug it in and take it out and store it every single time you think you might be using the computer for digital work.

I'll again recommend the external drive (USB2, Firewire or hotswappable drive caddy) again, in addition to an internal mirrored raid array. There is software that would allow you to simply plug in the drive and it'll copy everything automatically. So say you wake up Sunday, brush your teeth and plug the drive in, go about your day and an hour or two later everything is copied. Then you store the drive in its fireproof safe(small safes are cheap and common in the USA) out of site, in order to protect from fire and theft. I have a friend in Victoria, she says crime is terrible there, is that true?

With regards to loss of data during powerfailure.... You only lose whatever data is being edited at the time, generally, only files that are being accessed at the moment are subject to corruption and loss. Lots of mainstream apps will save your work internally every few minutes and you can recover the data next time you run it (I'm thinking of MS office and Open office). This possible loss of data is also true when a computer crashes. An external USB drive vs an internal drive isn't really proof against these cases of lost data.

I don't know that USB2 drives are any safer from electrical woes than an internal drive. There are a lot of variables. The drive uses its own outside powersource and its connected to the computer via USB which carries a current, thus it has points of entry. Furthermore you have to consider the quality of the USB chip and card in the computer. Thats usually a no brainer if using a modern computer (laptop or desktop) but there's lots of old crappy addin cards that might fry a USB device it dosn't get along with. Also its external power has to be protected just as the computer's does. Also an external drive is subject to knocks, falls and generally worse heat issues.

I'd recommend instead a high quality power supply in a desktop computer and a high quality fused surge protection. PC powersupplies do fail, and its usually because of their cooling fans die. It's part of good computer paintenance to clean the dust off it every 6 months or so. Usually I just pick off the dusk by rubbing my hand against it but moistureless canned air might be used too. Also pay attention to how your computer sounds, if fans stop you should investigate. The cpu fan should be cleaned up too.

[iconoclast]

zemlin,

Hello. I understand what you mean by backing up on all accounts and am guilty myself. Sadly the ways to guard against human error are few and imperfect.. haha
However mirrored RAID arrays safeguard against disk failure, of which there is much. Esp nowadays it seems to me, disks are more prone to failure. Data recovery on a failed disk is very expensive, more so than buying an extra drive for mirroring. The thing I like best about mirrored RAID is that it dosn't rely on human interaction.
Ofcourse this method dosn't protect against human error, fire or theft of ones computer, that's why multiple strategies of back up and redundance should be taken at different steps of the workflow.

[Stercus Vultus]

Originally Posted by zemlin

My intent is to backup old photos to DVD disks. 2008 will require a dozen DVDs - yes, that will take a while to burn, but it's a one-time thing. Before I do that, I need to setup a batch to export small JPGs from all my NEF files to make it easier to find things.

I hate to burst your bubble, zemlin, but burned DVDs (as opposed to pressed DVDs) are only suitable for medium term storage.
Burned DVDs will become unreadable after about 5 years because the burning starts an osmosis process that degrades the aluminium layer, eventually beyond recognition/readability.

I.o.w. DVDs are unsuitable for long term storage. If you rely on them for backup you will get a very nasty surprise.

[robertro]

I've struggled with this myself.

I shoot about 1000 images/month, and this quickly adds up. I had initially considered DVDs, but it was too awkward to burn them, and I too heard rumblings about their permanence. These days, I store all photos on two identical USB external drives, backing up from my local drive quarterly. I'd like to do this monthly, but time is scarce!

[OzzyJim]

thanks for your help,

iconoclast, I think I will be taking your advice and when using a mirrored Raid setup - backing up to a USB or pref a firewire / esata drive also. Because of the fact a powersurge can cook computers / external hard disks I had been using for the time being a USB powered portable HDD for backup - I never thought the USB controller may fry it!. I guess if that happens I'd hope the USB - IDE/SATA controller in the HDD case blows rather than the logic board itself so I can recover the data. I will have a look at the link to storgae review later today when I get a chance thanks for that.

I did see a quick video on youtube where FreeNAS was used on an old pc to act as a backup source and that scripts can be written to automate this. As for theft, in my local area it's not too bad actually, more from inside cars than houses at the moment. I will have another look into what software is available for backups.

Thanks,

James.