The Data Backup Thread (& request for more suggestions)

Pages

This thread is for discussing systems for backing up data.

The topic comes up often, and I figured it would be nice to have somewhere to point people.

I've listed below some online backup services, as well as tools for offline backups. Please reply and share your backup solutions, and I'll add them to the top post. Also, I know I am missing plenty of online backup services and offline tools from this list, please point them out!

BACKUP BASICS

* Onsite vs. Offsite: Onsite backups are backups that reside in the same physical location as the machine they were pulled from. Offsite backups are backups stored in a different location. There is merit to both. Onsite backups are typically quicker to restore from, while offsite backups protect against data loss from damage to the computer's physical location (fire, floods, etc), which would destroy onsite backups too.

Ideally, you want one of each. Offsite backups may be a hard drive in a safe deposit box, burned DVDs taken with you to work, or a service that stores your data on someone else's rack of servers "in the cloud".

* RAID is not a backup: There are various forms of RAID, but in general, the purpose of RAID is to provide redundancy. It protects against the immediate effects of drive failure. This, however, should not be confused for a backup. Why is it not a backup? Backups protect against more than drive failure. If you suffer data corruption, your RAID system is going to write that corrupted data across all drives in the array. Bam, data loss. RAID is simply for maintaining uptime in the face of drive failure.

BACKUP SERVICES

There are online services that provide a few different types of backups:

* Normal Backup: Couldn't think of a better name for this. You select files/folders on your system to be backed up, and copies of these selections are stored for safe keeping.

* Sync Folder: This is a folder on your computer whose contents are automatically sync'd with a cloud-based service. You stick a file in this folder, it gets uploaded and stored in the cloud. Generally used for sharing documents across multiple PCs, but also serves as a useful backup.

* Network Share: Similar to a sync folder, but here, the remote storage is mounted as a network share. The local storage is just a cache system for temporarily holding items being read from/written to the remote storage. Isn't necessarily a "backup" per se, except the remote storage is typically a cloud-based solution that has its own redundancy, so your files are a lot safer there than your own hard drive.

CrashPlan (normal backup): Excellent cloud backup system, with inexpensive unlimited storage plans. Works on Linux, Mac, Windows, and Solaris.

JungleDisk (normal backup, sync folder, network share): Amazon S3-based cloud storage. One of my favorites as it supports every kind of backup you could want, and on all the major platforms. Works on Linux, Mac, and Windows.

Carbonite (normal backup): A flat-fee unlimited storage backup system. Works on Mac and Windows.

Mozy (normal backup): Similar to Carbonite, a flat-fee unlimited storage backup system. Also has a free version with a 2 GB space limit. Works on Mac and Windows.

Dropbox (free, sync folder): 2GB of cloud storage, (up to 8GB with referrals). A folder whose contents stay in sync on all your computers/ Works on Linux, Mac, and Windows.

Live Mesh (free, sync folder): 5GB of cloud storage. Similar to Dropbox. Windows only.

Ubuntu One (free, sync folder): 2GB of cloud storage, also stores contacts and other user data. 50GB storage available in $10/mo paid version. Ubuntu Linux only.

ROLLING YOUR OWN

Sync to external hard drive: There are many pieces of software you can use to sync files from your computer to an external hard drive. Some tools include:
* rsync (UNIXes)
* rsnapshot (Linux)
* LuckyBackup (Linux)
* SyncBack (Windows)
* Synkron (Linux, Mac, Windows)
* SyncToy (Windows)
* Unison (UNIXes, Windows)
* Second Copy (Windows)

Sync to NAS: Similar to above, for the slightly more hardcore. Sync to a NAS (network attached storage) instead of an external drive. Useful for backing up multiple computers on a network, instead of just one. Same software tools as above apply.

Burn data to DVD-R: Most systems have DVD writers these days, which is an easy way to get a copy of your data off of your hard drive and into a format that is easily taken offsite. For backup jobs that exceed the size of a single DVD, there are software packages which will span backup jobs across multiple discs. Some of these tools include:
* Macrium (Windows)
* DiscSpan (Linux)
* ... more, gotta find 'em...

Windows Home Server: OS for installing on a home server, with utilities for automatically backing up Windows-based PCs on your home network. Windows only.

Time Machine: Built-in backup software in OS X. Backs up to external disk or to a Time Capsule (wifi-enabled networked hard drive). Mac only.

Distributed version control system: Mostly the realm of software developers. Usually used for maintaining a central code repository. The nice part about the "distributed" VCSs is that each checked-out version of the repository is a full copy of the repository itself. Some people use a DVCS like Git to backup some of their files to external services like GitHub.

(Below is where I berate people for not having a backup system in place. Skip if you want... at your own risk! )

#=====#
My Soapbox Rant:

Do you fail the *Legion* Backup Test? It's a simple principle. The test states that this very moment, you should be able to completely wipe your OS (Windows, etc) from your computer, and be able to get back to exactly where you started from with a little OS and program reinstalling, and recovery from your backups. In other words, if you are unable to wipe out your OS this instant and still have everything you need safe, you fail the *Legion* Backup Test.

Yes, I enjoy naming sh*t after myself.

I am of the firm opinion that every single computer user needs to follow a few tenets:

1. Recognize that you have important, irreplaceable (or at least not easily replaced) data
People often say "I don't have anything important". Then they lose a hard drive and freak out because digital family photos are lost forever. What are you out if I walk into your house and take your computer and disappear with it forever?

2. Treat your OS (and hard drive) as disposable
Many people are truly playing with fire with their data. If you have no backup, the life of your data hangs on whether your OS remains functional and your hard drive remains alive. If you're a techie, you can deal with your OS breaking, but even you are screwed the moment your hard drive fails, and fail suddenly and spectacularly they can.

Oftentimes, the OS isn't necessarily broken, but it's the presence of non-backed-up data that causes people to spend forever trying to fix Windows instead of spending the half hour it takes to wipe it and start over. If this is the case, you fail the *Legion* Backup Test.

Sometimes, you haven't even lost data or the ability to use the PC, but some piece of malware has exploited a hole in Flash and has taken root in your computer. You should be nuking Windows right now but you've got MP3s and porn you don't want to lose and so now you're spending hours trying to clean the system up.

The fact is, whether you like it or not, your OS is disposable. You should treat it that way. Once you do, you realize that dealing with computers becomes about 1000 times easier.

3. Set your most important data to be backed up automatically
Backups that are a pain are backups that don't get made. If you forget to refresh the backup on your external drive for weeks, you're not really protecting yourself.

Most of the online-based backup services have the ability to be set to automatically perform backup tasks. Your most important stuff needs to be backed up in a "set it and forget it" system.

4. Even a crappy backup system is infinitely better than none
It's not necessary to set up a hardcore backup system with all sorts of redundancy.

Ideally, one should have a combination of onsite and offsite backups. But if you're starting from scratch, just having a little Windows utility backing up to a Western Digital My Book external drive, or burning your documents folder to DVD every week and taking the disc to work, puts you in a much better position than before.

#=====#

Great post!

I do manual backups of my own data (pictures, music) to a secondary drive as they don't update that often (It would suck if I lose ALL my pictures, but if I lose the pictures of my latest vacation, I'll live).

But my GF has a lot more key documents and I'd like to use a sync for her. My problem is that I've found most sync programs to be fairly resource intensive and cause problems. I've recently tried the free version of Genie Timeline, and it not only slowed down her computer but also crashed firefox regularly. Right now I just have regular old windows 7 backup doing scheduled updates to an external drive.
I do want to put some of her stuff on the cloud as well, though, and a sync folder would be nice - which of Dropbox/Mesh would you say is least resource intensive and least likely to cause problems? We have a crappy upload speed, so it would likely be for Word documents only.

Ok, so I tried dropbox and I have a problem. To do the automatic sync with dropbox, you have to put stuff in the "my dropbox" folder. I don't like that. I want a program that I can tell what folder to sync to the could, not a program that tells me what folder to use. My gf has a very specific file structure and having to use the dropbox folder would mess that up, and manually dragging and dropping stuff to the "my dropbox" folder defeats the purpose of an automatic sync system.

Dysplastic wrote:

Ok, so I tried dropbox and I have a problem. To do the automatic sync with dropbox, you have to put stuff in the "my dropbox" folder.

Symbolic links could take care of this but that's a UNIXy solution.

I think Live Mesh lets you pick your sync folders. JungleDisk does too but is not free.

This is a great resource, Legion. Thanks for posting it.

Let me join in the chorus about RAID not being a backup. RAID prevents downtime from drive failure, and in a few limited cases can speed up drive I/O. You're much better off with two separate filesystems than a RAID-1... use the second drive as a backup target.

If you fat-finger something, or a buggy program and/or failing hardware writes out corrupt data, RAID faithfully does exactly what it's told, and corrupts or deletes all copies. When you have a separate filesystem, or online backup, and have several generations stored, you have a much better chance of recovery.

*Legion* wrote:
Dysplastic wrote:

Ok, so I tried dropbox and I have a problem. To do the automatic sync with dropbox, you have to put stuff in the "my dropbox" folder.

Symbolic links could take care of this but that's a UNIXy solution.

I think Live Mesh lets you pick your sync folders. JungleDisk does too but is not free.

There's a Dropbox add-on called Folder Sync that I've heard about but I have not tried it myself yet.

I got together with my neighbor a while back and we picked up two NAS boxes and installed them in each others house. Mine resides in his house and is connected to my wireless network and vice versa. I use Second Copy to sync data between my PCs and the NAS device. It's offsite storage at 802.11n speeds, so it's very convenient.

Of course there are plenty of natural disasters that could take out both houses at once, so it's not 100% protection, but it's good enough for my needs right now.

Of course this type of plan requires good neighbors, so it's not for everyone

Serengeti wrote:

Of course this type of plan requires good neighbors, so it's not for everyone

Encryption is your friend. Depending on what you're backing up it could be a good idea regardless, for example financial records being stored in someone else's house. Even if you trust them, if they get burgled it would be a bad thing for you as well.

Scratched wrote:
Serengeti wrote:

Of course this type of plan requires good neighbors, so it's not for everyone

Encryption is your friend. Depending on what you're backing up it could be a good idea regardless, for example financial records being stored in someone else's house. Even if you trust them, if they get burgled it would be a bad thing for you as well.

Any documents that are sensitive are PGP encrypted, so that's not a concern for me anyway.

Great post. I feel this. I have a home made WHS with file redundancy turned on for on site and then also have a VPN to my Dad's house where I have an old pc working as a Nas that he turns on once a week for me to sync via the WHS backup system. It works well enough but can take a while to remote backup depending on how many photos my wife takes. I don't off site backup my OS, just valuable folders. I also use dropbox to sync and backup my save games with symbolic links. I got a little utility that makes them really to create.

I've got an WHS on which all of my data lives. It's set to automatically duplicate every file onto separate drives (three drives currently installed). It also images all of my machines each night. I then use SyncToy to synchronize the data files from the WHS back to my main PC's hard drive. Once a month I backup the WHS to an external USB.

So I have two copies of every file instantly, three copies nightly, and four copies monthly. The biggest weakness right now is all those copies are within ten feet of each other. My buddy and I keep meaning to buy 1TB external drives and swap them back and forth each month so we're each storing the others data, just haven't gotten around to it yet.

By the way, how secure is Dropbox? If for instance you had an Excel file that contained all of your passwords, and that file itself was password protected, would you be comfortable putting that on Dropbox for easy access on all of your computers?

Teneman wrote:

I've got an WHS on which all of my data lives. It's set to automatically duplicate every file onto separate drives (three drives currently installed). It also images all of my machines each night. I then use SyncToy to synchronize the data files from the WHS back to my main PC's hard drive. Once a month I backup the WHS to an external USB.

So I have two copies of every file instantly, three copies nightly, and four copies monthly. The biggest weakness right now is all those copies are within ten feet of each other. My buddy and I keep meaning to buy 1TB external drives and swap them back and forth each month so we're each storing the others data, just haven't gotten around to it yet.

By the way, how secure is Dropbox? If for instance you had an Excel file that contained all of your passwords, and that file itself was password protected, would you be comfortable putting that on Dropbox for easy access on all of your computers?

I use Truecrypt with my dropbox to make sure... Truecrypt is great I use it to make sure the things I want to be private stay that way.

Dysplastic wrote:

Ok, so I tried dropbox and I have a problem. To do the automatic sync with dropbox, you have to put stuff in the "my dropbox" folder. I don't like that. I want a program that I can tell what folder to sync to the could, not a program that tells me what folder to use. My gf has a very specific file structure and having to use the dropbox folder would mess that up, and manually dragging and dropping stuff to the "my dropbox" folder defeats the purpose of an automatic sync system.

What you can do is just create a Junction It's a symbolic link but an NTFS version of it, works great and people use it for a bunch of stuff. An example would be moving your steam folder.

To the OP on WHS: Windows Home Server isn't Windows only. It can be configured for Time Machine storage as well, and the upcoming Vail version has this built in natively so you don't have to do a lot of setup.

As for the WHS offsite situation, has anybody ever looked at one over VPN? I was thinking I might set a box at my mom's house and have it VPN connected to my house. Then I could possibly run a scheduled sync of important folders across two WHS boxes. I would pre-seed it with all the big files so that it wouldn't be a huge data connection problem and it would only need to move the new stuff over time.

Probably an easier way, but i wouldn't mind getting my mom on a WHS box to back up all of her financial records anyway. I'm thinking I might just build a more powerful box for me with Vail instead of just upgrading my AMD 4830e box, as the Vail upgrade will have support for more media streaming and transcoding to mobile devices. That will require more horsepower I bet.

MonoCheli wrote:
Teneman wrote:

By the way, how secure is Dropbox? If for instance you had an Excel file that contained all of your passwords, and that file itself was password protected, would you be comfortable putting that on Dropbox for easy access on all of your computers?

I use Truecrypt with my dropbox to make sure... Truecrypt is great I use it to make sure the things I want to be private stay that way.

So you create a Truecrypt volume file, send it to Dropbox, and then store actual files within it? As far as Dropbox is concerned they're storing a single file? That's brilliant. I currently use Truecrypt in that fashion on a thumbdrive, but never even considered doing the same for Dropbox. Thanks for the tip!

Syncback has been fantastic for me and it's free! Once I set it all up, it does a daily sync with my files folder HD and game HD. I have it at sync instead of backup because if I delete something I usually want it gone.

But big thumbs up for Syncback if you are looking for a free and easy way to backup.

Teneman wrote:

So you create a Truecrypt volume file, send it to Dropbox, and then store actual files within it? As far as Dropbox is concerned they're storing a single file? That's brilliant. I currently use Truecrypt in that fashion on a thumbdrive, but never even considered doing the same for Dropbox. Thanks for the tip!

Protip

MannishBoy wrote:

As for the WHS offsite situation, has anybody ever looked at one over VPN? I was thinking I might set a box at my mom's house and have it VPN connected to my house. Then I could possibly run a scheduled sync of important folders across two WHS boxes. I would pre-seed it with all the big files so that it wouldn't be a huge data connection problem and it would only need to move the new stuff over time.

Probably an easier way, but i wouldn't mind getting my mom on a WHS box to back up all of her financial records anyway. I'm thinking I might just build a more powerful box for me with Vail instead of just upgrading my AMD 4830e box, as the Vail upgrade will have support for more media streaming and transcoding to mobile devices. That will require more horsepower I bet.

Check out http://www.homeserversync.co.uk/topten.htm I haven't tried it but it looks promising except I personally feel the cost is too high baecause the link it with 100gb of storage. I would like to find a way to create my own Dropbox that is stored on my WHS not in the cloud so I can use as much space as I want and keep my information more private.

*Legion* wrote:

This thread is for discussing systems for backing up data.

The topic comes up often, and I figured it would be nice to have somewhere to point people.

you are a genius, thanks for putting this together

I've been using Allway Sync as my backup/sync software for over a year now and am completely dependent on it.

What I really like is that it handles backup to Local Drive/Network Drives/FTP and Amazon S3 all in one application, so I can setup a local backup job then duplicate it and change the destination location and bam! I have my offsite backup all ready to go.

There are a couple of annoying things though - one, it's scheduler support is quite limited and relies on Windows Scheduler for more complex calendars (which I loathe) and two, the UI for defining folders to exclude is complete crap.

I still highly recommend it though - plus there's a free version that has no limitations so it's great if your backup needs are not very heavy.

I've been using the Legion backup method since Windows 98 and didn't even know it!!!

I always use a separate, smaller drive for the OS, and don't keep any irreplaceable data on it. I have a second physical hard drive that has my data, and as storage started to get cheaper a decade ago I moved to having 3 physical drives in the box and a scheduled backup job that backups up everything important on the data drive to the third drive. The third drive is also used for storing anything that can be easily replaced via downloading.

I just performed the Legion test this week after putting in a new motherboard and CPU. The only thing I had forgotten was to copy my Firefox profile, but that wouldn't have been a disaster if I'd lost it. Since I was put in a new, larger OS drive as well, I was able to hook up the old one and recover that small bit of data. Yay!

The only thing I'm susceptible to is the box catching on fire and destroying both data drives I guess. But I do a DVD backup of the really, really important stuff every year or so.

Now, if the house burns down.... but dammit I'm not doing off site backups for my home PC. Gotta draw the paranoia line somewhere.

Unknown Soldier wrote:

The only thing I'm susceptible to is the box catching on fire and destroying both data drives I guess. But I do a DVD backup of the really, really important stuff every year or so.

Now, if the house burns down.... but dammit I'm not doing off site backups for my home PC. Gotta draw the paranoia line somewhere.

That's what cloud storage is for. Use JungleDisk and push your important documents (encrypted) into the cloud for safe keeping.

Maybe I'm just hoping it will finally happen so I can kiss all this technology goodbye and go live in a cave in the mountains and eat berries till a bear has me for breakfast.

Or maybe I'm just sick of I.T.

I fail the *Legion* test. I do back-ups to an external drive every month to preserve my photos and music, but I'd like something more consistent for my important word and excel files.

Concerning encryption, it sounds like Truecrypt is the preferred method. How user friendly is it? I'd like to make sure I'm not flashing my financial records all over the cloud when I upload them with one of the services here.

At my work place we are wanting to switch to online back up for our workstations. We currently use a networked NAS to store all our projects and 4 desktops work and save directly to the NAS. My boss would once a week then physically copy that drive to another and take it home. He is sick of this. What we dont need is constant backup of each individual workstation, just some way to copy our network drive to some sort of cloud storage that automates versions etc. We need around 500gigs to 1tb of storage to make this work also. I initially thought run drop box off a workstation but I couldnt find any plans with that much storage. Thoughts/recommendations?

Blotto The Clown wrote:

At my work place we are wanting to switch to online back up for our workstations. We currently use a networked NAS to store all our projects and 4 desktops work and save directly to the NAS. My boss would once a week then physically copy that drive to another and take it home. He is sick of this. What we dont need is constant backup of each individual workstation, just some way to copy our network drive to some sort of cloud storage that automates versions etc. We need around 500gigs to 1tb of storage to make this work also. I initially thought run drop box off a workstation but I couldnt find any plans with that much storage. Thoughts/recommendations?

http://www.crashplan.com/ ?

Didn't work for me, I kinda wanted something to keep all my porn, but it erases them if you erase them from your local PC...

What about Mozy, or the like?

Blotto The Clown wrote:

We need around 500gigs to 1tb of storage to make this work also. I initially thought run drop box off a workstation but I couldnt find any plans with that much storage. Thoughts/recommendations?

We back up about that much to S3 via JungleDisk Server.

JungleDisk compresses (to save on your S3 bill) and does block-level diffs so that only the altered bits get uploaded when something is changed. And it encrypts and decrypts locally, so the only thing that goes out over the wire and gets stored remotely is encrypted blobs, nothing viewable that Amazon could peek at. It does versioning and also has the ability for you to set it to keep deleted files for X days, to always keep at least Y copies of a file, etc., so that your backed up copies won't be immediately clobbered in the event of your NAS getting wiped out and then running the backup job.

JD Server also has a client app that you can run on a workstation to communicate with the daemon running on the server.

Ill look into jungle disk, thanks. With JD I need a physical server? I kinda was hoping that I could use a workstation computer with some software and just point it at the NAS and tell it to back the whole thing up with redundancies. I was hoping to avoid purchasing a new piece of hardware

Blotto The Clown wrote:

Ill look into jungle disk, thanks. With JD I need a physical server?

Nope. You can run JD Desktop, Workgroup Edition or Server Edition from any machine that has access to the NAS mapped as a drive.

Basically, they're all the same core app with some separate features:

* Desktop is the basic app
* Workgroup is designed to let multiple systems share the same account (more for using JD as a network share than backup)
* Server is designed to be able to run headless if desired, and accessed remotely from a separate client app

I would probably recommend Server Edition for the setup you want. You have one machine with JD Server installed and that runs the backups, then the other machines can have the JD Server Management client app and interact with the daemon remotely.

I thought that I'd have passed the *Legion* Backup Test up until last night when I tried to restore my dead PC from a NAS Syncback backup. Turns out that since Win7 refuses to boot, I'm pretty much SOL on any recovery I'd like to do. I installed Win7 on a new HDD last night, and am slowly re-installing everything. But I'm over Syncback, as it turned out to be a false safety net. I want a good image-based backup solution for my OS partition. Any recommendations? I've used BESR at work with good luck, and I'm looking for something similar, but not as costly. Is there anything good out there that's reasonably priced?

Pages