The Data Backup Thread (& request for more suggestions)

I need to adjust my backup strategy and am looking for fresh recommendations.

I’d been using Windows File History —> Synology NAS —> Backblaze B2. Then my wife got a Mac, and Synology’s Time Machine support is not good.

Should she just get a personal Backblaze account? Or should we look at something else? We’ve got the NAS, the B2 storage, and OneDrive already available as destination options.

It’d be convenient to have the same cross-platform software running on her laptop, my laptop, and our gaming PC, but that’s not a must.

There's almost certainly someone better qualified on here who can give you a better answer. But for what it's worth, I send my client PCs to the Synology via Veeam, which has a free agent and I believe is also available on Mac. Once on the Synology we go to Backblaze from there. That way you won't need a separate Backblaze account for her.

(trying to move this conversation over to here from the "Finished Any Games" thread)

Vrikk wrote:
Jonman wrote:
Budo wrote:

Meh, I was close to getting a new one anyway. The annoying part is the cash I have to drop for data recovery on the hard drive.

You're really selling that $50 I give to Backblaze annually.

Wait, what? Why is mine $70? Is it because I back up an external hard drive to Backblaze instead of my internal SSD?

(Backblaze is great, btw. Everyone should have it. It's dirt cheap and wonderful.)

What's the advantage of Backblaze, over OneDrive's 1TB of cloud storage that I get with Office 365?

merphle wrote:

What's the advantage of Backblaze, over OneDrive's 1TB of cloud storage that I get with Office 365?

Well, not being limited to 1TB for starters.

*Legion* wrote:
merphle wrote:

What's the advantage of Backblaze, over OneDrive's 1TB of cloud storage that I get with Office 365?

Well, not being limited to 1TB for starters.

Yeah, fair enough. I don't think I'm anywhere near that limit, though I'm only backing up documents, photos, game-saves - stuff that's in my home dir, basically.

1TB ought to be enough for anybody.

merphle wrote:
*Legion* wrote:
merphle wrote:

What's the advantage of Backblaze, over OneDrive's 1TB of cloud storage that I get with Office 365?

Well, not being limited to 1TB for starters.

Yeah, fair enough. I don't think I'm anywhere near that limit, though I'm only backing up documents, photos, game-saves - stuff that's in my home dir, basically.

1TB ought to be enough for anybody. ;)

1TB wouldn't cover half my furry porn.

merphle wrote:
*Legion* wrote:
merphle wrote:

What's the advantage of Backblaze, over OneDrive's 1TB of cloud storage that I get with Office 365?

Well, not being limited to 1TB for starters.

Yeah, fair enough. I don't think I'm anywhere near that limit, though I'm only backing up documents, photos, game-saves - stuff that's in my home dir, basically.

1TB ought to be enough for anybody. ;)

OneDrive is a perfectly adequate solution for low-volume backups. It's what my parents started using after CrashPlan killed off CrashPlan Home, and I've seen no reason to nudge them towards a different solution. Granted, OneDrive is more of a "cloud drive with backup functionality cobbled in" rather than a true comprehensive backup offering, but for a personal end-user system, it works.

If you needed your backups pre-encrypted with a key that the remote provider doesn't have access to, that's the sort of functionality you'd need a service like Backblaze for. For businesses there's obviously value in that. For end users, if you don't mind that Microsoft technically could load up your Baldur's Gate savegame and see how trash your party is, you can probably forego it.

*Legion* wrote:

If you needed your backups pre-encrypted with a key that the remote provider doesn't have access to, that's the sort of functionality you'd need a service like Backblaze for. For businesses there's obviously value in that. For end users, if you don't mind that Microsoft technically could load up your Baldur's Gate savegame and see how trash your party is, you can probably forego it.

As it turns out, I already have a scheduled task that emails my BG2 savegame to BillG himself every week so he can laugh at my expense, so this isn't too much of a concern for me. Thank you!

I need to do something off-site I guess.

I've got an external drive in a fire proof safe with general everything backed up.

I've got Google photos and Amazon prime photos for most of the images from my wife and my cameras. But Amazon doesn't back up videos, and her Google account is maxed since she doesn't use a pixel. So I need something else for videos of the kids mostly.

I feel like steam cloud has most of my save games. But maybe one drive would be simple enough for that and other games not in steam.

I've been using Duplicati but I hate it. It's slow as hell and the UI is completely web-based.

What are my other options for local backup? I'm not looking for cloud backups at this time.

Stele wrote:

I need to do something off-site I guess.

I've got an external drive in a fire proof safe with general everything backed up.

I've got Google photos and Amazon prime photos for most of the images from my wife and my cameras. But Amazon doesn't back up videos, and her Google account is maxed since she doesn't use a pixel. So I need something else for videos of the kids mostly.

I feel like steam cloud has most of my save games. But maybe one drive would be simple enough for that and other games not in steam.

We use OneDrive since I already have that O365 family license for our phones and that's a ton of storage you get with that price. Also use Google Photos, but it's more compressed.

I use One Drive also for storage. It works great and is integrated into Windows fairly well. I believe they just added more options and tiers for storage if you don't have Office 365.

I've been using Veeam and it's alright. Use it at work so I figured why not at home. For what I'm doing it's free (personal use at home to backup to a PC/NAS - no need for support).

Their Mac, Linux, and Windows agents are good. I wouldn't say so for their AIX and Solaris agents, but who uses those at home?

Are you on Windows? How are you storing your backups? That might help folks narrow down suggestions. If you're backing up to a NAS, pre-built options like QNAP have their own tools for backups.

Duplicati shouldn't be slow after you take the initial backup, should it? Unless hundreds of gigabytes of files have changed every time you run it. I have jobs scheduled for every day at 3am and it takes minutes even though my backup is over 300gb.

If it's too slow in your use-case, fair enough. Or if you just don't want to use a web UI. But as an example I checked my most recent backups (I run one job to backup to my NAS and one to backup to an external hard drive) and they took a minute and 36 seconds and a minute and 21 seconds, respectively. And I work remotely/use my PC every day, so while I'm not editing huge amounts of data, files ARE changing every day.

I tried giving Windows File History and Windows Backup a try and both proved really unreliable. Even if they initially seemed to work, if I tried to restore a backup as a test a few days later, there's always some error. Maybe it can't find the backups even though I'm staring right at them. Maybe it has some uselessly generic error like the backup could not be completed.

The common suggestions for fixing the errors are pretty useless, like disconnecting all USB devices (I mean, I'm backing up to a USB drive, so... no deal.) Or reformatting and starting again which if you have to do constantly, kind of defeats the purpose.

So I kind of end up going back to Duplicati even after I try other things. The few times I've had to restore a backup, it's been reliable so far (knock on wood).

I'd be curious about alternative solutions too, but I think a lot of the best options are paid. Paying for reliable backups can be worth it, but I do worry a bit about having my data locked behind proprietary software.

Crashplan can make local backups. But I'm not sure if it's like the one Lou recommended, where local backups are free and you only pay for cloud backup functionality.

Duplicati isn't perfect, but the alternatives I've tried haven't been drastically better enough for me to want to pay for them long-term (I did like Crashplan while I was using it, though). And being able to export/restore your backup profiles and settings has been really useful for when I want to set it up on another device. Sure, there's a learning curve for the initial setup, but then you can export your settings and 90% of the work is done if you ever need to setup another backup.

My last successful backup with Duplicati took just shy of 4 hours. I'm backing up to a dedicated hard drive, since my main concern is losing data to drive failure. So there's no network latency involved. The folders I'm backing up are primarily under my Windows User folder. Total size of the folders I'm backing up is about 140 GB.

Oh wow, that sucks. And that's not the initial backup? My backup is over twice that size. I'm backing up to an external HDD (non-SSD) and a WebDAV folder, and combined they take under 3 minutes.

Have you checked the logs to see if there's any errors, like it getting hung up on a directory for permission issues?

It might also be worth asking for help in the Duplicati forums while you search for an alternative, because that does seem abnormally long.

There's also a "Live" version of the log you could keep an eye on as the backup runs.

Right, that 4 hours was just for overnight updates. I've seen warnings in the logs about accessing locked/cache files, which I've never given thought to because they're a pretty normal thing not to be able to access. I'll check the logs again when it's done repairing the DB (my old storage drive ran out of space).

General stats from last night's backup:

Time
Start 2023-01-19 05:00:00
End 2023-01-19 09:14:47
Duration 04:14:48

Source
Files Examined 372825 (109.05 GB)
Opened 246327 (70.67 GB)
Added 35900 (3.99 GB)
Modified 3835 (1.58 GB)
Deleted 25495

This full log has the most baffling info in the messages:
"2023-01-19 05:00:00 -05 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started",
"2023-01-19 08:35:14 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()",
"2023-01-19 08:35:21 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (21.46 KB)",

It seems to do nothing for a full 3 and a half hours.

Spoiler:

{
"DeletedFiles": 25495,
"DeletedFolders": 656,
"ModifiedFiles": 3835,
"ExaminedFiles": 372825,
"OpenedFiles": 246327,
"AddedFiles": 35900,
"SizeOfModifiedFiles": 1695067113,
"SizeOfAddedFiles": 4289187847,
"SizeOfExaminedFiles": 117089963460,
"SizeOfOpenedFiles": 75880974659,
"NotProcessedFiles": 0,
"AddedFolders": 2912,
"TooLargeFiles": 0,
"FilesWithError": 0,
"ModifiedFolders": 0,
"ModifiedSymlinks": 0,
"AddedSymlinks": 0,
"DeletedSymlinks": 0,
"PartialBackup": false,
"Dryrun": false,
"MainOperation": "Backup",
"CompactResults": null,
"VacuumResults": null,
"DeleteResults": null,
"RepairResults": null,
"TestResults": {
"MainOperation": "Test",
"VerificationsActualLength": 3,
"Verifications": [
{
"Key": "duplicati-20220210T021902Z.dlist.zip.aes",
"Value": []
},
{
"Key": "duplicati-i30e13369384c4ccf8ee331d2fbc42ef8.dindex.zip.aes",
"Value": []
},
{
"Key": "duplicati-bb284bdb67a4c492ab32bf1507078bb8f.dblock.zip.aes",
"Value": []
}
],
"ParsedResult": "Success",
"Version": "2.0.6.104 (2.0.6.104_canary_2022-06-15)",
"EndTime": "2023-01-19T14:14:47.9192251Z",
"BeginTime": "2023-01-19T14:14:47.2484375Z",
"Duration": "00:00:00.6707876",
"MessagesActualLength": 0,
"WarningsActualLength": 0,
"ErrorsActualLength": 0,
"Messages": null,
"Warnings": null,
"Errors": null,
"BackendStatistics": {
"RemoteCalls": 84,
"BytesUploaded": 2078209043,
"BytesDownloaded": 79628631,
"FilesUploaded": 79,
"FilesDownloaded": 3,
"FilesDeleted": 0,
"FoldersCreated": 0,
"RetryAttempts": 0,
"UnknownFileSize": 0,
"UnknownFileCount": 0,
"KnownFileCount": 22055,
"KnownFileSize": 571349259755,
"LastBackupDate": "2023-01-19T05:00:00-05:00",
"BackupListCount": 456,
"TotalQuotaSpace": 2000381014016,
"FreeQuotaSpace": 549141647360,
"AssignedQuotaSpace": -1,
"ReportedQuotaError": false,
"ReportedQuotaWarning": false,
"MainOperation": "Backup",
"ParsedResult": "Success",
"Version": "2.0.6.104 (2.0.6.104_canary_2022-06-15)",
"EndTime": "0001-01-01T00:00:00",
"BeginTime": "2023-01-19T10:00:00.0796445Z",
"Duration": "00:00:00",
"MessagesActualLength": 0,
"WarningsActualLength": 0,
"ErrorsActualLength": 0,
"Messages": null,
"Warnings": null,
"Errors": null
}
},
"ParsedResult": "Warning",
"Version": "2.0.6.104 (2.0.6.104_canary_2022-06-15)",
"EndTime": "2023-01-19T14:14:47.9632297Z",
"BeginTime": "2023-01-19T10:00:00.0796445Z",
"Duration": "04:14:47.8835852",
"MessagesActualLength": 169,
"WarningsActualLength": 148,
"ErrorsActualLength": 0,
"Messages": [
"2023-01-19 05:00:00 -05 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started",
"2023-01-19 08:35:14 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()",
"2023-01-19 08:35:21 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (21.46 KB)",
"2023-01-19 08:43:38 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bfd37deec183041b8ae8349df046fd7ee.dblock.zip.aes (49.91 MB)",
"2023-01-19 08:43:38 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b1f61b35b2bfe4731a9366256c14d1283.dblock.zip.aes (49.91 MB)",
"2023-01-19 08:43:44 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bfd37deec183041b8ae8349df046fd7ee.dblock.zip.aes (49.91 MB)",
"2023-01-19 08:43:45 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b1f61b35b2bfe4731a9366256c14d1283.dblock.zip.aes (49.91 MB)",
"2023-01-19 08:43:45 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i5809f73e89bb412eb9b566fb1cf7505c.dindex.zip.aes (215.95 KB)",
"2023-01-19 08:43:45 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i5809f73e89bb412eb9b566fb1cf7505c.dindex.zip.aes (215.95 KB)",
"2023-01-19 08:43:45 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i1e8d926f7c5a461a82c5e92aa77949fe.dindex.zip.aes (194.73 KB)",
"2023-01-19 08:43:45 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i1e8d926f7c5a461a82c5e92aa77949fe.dindex.zip.aes (194.73 KB)",
"2023-01-19 08:44:17 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b8f7ba24b5e8a467ea7c0c3359dfcb76e.dblock.zip.aes (49.94 MB)",
"2023-01-19 08:44:17 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-beae0c96af86e4ab3bc9fc031f747576c.dblock.zip.aes (49.96 MB)",
"2023-01-19 08:44:18 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b8f7ba24b5e8a467ea7c0c3359dfcb76e.dblock.zip.aes (49.94 MB)",
"2023-01-19 08:44:18 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-beae0c96af86e4ab3bc9fc031f747576c.dblock.zip.aes (49.96 MB)",
"2023-01-19 08:44:18 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i860cd93f604f4fd186fc89a062fa9b85.dindex.zip.aes (204.00 KB)",
"2023-01-19 08:44:18 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i860cd93f604f4fd186fc89a062fa9b85.dindex.zip.aes (204.00 KB)",
"2023-01-19 08:44:18 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ia92f5e2977af49a0bc95111113367736.dindex.zip.aes (205.23 KB)",
"2023-01-19 08:44:18 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ia92f5e2977af49a0bc95111113367736.dindex.zip.aes (205.23 KB)",
"2023-01-19 08:54:45 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-be0272f2f99a944fbbd10fc95d5c7cadb.dblock.zip.aes (49.94 MB)"
],
"Warnings": [
"2023-01-19 08:37:41 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\\Users\\stone\\AppData\\Local\\ElevatedDiagnostics\\",
"2023-01-19 08:37:41 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\\Users\\stone\\AppData\\Local\\ElevatedDiagnostics\\",
"2023-01-19 08:44:02 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Roaming\\Slack\\Local Storage\\leveldb\\LOCK",
"2023-01-19 08:44:03 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Roaming\\Slack\\IndexedDB\\https_app.slack.com_0.indexeddb.leveldb\\LOCK",
"2023-01-19 08:44:03 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Roaming\\Slack\\DawnCache\\data_0",
"2023-01-19 08:44:03 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Roaming\\Slack\\DawnCache\\data_1",
"2023-01-19 08:44:03 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Roaming\\Slack\\DawnCache\\data_2",
"2023-01-19 08:44:03 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Roaming\\Slack\\DawnCache\\data_3",
"2023-01-19 08:44:03 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Roaming\\Slack\\DawnCache\\index",
"2023-01-19 08:44:17 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Roaming\\Mozilla\\Firefox\\Profiles\\dqoibg6w.Stone\\parent.lock",
"2023-01-19 08:54:41 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Local\\Steam\\htmlcache\\LOCK",
"2023-01-19 08:54:41 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Local\\Steam\\htmlcache\\Session Storage\\LOCK",
"2023-01-19 08:54:41 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Local\\Steam\\htmlcache\\Local Storage\\leveldb\\LOCK",
"2023-01-19 08:54:41 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Local\\Steam\\htmlcache\\GPUCache\\data_0",
"2023-01-19 08:54:41 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Local\\Steam\\htmlcache\\GPUCache\\data_1",
"2023-01-19 08:54:41 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Local\\Steam\\htmlcache\\GPUCache\\data_2",
"2023-01-19 08:54:41 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Local\\Steam\\htmlcache\\GPUCache\\data_3",
"2023-01-19 08:54:41 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Local\\Steam\\htmlcache\\GPUCache\\index",
"2023-01-19 08:54:53 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Local\\Steam\\htmlcache\\Cache\\data_0",
"2023-01-19 08:54:53 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\stone\\AppData\\Local\\Steam\\htmlcache\\Cache\\data_1"
],
"Errors": [],
"BackendStatistics": {
"RemoteCalls": 84,
"BytesUploaded": 2078209043,
"BytesDownloaded": 79628631,
"FilesUploaded": 79,
"FilesDownloaded": 3,
"FilesDeleted": 0,
"FoldersCreated": 0,
"RetryAttempts": 0,
"UnknownFileSize": 0,
"UnknownFileCount": 0,
"KnownFileCount": 22055,
"KnownFileSize": 571349259755,
"LastBackupDate": "2023-01-19T05:00:00-05:00",
"BackupListCount": 456,
"TotalQuotaSpace": 2000381014016,
"FreeQuotaSpace": 549141647360,
"AssignedQuotaSpace": -1,
"ReportedQuotaError": false,
"ReportedQuotaWarning": false,
"MainOperation": "Backup",
"ParsedResult": "Success",
"Version": "2.0.6.104 (2.0.6.104_canary_2022-06-15)",
"EndTime": "0001-01-01T00:00:00",
"BeginTime": "2023-01-19T10:00:00.0796445Z",
"Duration": "00:00:00",
"MessagesActualLength": 0,
"WarningsActualLength": 0,
"ErrorsActualLength": 0,
"Messages": null,
"Warnings": null,
"Errors": null
}
}

Today I discovered that someone maintains a Docker image that runs Backblaze in WINE, and lets you mount your NAS storage to look like a physically attached drive, so that the Backblaze client will back it up.

And the people at selfhosters.net added a config to the Unraid community apps browser for easy install as an Unraid Docker app.

So... the app is now working its way through 14TB of uploads on my Unraid server.