Private Cloud / Backup Solution

I'm currently using the following backup solutions:

* all pictures and video's taken with my Android phones: Google Drive
* Music: all uploaded and synced to Google Music
* for all photo's before this became a thing: DropBox
* Documents and other files: SOS Online Backup
* Oh and I have a 80Gb drive lying around at my parent's place, with all my documents etc up till then

I also have an old Qnap NAS lying around, which I haven't used in years.

Apart from that I also use DropBox to easily access files everywhere, especially (public) work-related files. As my SOS Online Backup subscription will expire at the end of March, and as the price will about double, I'm looking to find another solution. I would love to still be able to access my data anywhere, but would also love to ditch DropBox for privacy reasons.

I saw an advert for Lima last week, and their product seemed to fit the bill perfectly: you connect their device to your router on one end and a hard drive on the other. The data is fully encrypted yet accessible through the Lima app. Unfortunately, Amazon was rife with critical reviews. Apparently Lima didn't fulfill their Kickstarter promises and delivered a buggy, unfinished and inflexible product.

So I'm wondering what Goodjerville is using as backup/archiving/private cloud storage solution? Is there anything out there like Lima that actually works?

Lima works, but the original hardware only had 100Mb ethernet, and the client software still has a higher CPU usage than I'd like. Last I looked it was also not great at preventing identical images from being backed up from multiple devices. MyCloud drives are much faster, I just don't know if they're as pedantic about encrypting every step.

I'm mulling over the design of a Syncthing implementation on my home network. I have the hardware and the drive space, but how to tie it into or replace the services that I've started occupying organically with something that might need some workarounds is a struggle.

So I have a Synology Diskstation NAS that I use at home on my network. It has all kinds of options for accessing things remotely that I haven't even started tapping into yet. I've got apps on my phone to access a cloud setup I can enable, photos, video, audio, etc. I've never used them though so maybe someone here has more experience and can provide some more information.

Asterith wrote:

So I have a Synology Diskstation NAS that I use at home on my network. It has all kinds of options for accessing things remotely that I haven't even started tapping into yet. I've got apps on my phone to access a cloud setup I can enable, photos, video, audio, etc. I've never used them though so maybe someone here has more experience and can provide some more information.

I have Synology Diskstation. I do use my phone to watch videos from it. I have all my media type stuff on it so I can access it from anywhere. One hard drive went bad and it was easy to replace without any lost. This is the only NAS I have ever had so I can say if it is the best option. I love the thing though.

I have my own Qnap NAS that I'm using to do this. There are a lot of options you can use to do this too. Qnap has their own version, so does Plex if you have their premium subscription, OwnCloud, etc.

I'm interested in following up on this, as I've been thinking about getting things backed up myself. Crashplan is the only site I've even looked into so far, but I haven't exactly done a lot of digging. Infrequently backing up my documents to an external WD backup drive is all I do. And relatively random linkages to Google Drive and Dropbox.

Thanks everyone for the suggestions so far, and I'm glad to hear other goodjers are interested too. For some reason, I always feel vaguely guilty when reaching out to the community like this.

Rezzy wrote:

I'm mulling over the design of a Syncthing implementation on my home network. I have the hardware and the drive space, but how to tie it into or replace the services that I've started occupying organically with something that might need some workarounds is a struggle.

It seems like that would require an actual pc being used as the NAS? What kind of specs would that need to have?

I got a newsletter from Lima yesterday, full of box quotes by tech sites boasting their amazing performance with the new Lima Extreme. A bit of googling revealed that these quotes came from articles about the Lima press release about their upgraded device (with more bandwith and better CPU) - all stating that reviews are pending. This did not inspire much confidence in Lima's integrity to be honest.

In general, a more techie solution to this speaks to my inner geek, but a less versatile yet user-friendly solution like WD MyCloud seems oh so tempting as well.

I'll wait for the Lima Extreme reviews to come in probably.

dejanzie wrote:

It seems like that would require an actual pc being used as the NAS? What kind of specs would that need to have?

The person that recommended it to me is running his main repository on a Raspberry Pi, so not much if you're not worried about raw speed. It encrypts traffic, so there is quite a bit of CPU churn involved, but it can run on pretty much any device.
Works similar to BitTorrent in that it syncs peer to peer when available.

What I did was to build my own, which was pretty expensive, when I did it, seven or eight years ago.

How I built mine: I used a Core 2 Duo motherboard with eight gigs of RAM. I wanted an eight-drive RAID6, and couldn't get that many ports on a motherboard back then, so I ended up buying a hardware RAID controller, an Areca 1222. (not made anymore, and not really necessary, these days.) I also bought a pair of 4-in-3 drive enclosures, using a special controller cable that condenses 4 SATA wires down to one larger square one. (This is a standard, but I don't remember what the standard is called...SFF something, but I don't remember what.) Buying two of those cables was quite expensive; they're meant for enterprise use, and cost a lot, like fifty dollars each. Between the bays and the hardware controller, this also gives me hotswap capability. I've used this a few times, over the years.

You could totally skip all that, these days, as many motherboards now come with tons of SATA ports, and Linux software RAID is very fast and very flexible. But when I built mine, this was one of the more reasonable ways to do it. I'm not sure you can do hot swap with mdraid, though. I've never tried, and I suspect the interface is probably pretty painful, if it works. Having the hardware RAID controller has been nice, in that regard. Plus, it's battery-backed, so I don't have to fear power loss, and I can run filesystems in the faster 'nobarrier' mode, which is too risky to do with regular mdraid.

From there, it was just a matter of assembling the RAID6 and installing Linux. I run stock Debian, and it has no trouble installing on an Areca board. It's just a single, large drive, so I can partition it and install without having to think much about it at all.

Doing a Linux install on software RAID with the Debian installer, though, can be kind of intricate. It's MUCH cheaper, and just as fast, but it's a lot harder. It ends up being a multi-step dance of partitioning the drives, building the RAID, activating the RAID, partitioning the new md device, and then either partitioning and installing onto the md device, or using it as an LVM drive, creating LVM devices, and installing onto those. This gets ridiculously intricate. It's way harder than it probably should be; the Debian installer badly needs a rework in this area. But the underlying logic is quite complex; you've got physical drives, and then md on top of that, and then (perhaps) a partioned md, and then either volumes directly on MD or LVM-on-MD and then volumes-on-LVM. With all those layers, it's a hard thing to abstract well.

It's kind of ludicrous, when you look at it fresh. A hardware RAID controller lets you avoid most of that, just presenting a single logical drive to Linux, but they're awful expensive, and if they break, fixing them is spendy. With mdraid, you can read your data on any system with enough SATA ports.

For a long time, I did partial backups of this solution, where the stuff I *really* cared about, I copied to my local workstation. I did this because I couldn't afford eight more drives for backup, and I really couldn't afford a second hardware RAID controller. Once prices on larger drives got cheaper, I set up a 4-drive RAID5 using mdraid on the motherboard ports, exactly the same size as the main volume, and set up a daily scripted backup from one volume to the other.

Overall, this gives me infinite flexibility. It's just a Linux box. But it was expensive to build, and took a lot of effort to get all set up correctly. I actually understand all the software it's running, and I configured it all myself. This is as far from plug-and-play as you could get with modern hardware. The time investment was significant; I'd guess at least thirty hours to design, build, and configure it. Ongoing maintenance is near zero, I just have to run apt update/upgrades once a week or so, but the initial build was a major project.

If you do something like this, a strong suggestion: keep a text file of what software you've installed. As you tweak and add things, you'll forget what's there and what you're actually using. If you decide to rebuild it, a list of software (and the backed-up config and data files) will let you reconstruct the server from scratch very quickly, sometimes just an hour or two. Without that list, you may end up bugfixing for quite awhile, as you notice things that are missing.

At this point, I think this was totally, totally worth the time I invested. It cost a week or so, seven years ago, but I've been using it ever since, and plan to continue. I've really gotten an amazing amount of mileage out of the thing, and can't even imagine not having it.

I know I'm late to the party, but I went a slightly different route to Malor (though in fairness, my timescale is alot shorter than Malor's so I've got the benefit of a few years of technological advancement)

My initial plan was to build a computer to act as a media server for my TV, (since at the time Australia didn't have Netflix) and initally I went down a similar path to Malor buying parts to build a dedicated Linux machine sitting next to the TV.

The most expensive part for me was the 4 harddrives, that I planned to set up as a Raid 5 array. Built it using mdadm (linux software raid) and began copying my library across. Unfortunately there was a failure of a sata communications chain....somewhere....ultimately I think one of the motherboard sata ports had a minor issue, that became a problem when filling the array with my media library, the array lost sync, crashed and required a rebuild of the array, resulting in the loss of the data. Rinse repeat when the array was rebuilt.

I never got far enough to add any sort of cloud support to it......However......

Fast forward a while and I start seeing this thing called a raspberry pi pop up in alot of projects. So naturally I decided it was important I had one to evaluate its potential..... <.< >.> ..........yes........
Anyway, I enjoyed playing with it and reading more about them I realised they had some excellent characteristics for always on servers (low power consumption, low heat generation, etc), so I started reading about cloud programs.

Enter Owncloud. Owncloud is a dropbox clone, it functions exactly like dropbox, even has apps in all the major eco systems with the one major exception that its all open source.

Side note here - there was recently a split in the open source community with some of the developers leaving owncloud and forking it to produce nextcloud. As at the end of 2016 there wasn't a major difference. I have stayed with owncloud for the moment as its what I've already got. Will I switch in the future? If the security/feature set/etc is advantagous sure, but at the moment I have no need to. Therefore where you read Owncloud below, It could just as easily be NextCloud - the choice is yours

Owncloud installation was relatively painless - add the owncloud repository to my sources, update and install. It ran a install wizard of sorts that allowed you to confirm the PHP version you want to use and the SQL implementation you would prefer. Pick those, set up accounts, set up a Dynamic DNS host, configure router and your right to go, you have a dropbox like program you can access anywhere.

You will have noticed that I haven't mentioned raid or encryption with the raspberry pi. Thats because like any linux system, it is fully customisable - it is just another computer (albeit a very small one). So if I want to encrypt it, I have to set it up seperatly (probably a truecrypt derivative), If I want a raid setup, I need to setup a mdadm raid again.

The raspberry pi has one major flaw as a server however - No SATA ports. Everything is done over USB (the bus for which incidentally is shared by the network port (standfast PI3 and its wireless)). What this means is if you want a large server (ie: bigger than flash disks) you need USB to SATA converters and a USB hub. It also means you will not be setting any speed records - remember too the Pi is an ARM processor, so its not terribly fast to begin with.
However, for my usecase (essentially a sandbox) these flaws are not show stoppers. However, I know that for some the lack of speed will be an issue. I suspect it will primerily show up during the building of the array in the first case, and I haven't yet tried it with a raid setup there will definetly be a performance hit when I setup the raid - but time will tell if it is manageable.

Well with the impending stopping of CrashPlan home I have decided to try and get Nextcloud working. This is my first time diving deep with Linux and so far have been having a hard time.

Hardware wise I have an Asus Eee PC with one of the 64 bit dual core Atoms in it that I thought would be nice as it has a battery, screen, etc. and is small and rather power efficient. My plan is to set it up with Nextcloud and have it use my Networked Windows serve as the file location.

I figured I should use Ubuntu Server 16.04 LTS as I don't want to fiddle with updates anymore than I have to. I can get Linux installed just fine and can do the Snap installation of Nextcloud but can't get into the setup for Nextcloud.

Sadly Linux has been something I wanted to learn how to use but just haven't had the time or need until now. I'm planning to keep trying to get it to work but if any of the Linux skilled goodjers out there would be up for a little guidance/troubleshooting with me I would be very grateful.

Can anyone suggest an online backup option comparable to Crashplan? I've tried Carbonite and had a few issues with it so i'm looking for something else. Anyone have experience with IDrive?

Backblaze is one of the big alternatives that a lot of crashplan users moved to. I have used it for a few months now and I know some other GWJers do too. If you are interested in it let me know and I can send you a referral code that gives three months free.

EDIT: Here is a referral link.

I have used BackBlaze and it's great so far. Have not had to do more than a test recovery.

The cloud is actual a demon. The demon can change into a gaseous state that is cloud like. This cloud wears a crown, It thinks of itself as king. Heavy is the head that wears the crown.

Seems a topic worth following. So many digital pictures and videos now, my wife would kill me if we lost them all. Need to do something.

Yes, that is why signed up for Backblaze. We needed some type of off-site or cloud storage to ensure we would not lose all of our pictures and videos in a fire or something, especially now that we have a kid.

Between Dropbox, Google Photos I have all my important documents, photos, and Videos backed up. Is there a need for an additional specialty cloud backup product? Neither of those companies is going anywhere soon as well.

Only if you want encrypted storage or some of the specialized backup services, like sending you a drive with all your files on it, to buy or return within a month. Well, they also do trickle backups; you don't have to manually upload them.

I would not put anything in Dropbox or Google Photos that I would not be okay sharing with the world.

Robear wrote:

Only if you want encrypted storage or some of the specialized backup services, like sending you a drive with all your files on it, to buy or return within a month. Well, they also do trickle backups; you don't have to manually upload them.

I would not put anything in Dropbox or Google Photos that I would not be okay sharing with the world.

I just assume anything stored in a cloud service regardless of brand is out in the wild.

This is a really interesting thread. I'm coming at it from the perspective of wanting to figure out a personal digital preservation solution for my family's photos/videos/records as well as setting up a backup for personal files. I've been working in digital preservation in the The National Records of Scotland for a few months now but I have yet to really kick off something at home.

A rule of thumb I've been told is to keep three copies of a digital object, across at least two different stoarge mediums, stored in at least two different geographic locations. That together with a good naming convention and strict selection process is a solid start and a good way in case of catastrophic loss of data.

I'm mainly wanting to digitise and preserve my parent's gargantuan family photo collection accumulated and be able to pass them along with a family tree down to future generations. I need to identify the people in the photos because otherwise I'll albums full of people I don't recognise. We're at an interesting point in time where "everything going digital" is actually quite a fragile position for these things.

Here's some links to resources for getting started in digital preservation that I'll be using. I'm probably going to blog my experiences as I go (another record for the archive ): The National Archives' How to Get Started page, The Library of Congress' Personal Digital Archiving, Digital Preservation Coalition's Personal Digital Archiving Tech Report.

It's the 3-2-1 strategy for backing up.

How I can never forget it...

Has anyone here tried ownCloud or Tonido? I was thinking about setting it up for my family to use (since they are too cheap/pigheaded to start using Google Photos or iCloud for their photos/videos). I am going to have quite a few TB's of space on my new file server so I figure I could spare them a few so that they would at least have a basic backup. I may end up signing up for BackBlaze as well but I am worried about what my ISP would do if I tried to transfer 20 TBs of stuff to them.

I'm experimenting with the NextCloudPi image on a Raspberry Pi 3. It's not bad. The hardest part was using letsencrypt when those ports aren't forwarded to the Pi at all.

I setup a NextCloud on an unused netbook as the server and have mapped out my important networked server folders to it. I then setup another netbook with an external hdd at my dads that has those folders synced. Works great so far. I would like a better way to know that it has backed up to the offsite pc besides remoting in and checking every now and then but it is working flawlessly so far.

BTW: I started backing my 4tb of personal photos and video to backblaze back around Christmas and still have ~.75tb to go. I have it wide open on my end but they will only take ~6Mbits/sec. So far I like the service it is cheaper and lighter then Crashplan was.