[slackline]

Get a NAS (Network Add-on Storage).  If you're really bothered you can get one with multiple HD's and set them up in a RAID (Redundant Array of Inexpensive Disks).  Alternatively you could just set up RAID within a desktop.

If you go with a NAS, then if you're using Macs you can setup a cron and schedule an rsync job to backup your files (very efficient way of doing incremental backups as after the first sync it only copies the bits of files that have changed).

Lots of options out there, it has the added bonus that NAS's tend to be packed with features as they are running embedded GNU/Linux distributions (as is your Western Digital MyBook btw, and you can hack it to have this functionality too).  This means you can have uPnP servers (which make your music & video available to things like PS3 and Xbox), run web-servers (and optionally wiki/forum software), general file server so you can mount them as network drives from your Macs/other computers and save data there directly and loads of other things (e.g. set up your own mp3 streaming radio on your homenetwork).

I use a rather dated Linksys NSLU2 with a couple of USB drives attached, but I'd recommend QNAP NAS's as they seem to be the most feature rich in terms of the software they imlement.

[Bubba]

Echoing Slacklines advice, definitely go the NAS route - I've got a little Icybox 4220 running 2 x 1 TB drives in it. It uses hardly any power, runs Linux so no defragging issues, and can act as a bittorrent client negating the need to leave a computer on.

I didn't go the mirrored RAID route, but I will just ensure I replace the drives every 3 years instead (it's an enclosure, not an external drive). This is a riskier policy but if a drive doesn't fail shortly after it's new then it's not that likely to fail within a few years.

If you want to splash some cash, then some of the high-end offerings from Raidsonic and other manufacturers are amazing - you could easily setup a huge amount of RAID mirrored storage.

I would also go for an enclosure rather than a NAS with a fixed built in drive. That way you can upgrade/replace drives easily as higher capacities become more economical - 2TB drives are now almost the same price as the 1TB drives I bought when I got my NAS.

[Bubba]

Yes you could have 2 drives mirrored as one, but I was greedy for the storage space

Yes, it needs to be cabled to the router, and it works no problem with my old 100mb cards, and my wireless 54g - I guess the router handles all that stuff.

[slackline]

Yes, it needs to be cabled to the router, and it works no problem with my old 100mb cards, and my wireless 54g - I guess the router handles all that stuff.

  As long as its connected to your router, it doesn't really matter what speed connection, it should work (although obviously the max speed will be limited by the slowest connection, so if you've a 100Mbps connection between your router and NAS, but you're using a laptop which only has 802.11g support then it will be limited by the wireless connection which has a theoretical max of 54Mbps).

[slackline]

NAS's are great, really lower power solution to having your media available to any computer on your network all the time.

[slackline]

 

Let me know if you've questions about rsync/cron.

[Johnny Brown]

Bummer. Sounds like there might be some kind of conflict?

Interested in this NAS think myself... as its via a router, is it possible to set it up for access through the net?

[slackline]

Bummer. Sounds like there might be some kind of conflict?

Yeah, its called "Macintosh" 

Interested in this NAS think myself... as its via a router, is it possible to set it up for access through the net?

Yes, you just set your router to port-forward to the NAS.  For example I have my NAS running a wiki that I use primarily to keep notes of computing shit I do.  As I do a lot of stuff at work too I want to be able to edit the wiki from work.  The standard port for http is port 80, so I set the router to forward requests on this port to the NAS, and hey presto.  If you install something like Transmission on the NAS for torrents then it sits there headless (i.e. without a graphical interface), but you manage everything via a WebUI, so you just point your browser at the relevant web-page and port (the built in daemon runs on port 9091) and you can manage stuff from any computer on your homenetwork.  If you want to manage them from anywhere on t'net then set your router to port-forward requests on 9091 to your NAS.  Similarly I've set up my own mp3 streaming at home, but have port-forwarded requests on the relevant port (can't remember which for this one) and can listen to my music streamed from my NAS whilst sat at work.

[Johnny Brown]

You had me at 'Yes'. The rest went over my head, but accessing your music from work sounds perfect. Haven't got a pot to piss in at the mo, but when I have I'll be getting this set up. Ta.

[Control freak]

So got a Netgear ReadyNAS Duo today with a couple of 2TB disks, plugged them in, booted it up, set it up as RAID1, got it talking to the network no problem ... then immediately broke it by misguidedly letting the Mac Time Machine utility try to use it. It's now grumpily rebuilding its volumes over 10 hours ... will man up and try to get rsync working instead once it's up again..

Ive had one of these for a while now - whilst its pretty good it aint perfect. File transfer times can be very slow, expecially via the inbuilt USB port. If you have the DLNA set to auto index then youll find transfers fail after a period of time. Theres a few other teething issues I had too (cant remember now) but all in all its pretty good, The remote client is good too which gives you access over the internet without any poking around with the router.

[slackline]

Ive had one of these for a while now - whilst its pretty good it aint perfect. File transfer times can be very slow, expecially via the inbuilt USB port.

But its a NAS, why are you using USB port?  (Guess you could be backing up an external USB drive to the NAS maybe?).

If you have the DLNA set to auto index then youll find transfers fail after a period of time.

Only an issue if you're using uPnP/DLNA, and I'd be that failures are a symptom of trying to transfer whilst an update to the index is occurring.  99% certain you'll be able to configure the frequency of auto-indexing (you can in Mediatomb anyway, would also depend if the software supports inotify functionality).

Its great that Linux is being used so widely these days 

[slackline]

Then the machine locks up, has to be forcibly rebooted and takes 10 hours to resync its two disks 

Thats about right I'm afraid for syncing two RAID1 disks.  I had a problem with one of my 1TB drives (x2 in a RAID1 setup) the other day and it took about 4-5 hours to sync across.  If this is an issue, you could potentially remove one of the discs, transfer/back everything up, then drop the disk in and let it sit there for 10hours syncing.

[slackline]

What baffles me is that it has nothing (yet) to sync ...

But the software that runs RAID1 doesn't know that, the discs are identical and all its doing is sitting there and going through every sector and checking that they're the same on the two drives.

[slackline]

As I understand it, rysync is tolerant of dropped transfers so perhaps will have to be using that until I have a better router. Slackers: do you initiate your rsync jobs from the client (which looks simpler) or from the NAS itself (the ReadyNAS in theory has some utility to do that but I am confident it won't work ...)?

Its not so much that rsync is tolerant to dropped transfers, its just that its different that saying "copy all of this directory" because files that do make it across won't be copied again (unless they've been modified) when you re-issue the command.

I initiate rsync from both my client (desktop/laptop) and my NAS, but thats because I ssh into the NAS and mess around from there (its no different to doing anything from a terminal on any of my other computers).

No idea what or how ReadyNAS works, but if you can enable ssh on the NAS and setup a user account (its a bad idea to do everything as root) then once thats all done something like the following will back up your $HOME directory on your mac to the users account on the NAS (obviously it can be adjusted to reflect any partitioning scheme and so forth that you have chosen for the NAS)...

Code: [Select]

rsync -av ~/* [nas host name|ip address]:~/.

If you're transferring lots of files some benefit can be gained from including the '-z' flag which enables compression, but its pretty pointless for JPEG/OGG/MP3/AVI since these are compressed formats anyway.

If you don't want to mess around with the CLI you might consider installing grsync which is a GTK+ GUI for rsync and should be easy to install on a Mac.  How you go about doing this is a matter of choice, and I don't know masses about installing additional software on OSX, but I do know that Gentoo's portage package management system can be used to install packages (and their dependencies) from source under OSX without causing any conflicts to the main system.  See here to get started.

[Control freak]

Has anyone used / seen of of these babies G Safe by Hitachi ?

[underground]

Just bought a Readynas duo from ebuyer - 200 quid which included a preinstalled 1TB Seagate barracuda drive - plus until september ish, a downloadable form off the netgear site plus an invoice and the barcode off the box gets a free 'nother 1TB drive.

[slackline]

Its probably this one. Most NAS's are cabled to connect to networks, although clients (i.e. laptops) can connect to wireless home networks and still mount the network file systems.  It should be agnostic to the OS on the clients.

[slackline]

  I linked to indicate that its wired (I'm yet to come across wireless NAS's, although I'm sure there are some already out there on the market).

What problems are you still having?

If its syncing a laptop connected wirelessly to your network can you sync ok with a wired connection (this would suggest that your wireless network connection from your laptop to your wireless router is dodgy and drops connections).

Hundreds of layers and points at where things can go wrong, process of elimination to go through them all.

Any logs from the NAS itself (you'll likely have to ssh into it and look in /var/log/ for something relevant)?

[underground]

It is that one slackers linked to. Wired to an Ethernet socket on my router. A win7 machine connects to this also by wire, and a netbook running xp by wireless sees it fine too, the shares all just appeared by magic.

Transfers are slow- apparently a gig of RAM can help

[slackline]

It is that one slackers linked to. Wired to an Ethernet socket on my router. A win7 machine connects to this also by wire, and a netbook running xp by wireless sees it fine too, the shares all just appeared by magic.

Transfers are slow- apparently a gig of RAM can help

Why would more RAM increase the speed of transfers?  The speed at which RAM communicates with other components on the motherboard/within the NAS is way, way, way quicker than your wired or wireless network will ever reach.  Start with looking at improving the network connections first.  The spec of the NAS indicates that it has gigabit ethernet which is pretty fast, but does your router support this (what model is it, check out the spec)?  On the wireless front do you have an 802.11b, 802.11g or 802.11n on the router?  What spec is the wireless card on the laptop and is it actually using the top spec (e.g. could be capable of all three but for some reason n might not be working so its fallen back to g which is half the speed)?

One area where RAM might make a slight difference is if you're syncing a lot of very small files as comparing long lists between server and host would take a lot of time.

More RAM will come in handy if you want to do things like using it for multiple functions such as headless torrent client (check out Transmission), use it as a uPnP server for PS3/Xbox/etc. possibly with transcoding on the fly (although I'd have reservations if the CPU is powerful enough for this).

[dobbin]

RAM definately has an impact on the speed of transferring huge files around.

THe IP stack chops the file into packets which are held in memory until the ack comes back, plus in a GigE situation it'll pad the frames if it doesnt fill them. There is memory overhead in managing large transfers down the NIC, having seen this in practice at work I can vouch for it.

You are also right that any filesystem comparison stuff going on above that level will have additional mammary needs. (i have mammary needs)

Consider also that for GigE to work you need either Cat5e or Cat6 cabling.

[slackline]

One of the reasons I advocate rsync is its smart way of syncing only bits of files that have changed after the initial mirroring/syncing.

Good point about Cat5e had to buy some new cables to get some Belkin Gigabit Powerline adapters to play properly (now if only I could get my fucking PS3 to get a wired connection  )

[Control freak]

What problems are you still having?

I am more or less sorted now. The key problem was trying to use it with my wifi network ... lots of dropped transfers. Also trying to use it with Mac utilities like Time Machine. rsync backups over ethernet work OK.

Have a play around with the packet sizes in the ReadyNas config and make sure jumbo frames are disabled unless your router supports them. You can also try playing with the read/write caching. Ive got mine running to my router via a pair of power line ethernet adapters and from there to my PS3 over wireless. Even with all this I can easily stream divx dvds compressed to about 900mb

Make sure also that auto indexing is turned off on your media shares as this will cause drop outs when transferring files

[slackline]

What problems are you still having?

I am more or less sorted now. The key problem was trying to use it with my wifi network ... lots of dropped transfers. Also trying to use it with Mac utilities like Time Machine. rsync backups over ethernet work OK.

All good then, and sounds like the problems don't stem from the NAS, but more network & Mac issues (although I've no idea/experience of the vagaries of Time Machine to comment in detail).

[underground]

It is that one slackers linked to. Wired to an Ethernet socket on my router. A win7 machine connects to this also by wire, and a netbook running xp by wireless sees it fine too, the shares all just appeared by magic.

Transfers are slow- apparently a gig of RAM can help

Why would more RAM increase the speed of transfers?  The speed at which RAM communicates with other components on the motherboard/within the NAS is way, way, way quicker than your wired or wireless network will ever reach.  Start with looking at improving the network connections first.  The spec of the NAS indicates that it has gigabit ethernet which is pretty fast, but does your router support this (what model is it, check out the spec)?  On the wireless front do you have an 802.11b, 802.11g or 802.11n on the router?  What spec is the wireless card on the laptop and is it actually using the top spec (e.g. could be capable of all three but for some reason n might not be working so its fallen back to g which is half the speed)?

One area where RAM might make a slight difference is if you're syncing a lot of very small files as comparing long lists between server and host would take a lot of time.

More RAM will come in handy if you want to do things like using it for multiple functions such as headless torrent client (check out Transmission), use it as a uPnP server for PS3/Xbox/etc. possibly with transcoding on the fly (although I'd have reservations if the CPU is powerful enough for this).

Well, it said so on the internet - that's the honest answer! I have a spare 1Gb module out of my netbook but apparently it will only work with DDR1, and I think my stick is DDR2. Anyway, something to do with 'swapping' - presumably bigger RAM = more caching, but I'm really flying by the seat of my pants with this thing. As far as it being just a hard drive, fine, but e.g. I just downloaded Transmission, it came as a .bin file, and I have no idea how to install it - don't know even how to access this linux underlying os in the thing....