Sunday, April 28, 2013

Home Networking for Enthusiasts

As summer approaches, many people are thinking about outside projects and planning for holidays. As an IT geek, I'm planning a more professional and capable home networking setup.

I live in a 160+ year old farmhouse. Electrical wiring upgrades have occurred as needed since the house was first fitted for electricity by my grandparents in the 1940s. When ownership passed to me in 2007, I couldn't even get insurance without upgrading the old fuse boxes (separate upstairs and downstairs apartments) to modern breaker boxes. I'd think they'd be more concerned about old wiring. Thankfully, it's all post-1940.

My MPOE (telco, cable Internet, and power) is in the southwest corner of the basement. The basement itself has a poured concrete floor, mostly reinforced walls (via concrete block) and both granite and old clay brick features. The "ceiling" is mostly less than 6' with fixtures that extend below that level. This is an old "working" basement rather than a finished one. There is a drain at the south wall and up to a half inch of water on the floor during a wet spring. This spring has been quite dry, thus no water on the floor.

High airflow computer power supplies hate humidity. I've stupidly burned through a couple discovering this. Low airflow and fanless systems seem to accumulate enough dry heat to survive. Since the basement is both my MPOE and an otherwise naturally cooled, out of the way space, this seems like the place to put servers and networking equipment. The humidity calls for a basement dehumidifier. Even without a server rack, the mold and deteriorating brick situation likely call for a dehumidifier. I can set humidity at a reasonable level and run collected water into my existing drain.

Whether your "server closet" is a basement, closet, attic or something else, consistent (lowish) temperatures and low humidity are a necessity. I'm somewhat jealous of my Northern California friends; even during the winter wet season, a section of an attached garage probably works just fine as long as heat can be exhausted upwards or out.

I have something of a fetish for rackmount equipment. For many, a simple 12U wall rack would work just fine. For serious geeks, nothing less than 42U will do. I'm somewhere in between, let's say 25U. I have an arch space under my chimney that is centrally located, not far my MPOE, and otherwise unused. We'll see whether I can use that or need to run exhaust heat to an exterior window. It might actually be easier to run exhaust heat up the chimney along with the furnace exhaust.

I started out thinking that my Asus RT-N66U might be capable enough to act as my main firewall/router. Now I'm thinking I need a more elaborate setup:
Cable Internet Gateway => External Firewall/Router => DMZ => Internal Firewall/Router
Equipment is not a problem; I have a ton of old, unused equipment perfectly suited for routing and filtering packets. Cases are a bit of a problem. I strongly prefer rackmount to a bunch of generic PC and consumer appliance cases. I want my rack to be beautiful; call it what it is: a pointless fetish. I demand satisfaction. ;-P

Rackmount cases are seriously expensive (quality 1U => $200 USD), presumably because they're specialized, enterprise equipment. I'm currently looking into repurposing some old 1U Cobalt RaQ3 cases to hold mini-ITX boards. Christian Vogel has kindly shared information about the front panel pinout of these cases on his blog. This is very cool, as I've been wanting to get into Arduino development and interfacing for a while anyway.

As always, feedback here or on G+ is welcome. Let's see some pictures of your setups. How are you managing environmental factors for your home equipment? What are your tips for those who are inclined to be your own networking and server gurus at home? As a side note, I've been gradually moving critical services to the AWS cloud, yet I still need decent networking, security, storage, media servers and playback, and home automation at home. I suspect others are in the same boat. Let's hear about it!

Sunday, April 7, 2013

ZFS Storage Conversion Postmortem

Until recently, my home data storage "solution" consisted of a single PC running Arch Linux and 3x largish SATA drives running JFS in a JBOD. The Linux box also runs my MythTV backend, and the whole setup evolved a bit organically. I'm willing to lose the TV recordings if something fails. CD and DVD rips, photos, and documents are replicated between the Linux server, Windows desktop, and Mac portable in a very manual and haphazard way. I've been lucky not to lose anything important.

With my (personal) office equipment now moved back home from my former telework space, it's time to get my data better organized and replicated. I'm documenting my specific hardware and software configuration here, mostly as a log for future me. There may be generally useful information hidden among the specifics; if so, I apologize in advance for making you sift for it.

Newly available equipment:

  • 9x 1TB WD Black drives purchased between late-'08 and early-'10 for an aborted storage build
  • 2x 3TB WD Red drives purchased a couple months ago
  • Synology DS1511+ 5x bay NAS chassis
  • Dell PowerEdge T610 server that had been used for work-related virtualization

The WD Blacks got some use in the NAS and as temporary backup drives; some just sat in a drawer for a couple years. :-/

The Dell T610 is total overkill for most of what I do now.

  • 2x Intel Xeon E5649 processors at 2.53GHz, yielding 12 cores
  • 48GB (6x 8GB) ECC DDR3 RDIMMs
  • 2x Broadcom NetXtreme II GigE NICs
  • 2x Intel ET GigE NICs
  • SAS 6/iR (LSI 1068e) storage controller
  • 2x 250GB Samsung drives in a hardware RAID1 system mirror
  • 6x drive blanks; Dell likes to sell high-margin drives with their caddies
So I wanted to turn the Dell into a backup storage and development virtualization box that is used mostly on an as-needed basis. I went into this plan without much of a sense of what system power usage would be, just that it would be high compared to my other systems.

Dell T610 disrobed w/ caddies and drives
The Dell was initially running Debian 6. My actual work went something like this:

  1. Take the faceplate off the Dell to discover 6x drive blanks. Um, yay.
  2. Discover that drives will function without caddies as long as I don't jostle anything.
  3. Try 3TB WD Red in the Dell only to discover the controller maxes out at 2TB.
  4. Decide to add 6x 1TB WD Blacks to the Dell, since that should work.
  5. 3x of those WD Blacks were running a RAID5 in my Synology NAS.
  6. Place order for 8x cheap drive caddies before things go much further.
  7. Start rsync backup of Synology NAS to my old MythTV server.
  8. Start serious burn-in test of 3x unused WD Blacks in the Dell.
  9. Dissolve RAID and yank 3x WD Blacks from the NAS after rsync completes.
  10. Hey, those caddies got here fast!
  11. Install 2x 3TB WD Reds in the NAS as a pure Linux RAID1.
  12. Start rsync restore from the MythTV server to the Synology NAS.
  13. Get 6x WD Blacks into Dell caddies and installed.
  14. Start less serious burn-in of the 3x WD Blacks that had been in the NAS.
  15. Install Debian Wheezy onto the Dell system drive mirror /dev/sdg.
  16. Broadcom bnx2 requires non-free firmware; switch to Intel NIC for install.
  17. Figure out how to make grub write the boot record to sdg not sda.
  18. Install modules and tools from ZFS on Linux; that was easy!
  19. Struggle to zero out the WD Blacks w/ mdadm, dd, and parted.
  20. Create a RAID-Z1 pool with 5x WD Blacks and one warm spare.
  21. Install cpufrequtils to reduce system power usage.
  22. Begin testing the hell out of the ZFS pool.
Idle power usage with 8x drives is 173W. I still want to abuse ZFS a bit more by running through simulated failure scenarios. I've used ZFS on Solaris and FreeBSD, but never on Linux before this. So far so good. There's still work to do setting up automated backups between the Dell and the NAS and generally getting my data more organized. At least there's now a framework in which that can happen.

Wednesday, April 3, 2013

Home Storage for Enthusiasts

The cloud is here, but do you really trust all your CD, DVD, and Blu-ray rips and personal photos and videos to the cloud? This blog post is about home storage options, mostly focusing on Linux-based appliances and small servers. Windows Home Server (WHS) is also a rather cool technology, although its future and forward migration path is by no means certain. Being a Linux guy, that's where I'll focus.

For those who don't want to fiddle, I highly recommend a home NAS appliance. I have a Synology DS1511+, although there are other great options as well. Be prepared to spend some money for the sake of convenience. This isn't as cheap and dirty as adding drives and controllers to an old PC. +Piaw Na reminds us to consider what happens when your NAS appliance fails. Many NAS vendors use proprietary storage mechanisms that allow mixing disks of various sizes. Consider a NAS configuration that supports stock Linux software RAID and ext3/4 filesystems. You or your local Linux guru can then access your data if your NAS appliance fails.

Cheap and dirty is kind of where I'd like to focus. This means x86 hardware, Linux, near-line (or prosumer) SATA drives, and software-based data redundancy. Proper RAID disk controllers have battery backup for recovery after power failure and are prohibitively expensive for home use. Software RAID5 may not achieve trustworthy performance. The first rule of cheap and dirty SATA drives is that manufacturers will cheat to obtain better benchmark performance. Drives will cache writes, violate commands to sync ordered writes to disk, and fail very ungracefully on system or power failure. Your drives are optimizing for the common case where nothing ever fails. That's well and good until something fails...

Here are some tips for building that Linux-based file server. Thanks to +Yan-Fa Li for his additional pointers and reminders.

  1. Consider systems that support ECC (error correcting) memory. Some consumer AMD processors and boards used to support this. Intel Xeon systems generally support ECC, as do Core i3-32XX processors on certain motherboards. Data and filesystem metadata that is corrupted in memory never makes it onto your disks correctly.
  2. Disable write caching on your disk drives.
    hdparm -W0 /dev/sdX
    You'll need to automate this to run at boot time.
  3. Exercise your drives before adding them to your storage pool.
    badblocks -v -w -s -c 1024 /dev/sdX
    smartctl -t long /dev/sdX

    Drives seem to follow a reliability bathtub curve. New and old drives seem more prone to failure. Check your drives before relying upon them.
  4. Consider your recovery requirements. What data can you afford to lose? What data can you reconstruct (re-rip optical media, re-download, etc.)? What data can suffer some temporary (hours, days) unavailability? What data must always be available?
  5. Enable ERC/TLER error recovery timeouts where possible when using multi-drive arrays. Consider near-line storage quality drives or better (compared to consumer drives) when building your storage arrays. The current WD Red series drives are practically aimed at the prosumer and small business mass storage markets.
  6. Remember that RAID0 isn't RAID; it's not redundant. RAID0 is only for transient or reconstructable data.
  7. RAID1 and variants (10, 1E) are a great choice when you can afford the drives and loss of capacity. Performance and reliability can be quite good. You're throwing drives at the problem and reaping the rewards.
  8. Software RAID5 is scarier than you might think. Data and reconstruction information is scattered across all drives in the array. How much do you trust your drives, controllers, and system/power stability to keep this all in sync? Putting most filesystems atop an unreliable RAID5 is a recipe for disaster. Battery-backed hardware RAID5/6 has its place. I'm reasonably convinced that software RAID5/6 doesn't. Beware the write hole.
  9. ZFS is cool on Solaris and FreeBSD. It's now even cooler with ZFS on Linux. ZFS RAID-Z can be a reasonable and reliable software replacement for hardware RAID5/6. You're not going to see blazing speeds, but you're getting end-to-end checksumming. If you want blazing speeds, get an SSD. +Yan-Fa Li mentioned that he gets 300MiB/s from his 6 drive ZFS setup, enough to saturate gigabit Ethernet. Maybe leave the competition to the folks over on [H]ard|OCP and consider your specific use cases.
  10. Btrfs is the next great Linux filesystem that somehow never arrives. If ZFS was licensed for inclusion in the Linux kernel proper, btrfs might just fade away. I've used both, and btrfs doesn't even seem to aspire to be as good as ZFS already is. Sorry!
  11. Runtime redundancy is no substitute for backups. What happens if an entire storage system is lost or fried? Consider maintaining a limited backup storage system on site and copying treasured data to the cloud. Companies historically store offsite backups in case of disaster. Cloud storage can provide offsite backup insurance for the rest of us.
These tips are something of a work in progress as I build my home storage array and backup. Follow the discussion on Google+.