(Warning: Please don’t take this as an admonition to engage in unwanted intimate contact with waiters or other hospitality personnel.)

But seriously, have you messed with hardware recently?

Just a few years ago, the thought of a senior sysadmin who didn’t know the current de facto standard platform like he knew his kid’s first name would’ve been unthinkable. But as more people move toward cloud, virtualization, siloing, and general service provider clientship, it’s not unthinkable anymore.

I feel like one of a dying breed in my own environment, a sysadmin with live hardware skills. I primarily support departments that have high compute/ram/IO requirements, so until we’re ready to put Nutanix’s Hadoop workload platform to the test, we’re still doing analytics on bare metal. (Virtualize Hadoop? That’s like running Oracle on NFS. Oh, wait.)

I’m okay with this, as I’ve been doing the dirty work of hardware, from cardboard disposal to troubleshooting blinky lights for most of my two decades in technical operations.

But the challenge comes in when a coworker, who’s more into the “devops” and “software defined career” side of technical operations, gets tasked with figuring out why one of my hardware servers won’t boot, or why Kickstart/Jumpstart/FAI doesn’t find a hard drive. You can’t run your VMware CLI to figure this out, especially if VMware has never touched the machine, and Nagios won’t give you any hints if there’s no OS.

Be our host, be our host

If you’re in the final job of your career, at a company that uses no physical hardware, then this probably doesn’t apply to you.

But for the other 99% of you, if you haven’t dealt with hardware recently, it might be a good time to do so. Find an inexpensive but server-grade system and start messing with it. Get some familiarity with the current interconnects, PCI-e connectors (and why x16 isn’t always x16), and perhaps most poignant to my experience, how to work your way around a RAID card.

I recently picked up an older workstation, an HP XW9400 dual Opteron workstation, that’s perfect for this sort of effort. It is a heavy duty piece of hardware, with a 1000w power supply, but it has PCI-X, PCI, and PCI-e slots enough to use almost any gear I want to test out. It also has an onboard 8-port LSI 1068 SAS controller, which is the same family as I work with on Cisco and Dell hardware platforms.

I can also throw a PCI-X Fibre Channel card in, or a 10 Gigabit Ethernet card, or whatever I may want to mess with next. The one missing piece that every sysadmin should have some awareness of is lights-out management, so if you haven’t done iLo or iDRAC or CIMC or the like, you may want to consider a slightly outdated (read: affordable) HP/Dell/Cisco server with the risk of having fewer slots for expansion. The flavors are different but the behavior and expectations are close enough.

Buddy can you spare a server?

If your company isn’t throwing hardware away, consider eBay or your local Craigslist to source a base server, upgrades if needed, and whatever else you want. I built my first hadoop cluster on a pile of $100 HP DL180 servers off eBay, and continued buying gear from the seller who always had a supply of spare parts. Most of them won’t complain if you buy from their web store vs eBay, or the other way around–whatever your budget and preferences allow.

You can find a few of the XW9400 workstations for $200ish buy-it-now with a reasonable base config, if you don’t want to wait for an auction to finish. Don’t worry about the OS license, and make sure you check the shipping cost before bidding/buying.

But where do we go from here?

So you have this nice 1000w server and you’ve figured out the ins and outs of RAID configuration. What now?

Well, you were looking for a nice new space heater, right?

Seriously though, you probably have a good start for your home lab now. For me, this machine will probably be good for testing PCI-X and PCI-e cards going forward, but if I get past that stage, it’d make a great storage server. Or for that matter, a reasonable virtualization box, with a bit of upgrading. I pulled up the quickspecs on HP’s website for this model, and they sold hex-core Opterons for these workstations, and they support up to 64gb of RAM. Mind you, the 64GB of RAM would cost me about $2k on eBay, so that won’t happen this month… but 32GB of RAM will set me back less  than $200. I can probably stash half a dozen disks inside, or use a $40 SAS adapter to connect an external array.

If you got a rackmount box, you may not want to run it at home for the sake of domestic tranquility. But if you have a quiet closet, rack space at work, or the budget for a small colo service, give it some thought. Check your upgrade paths, or find a good use to fit your current config. Or worst case, if you kept the box, post it on eBay.

I have a stack of servers in my lab waiting for triage, so there may be a hardware troubleshooting post coming in March.

References:

• Check out my TFD colleague Chris Wahl’s posts on his home lab category.
• HP QuickSpecs. Good place to see options for HP workstations and servers, before digging into eBay

Got other links you think my readers would relate to this topic? Any thoughts of your own to share with other readers? Send me a comment below.