Enterprise-Class networking on the cheap for your home lab

This entry is part of my POHO (Psycho Overkill Home Office) series. 

I have a habit of overdoing my networking. My home core switch is a Juniper EX-series (courtesy of a Bay Area Juniper User Group meeting raffle), and for a while I had a 10-Gigabit Ethernet (10GbE) Extreme Networks switch (that cost less than a good laptop) ready to go in. Do I really need it? Probably not. I sold it last year but am now thinking about 10GbE again.

I’m here today to share some of my tips for finding affordable enterprise-class networking for your personal, home, lab, or photo shoot purposes. I also welcome your thoughts and experiences in the comments below. Let me know if you’ve found other ways to improve your dollars-to-metal KPIs at home or in a lab.

Caveat: I do not advocate these methods for anything your company may depend on, or anything production grade in general. However, if you’re closer to hobby network than to Fortune 500 core network, this may help you build beyond your budget.

Foreword: About 10 Gigabit Transceiver Formats

There are three formats or “sockets” for transceivers for 10GBE ports. Each of them can present short (SR), long (LR), or extended range (ER/XR/ZR) fiber interfaces, or “captive” cables in the form of CX4, twinax, or RJ45-type copper.

Module formats courtesy of  @networkhardware http://www.networkhardware.com/cisco-optics-cheat-sheetXENPAK are the oldest, largest, and least expensive (generally) of the form factors, and can be found integrated into older host interface cards as well as switches like the Extreme Summit 400. XFP are almost as old, smaller and lighter, and probably a bit more pricey than XENPAK. The current generation is SFP+, which has the same size as the gigabit SFP (almost the same size as a RJ45 connector) but supports 1 and 10 gigabit. (The second transceiver pictured above is X2, which I haven’t run into yet.)

Depending on distance between ports and the host adapter you choose, you may find one of these more desirable than the others. Fiber may be more flexible in terms of length and availability, and you can go from SC to LC if your transceivers don’t match, but CX4 or twinax will be more resilient to tension.

Be warned that some networking vendors break the standards and check for their own brand of transceivers, not allowing generic or other brand devices even if they are physically identical down to the manufacturer.  Vendor forums or a quick Google search will help you track down these issues and plan for, or work around, them.

1. Uplink ports are usually just ports. Stacking ports, not so much.

The cheapest and (maybe) easiest way to get 10GbE ports is to buy a 1GbE switch that has a few 10GbE uplink ports. My Extreme was this kind of solution–48 ports of 10/100/1000Base-T[X] and a two-port module on the back for 10GbE via XENPAK modules. There are a lot of switches out there on the used market that offer 2-4 “uplink” ports that can be used to connect a host.

Be warned that in many cases, the uplink modules will cost more than the base switch, if they’re not already included. I’ve been watching for the 10GBE uplink modules for a couple of my cheap 48 port switches, and they’re 4-5x what I paid for those switches. Dig into your datasheets and search for the module part numbers, and see what they’re going to cost you before investing in a base switch. (And see #2 below on this topic too.)

And don’t forget about optics. You can mix SC and LC endpoints with fairly affordable fiber cables, but if your uplink modules don’t have fixed connectors, you will need to get some sort of transceivers (optical, CX4, captive cables, TX) that can connect with your host adapters.

You might be able to use stacking ports as regular network ports, but that will warrant more research (and maybe finding a friend at the network vendor). I wouldn’t count on this option unless you already know otherwise.

2. Look for rebranded (or debranded) gear.

I got my first pair of 10GbE host adapters for about $23 each shipped from an eBay seller. Fully functional, PCI-X (backwards compatible with PCI if you don’t need full 10 gigabit speeds), with optical XENPAK modules built in.

Xframe_II_lores courtesy of hp.com information library http://on.rsts11.com/1buRCPG

Why so cheap? The seller had posted them with an HP part number, which 99% of the time returned compatibility only with HP/UX. Turns out they are very compatible with Linux, and while they don’t appear to be supported under VMware, you could put something like that in a homebrew SAN and run it into the core network at 10GbE.

Not everything is available rebranded, but you’ll find some network companies selling their products under IBM or Sun or Dell labels or the like. Switches, firewalls, and expansion modules have been known to show up under multiple vendors’ part numbers, and sometimes one vendor’s part can be half the price of the original manufacturer’s part, for identical metal and silicon.

This goes back quite a ways, at least to when Dell resold the Netgear FS switch line, Cisco resold QNAP storage arrays and HP Proliant servers, and IBM resold Brocade Silkworm SAN switches. Just like searching for typos, searching for alternate part numbers may help you get a good deal.

3. If you have a bit of budget, ask around about new and refurbished gear.

The big PC makers have outlet and financial services stores on their websites that sometimes have off-lease models for sale. There may also be first-market options if you have a bit more of a budget than I do.

ICX6430 via brocade.com http://on.rsts11.com/1buS1l3Recently on Twitter Gabe Chapman and were discussing low-density 10GbE options; Garrett from Brocade chimed in to point us toward a Brocade ICX6450-24 switch as an option. A quick web search showed that for about $3000 street price, you can get a brand new switch with 4 licensed 10GbE ports and a warranty. You could get it for about $2300 with only 2 10GBE ports, and then buy the extras later (or run them at gigabit speeds).

If you’re stocking a little closer to the revenue end of the network, or if you can’t quickly replace a failed model, you might be better off choosing new/retail over a $400 switch as-is on eBay with the same port count.

Disclaimer: I have no connection with Brocade; this is just an example from a Twitter conversation that included a helpful Brocade guy and a quick Google Shopping search. Although I’d be willing to review an ICX6450-24 if the option arose. 🙂 

4. Watch out for port/feature licensing!

Some vendors, especially those in the Fibre Channel world, offer port licensing as a way to reduce the initial outlay for a switch.  A lot of smaller FC switches worked this way, and when I had to sneak a SAN into the budget at a company where the finance folks insisted all servers cost $1000 because that’s what a PC costs at Fry’s, it saved my storage plan from extinction.

In the case of the ICX6450-24 above, the base switch has four SFP+ ports, two of which are enabled for 10GbE out of the box, and two of which are limited to regular gigabit speed. To get the other two ports up to 10GbE, you buy a license kit for about $800 (street) and enable the ports. That’s not too bad for a brand new enterprise switch, but if you buy a ten year old switch that has unlicensed ports, you may have trouble getting the license codes (even for a price).

You’ll probably want to talk to someone knowledgeable about the platform you’re considering, to evaluate the risk. Some vendors tie features to a serial number (Juniper for example), so as long as your device is licensed and the serial number is intact, you may be able to reinstall the licenses automatically. Others require a key code, so unless you can get into the switch and retrieve that, a factory reset could wipe out some of your ports. And in either case, if the equipment you’re buying doesn’t have the features you want, it may be expensive or impossible to obtain them.

5. Consider port-channel or other aggregation methods.

I’ll admit this is sort of a cop-out, but 8 ports of Gigabit Ethernet will likely be cheaper than a single port of 10GbE. You can get 4-port PCI-E 1GBE cards for $75 or so (as low as $25 if you don’t need ESXi 5/5.5 support), and a 48-port GigE switch that supports LACP or the like for under $200. So that’s under $150 per 8gbit link including cables. Check your OS or virtualization platform HCL to make sure the cheaper cards are compatible, of course, but it’s worth checking out this option if it works for your needs.

So where do we go from here?

Those are my tips so far… I’d welcome your comments below on how (or if) they’ve worked for you, or if you have any tips from your own experience to share with other readers. Maybe you’ve found a vendor whose 10GbE switches are more affordable for the home lab, or just had a good experience with a home-lab-friendly reseller? Please chime in.

Advertisements

10 thoughts on “Enterprise-Class networking on the cheap for your home lab

  1. Great post! The X2 modules are used in some low-density 10G blades for the Cisco 6500 and 4500 lines, as well as in the older Cisco 4948-10G switch which had 2 X2 uplink ports.

    10G has started to creep into lower-market solutions as well. You can actually get a new Cisco Small Business SG500-X model which includes 4 10G SFP+ ports for around $1200. This is a reasonably full-featured L2/L3 switch.

    NetGear even has an unmanaged 8-port 10GBASET with 1 SFP+ socket (model XS708E) that is sub-$1000.

    I’ve found the upper-end of small business solutions to work pretty well for serious home setups.

    • Thanks Bob! I think I had heard of X2 but had never seen them before.

      I have seen some reasonably affordable 10GBase-T switches out there, but the original spark for this conversation was Gabe looking for 8 SFP+ ports, which got complicated.

      I will look into the 10GBase-T options and update the post with them. I’m seeing some TX modules for my SMC gear that seem to be affordable, and I may have to start shopping for 10GBase-T host adapters. 🙂

  2. Great writeup! I had never seen X2 either, great comparison. One note so we don’t confuse the casual reader. Link aggregation protocols such as LACP. PaGp or even GEC does not exactly combine throughput from system to system. Even if the OS believes is has a single virtual team, the switches still must communicate to and from a single mac address at layer 2. Max throughput regardless of how many 1GB/sec links bound regardless of the protocol is 1GB/sec from system to system. What you do get from LACP is failover and the ability to achieve multiple 1GB streams to multiple systems simultaneously.

    It’s a shame we can’t just twist-tie some wires together and do 10GBe using contention, switchless networking (used to call them hubs) between only a couple systems would make 10GBe truly affordable.

  3. Great post, I’m surprised the 10GbE prices haven’t come down to a home affordable amount yet. I guess it wasn’t that long ago that gigabit became the standard for home stuff. One thing, you wrote 10GBE a lot, it should be 10GbE. I know that’s nitpicking, but such a great and informative post should have the right info.

    • Thanks for the comment, Scott, and yes it is 10-gigabit and not 10-gigabyte Ethernet. I guess 10GE is also fairly commonly used as the moniker as well.

      Ten gigabit Ethernet is becoming affordable on the used market… still closer to SoHo/smaller SMB but better than it was a couple of years ago. If you’re willing to go Xenpak, XFP, and/or CX4, you can even get multiport (12-24 port) switches for under $500. Gigabit Ethernet switches with a small number of 10GbE uplink ports are likely to be a bit more common, and more likely to be available in 10GBaseT. And the Cisco Small Business SG500XG-8F8T (the one I use in one of my labs) switch is down under $2500 on the market, new in box, if you have a bit of budget.

  4. Pingback: 10 Gbps network for less than $60 | Varun Priolkar

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s