Cisco C22 M3 “Build” report: From Zero to vSphere in… two days?

Hi folks. The pile of project boxes in my home lab has gotten taller than I am, so when a Twitter follower asked me about running VMware vSphere on one of the systems not too far down in the stack, I took the challenge and said I’d try to get it going to see what I could report back.

Disclosure: While my day job is with Cisco, this computer was purchased out of my own pocket and used no proprietary/employee-only access to software or information. I do not provide end-user support for Cisco gear, nor do I recommend using used/aftermarket gear for production environments.

That system is a now-discontinued Cisco UCS C22 M3S. Yes, C22, not C220. It was an economy variant of the C220, more or less, with a lower cost and lower supported memory capacity as I recall. The one I have features a pair of Intel Xeon E5-2407 v2 processors (quad core 2.4GHz) and 48GB of RAM. The RAID controller is LSI’s 9220-8i, and for now I have a single 73GB hard drive installed because that’s what I found on my bench.

This is a standalone system, even though it’s sitting underneath a UCS 6296 Fabric Interconnect that’s freshly updated as well. I have the two on-board Gigabit Ethernet ports as well as a 4-port Gigabit Ethernet add-on card. And by way of disclosure, while I do work for Cisco and probably could have gotten a better box through work, I bought this one in a local auction out of my own pocket.

Warming up the system

The first thing I needed to do was make sure firmware, management controller, and so forth were up to date and usable. Cisco has long followed the industry standard in servers by making firmware and drivers freely available. I wrote about this back in 2014, when HPE decided to buck the standard, even before I worked for Cisco. You do have to register with a valid email address, but no service contract or warranty is required.

Since I was going to run this machine in standalone mode, I went to the Cisco support site and downloaded the Host Update Utility (HUU) in ISO form.

Updating firmware with the Host Update Utility (HUU) ISO

I loaded up Balena Etcher, a program used to write ISO images and other disk formats to USB flash drives. USB ports are easy to come by on modern computers, but optical drives are not as common. I “burned” the ISO to a flash drive and went to boot it up on the C22.

No luck. I got an error message on screen as the Host Update Utility loaded, referring to Error 906, “firmware copy failed.”

Doing some searching, I found that there were quirks to the bootability of the image. A colleague at Cisco had posted a script to the public community site in 2014, and updated it in 2017, which would resolve this issue. So I brought up my home office Linux box (ironically a HPE Microserver Gen8 that I wrote about in January), copied the script and the iso over, and burned the USB drive again with his script. This time it worked.

Recovering a corrupted BIOS flash image with recovery.cap

Alas, while four of the five firmware components upgraded, the BIOS upgrade was corrupted somehow. Probably my fault, but either way I had to resolve it before I could move forward.

Corrupt bios recovery, before and after

Seemed pretty obvious, and I figured the recovery.cap file would have been copied to the flash drive upon boot, but I figured wrong. You have to extract it from a squashfs archive inside the HUU ISO file. There’s even a ‘getfw’ program in the ISO to do this. Easy, right?

Of course not.

Turns out newer versions of OpenSSL won’t decrypt the filesystem image and extract the needed file, and even my year-out-of-date CentOS 7 box was too new. So I spun up a VM with the original CentOS 7 image and extracted there.

  1. Get the HUU for your system and UCS version (don’t use a C22 BIOS on a C240 or vice versa, for example).
  2. Mount or extract the ISO file
  3. Copy the GETFW/getfw binary out
  4. Unmount the ISO file
  5. ./getfw -b -s <HUU ISO FILE> -d .

This will drop a “bios.cap” file in the current directory. Rename it to “recovery.cap” … put it on a flash drive (plain DOS formatted one is fine), put it into the system, and reset your machine. You’ll go from the first screen with “Could not find a recovery.cap file..” to the second screen transferring to controller. And in a few minutes, your system should be recovered.

Preparing to boot the system

This is the easiest part, in most cases,  but there are a couple of things you may have to modify in the Integrated Management Controller (IMC) and the LSI WebBIOS interface.

Set your boot order. I usually go USB first (so I don’t have to catch the F6 prompt) followed by the PCIe RAID card. The RAID card will only show up if supported and bootable drives are installed though. This can be changed on the fly if you like, but I prefer to do it up front.

Check your RAID controller settings. Follow the BIOS screen instruction for going into WebBIOS (the text interface to configuring the RAID card), and make sure that you have disks presented in virtual drives. I had plugged a UCS drive and a random SSD in and only the UCS drive (a 73GB SAS drive) showed up. It did not appear to the F6 Boot Order menu though, as it was not set bootable in WebBIOS. A few key taps fixed this, and the drive appeared. Again, you can change the boot order after installing, but why not do it first?

Moving forward with VMware installation

This is the easy part, more or less. I went to VMware’s site and grabbed the Cisco custom ISO (which should have current drivers and configurations for Cisco components, especially the RAID controller and network cards). You can also install with the standard vSphere installer if you like.

I burned the 344 MB ISO to a flash drive, finding again that Etcher didn’t like it (complaining not being a bootable ISO) but Rufus did. With a Rufus-burned 8GB drive (choose “yes” to replace menu.c32 by the way), I was able to install the vSphere system and bring it up.

On first install attempt, I did see this message for about a second, and had no drives show up.

Turns out this error warns you that log files are not stored permanently when booting from a USB installation drive, and it was unrelated to the missing drives (which didn’t show up because I originally had an unconfigured SSD and no configured drives installed–see previous section to resolve this).

But when I had the hard drive configured, the install went smoothly.

It is somewhat funny that I’m working with 48GB of RAM and only 60ish GB of storage at the moment, but from here I was able to copy over my OS installation ISOs (8GB over powerline networking made it an overnight job) and bring up my first VM on the new system.

So where do we go from here?

For now, the initial goal of confirming that vSphere will install neatly on a C22 M3 with the 9220-8i RAID controller has been accomplished.

Next up, adding some more storage (maybe SSD if I can find something that will work), maybe bumping the RAM up a bit, and doing something useful with the box. It only draws 80-100 watts under light use, so I’m okay with it being a 24/7 machine, and it’s quiet and in the garage so it shouldn’t scare the neighbors.

If you’re looking to turn up an older Cisco UCS server in your home lab, get familiar with the software center on Cisco.com, as well as the Cisco Community site. Lots of useful information out there as well as on the Reddit /r/homelab site.

Have you rescued old UCS servers for your homelab? Share your thoughts or questions below, or join the conversation on Facebook and Twitter.

 

Upgrading the HPE Microserver Gen 8 and putting it into service

A year and a half after my original write-up of the Ivy Bridge-based Gen8 Microserver, I’m finally doing a last round of pre-launch updates and documenting the upgrades I made.

You can read the original write-up (as updated to December 2018) here: Warming up the HP Microserver Gen8 and PS1810-8G switch

More links at the end of this post. Pricing has been updated as of 2019-08-15, but is still subject to change without notice.

Where do we start?

The HPE Microserver Gen8 as I received it had the Intel Pentium G2020T processor, a dual core, dual thread, 2.5 GHz processor with integrated Intel HD Graphics. For an ultra-low-end workgroup or SOHO server, that’s not too bad, and it’s better than the Celeron G1610T option.

gen8-cpus

Stock processor options for the HP Microserver Gen8

But since we’re not worried about the warranty and do want a bit more power, we looked at the following options for a CPU upgrade.

Xeon Processor CPU speed C/T TDP Integrated graphics? eBay price/link
August 2019 (December 2018)
E3-1230 v2 3.30 – 3.70 4/8 69 No 49.00 (was 75.00)
E3-1260L (v1) 2.40 – 3.30 4/8 45 HD2000 34.30 (was 57.00)
E3-1265L v2 2.50 – 3.50 4/8 45 HD2500 99.00 (was 100.00)

Since we didn’t have a use case in mind for this, we went for the E3-1265L v2 processor. CPU speed is reasonable, power is within the envelope for this system’s cooling capacity, and the price didn’t turn out too bad (although it was almost twice as much a year and a half ago).

The system arrived with 16GB of memory, which is the maximum supported with this generation of processor and a two-DIMM-slot motherboard (the CPU will handle 32GB but no more than 8GB per DIMM, and the Memphis Electronics 16GB DDR3 DIMMs require a newer generation of CPU).

The system also shipped with a single 500GB SATA drive and three empty trays for expansion, connected to the onboard B120i storage controller. There’s a low profile slot at the top suitable for an optical drive, or a hard drive carrier. According to the specs, the first two bays are 6gbit SATA and the last two bays are 3gbit SATA. You can add a P222 Smart Controller to provide battery-backed cache and expanded RAID options; these can be had for as low as $25 on eBay.

I installed a 32GB Micro-SD card for OS boot. Like the previous Microservers, the Gen8 offers an internal USB port, but Gen8 adds a MicroSD slot which may be less likely to snap off during maintenance. If I were running a heavy duty Windows or Linux server on this machine, I’d probably either put an SSD on a PCIe carrier card or use the optical drive SATA connector on the board to mount a boot drive in the optical bay. But for VMware or appliance-type platforms, or for light use Linux, the MicroSD should be enough.

Bringing the Microserver Gen8 up to date

One of the first things I do when building or populating a system is to upgrade any applicable firmware on the system. This could include the lights-out management, the system BIOS itself, drive controllers, optical drives, etc.

This gets complicated with HPE gear, as they decided to restrict all but “critical” BIOS update to customers with active support contracts or warranties. There are dubious workarounds, but it’s more of a pain than for any other mainstream vendor. Luckily (and I say that sadly), some of the critical vulnerabilities around Intel microcode in the past year led to the most recent Microserver Gen8 BIOS being considered critical.

So I gathered the latest BIOS, the ILO 4 firmware for out-of-band management, and the latest firmware for the PS1810-8G switch that this system will be connected to. (Unlike the computer systems, HPE’s networking gear carries a lifetime limited warranty and free access to firmware updates.)

With the switch connected to our upstream POE switch and the Microserver’s three network ports (two gigabit LAN, one ILO) connected to the switch, I upgraded the firmware on all three components and installed CentOS 7 from the latest ISO image via external USB flash drive. Additionally, I got a free 60-day trial license for ILO 4 Advanced from HPE.

One quirk I ran into was with regard to the .NET-based remote console and Chrome browser. In short, it doesn’t work unless you install a plugin to handle the .NET launching. I didn’t want to bother with Java either, so I accessed ILO from Microsoft Edge and used the .NET option from there.

Where do we go from here?

In the near term, I’m planning to install the Aquantia AQN-107 10GBase-T/NBase-T adapter and use it to test a couple of new devices in the home lab. Linux with iPerf or the like should be a good endpoint, and with a Thunderbolt 3-to-NBase-T adapter and an economical NBase-T/10G switch to work with, it should be compact and functional.

Longer term, with the former VMware “$25 server” being converted to EdgeLinux (from the makers of the Antsle servers we wrote about here and here), I will probably have this box serve as my in-home vSphere / ESXi system.

There’s a very small chance that I’ll break down and get the new Gen10 machine, but with as many spare computers as I have in the home lab now, it’s not a high priority.

What have you done with your Microserver recently? Share in the comments, or join the conversation on Facebook or Twitter.

For more information on the Microserver Gen 8 (especially around expandability):

HomeServerShow.com has an exhaustive page on Gen8 upgrades and other features and functions.

ServeTheHome has their release-time update on the Gen8 system here: HP ProLiant Microserver Gen8 Updated Specs and Pricing

And if you want the latest and greatest, the Microserver Gen10 came out a year ago with AMD Opteron X3000 processors.

Five fun and useful uses for an extra PCI slot

Anytime I pick up a “new” desktop/server/workstation class system, I check out the available PCI slots to see what my I/O options will be. There’s usually some open space there, and sometimes (especially when I’m building a system from board+case) there will be available “slots” on the case with no corresponding PCI slot on the system board.

Based on a recent home office closet cleaning, I’ve been inspired to share my top five uses for a spare slot.  Continue reading

Overkill in the rsts11 lab workshop – a homelab update for 2017

After being chosen as a VMware vExpert for 2017 this month, I was inspired to get working on refreshing my vSphere “homelab” environment despite a busy travel month in late February/early March. This won’t be a deep technical dive into lab building; rather, I just wanted to share some ideas and adventures from my lab gear accumulation over the past year.

As a disclosure, while I do work for Cisco, my vExpert status and homelab building are at most peripherally-connected (the homelab at home connects to a Meraki switch whose license I get an employee discount on, for example). And even though I’m occasionally surprised when I use older higher end Dell or HP gear, it’s not a conflict of interest or an out-of-bounds effort. It’s just what I get a great deal on at local used hardware shops from time to time.

The legacy lab at Andromedary HQ

Also read: New Hardware thoughts for home labs (Winter 2013)

C6100

Stock Photo of a Dell C6100 chassis

Continue reading

I’ve been called certifiable before – a sysadmin’s developing thoughts on certification

I’ve been a system administrator in some form or another since, I suppose, Summer 1988 when I provided ad hoc support for the RSTS/11 system at my college. I made a few bucks doing it as a lab assistant for two years, but I was probably too much of a proto-BOFH to stay on the payroll. I still fielded more questions than most of the lab assistants, and it prepared me moderately well for the following 25 years of user, system, and platform support.

One thing I’ve rarely ever done is get formally trained, or even less often, certified in a technology. I was three classes short of a computer science undergrad major just for fun, which should tell you I’m certifiable (didn’t take RPG, COBOL, or Calculus, but I did a bit of recreational Discrete Mathematics and two doses of Machine Structures).

Around the turn of the century, I took the Legato Certified Administrator (Data Protection) class and exam, and got certified on a technology I’d been deploying and managing for a few years at the time. In 2010 I took the Cloudera Hadoop Administrator course. I almost passed the certification exam then, but didn’t have time to go back and retake it before the retake offer expired. And that’s the extent of my formal training to date.

So what’s changed now?

Having been welcomed into the communities around Cisco’s datacenter technology and VMware’s virtualization platforms, I’m feeling an unnatural desire to work toward certifications in both of those areas. I have the 200-120 box set for CCNA Routing & Switching, although I’ve been leaning toward the datacenter path. I’m still trying to figure out what path to take with VMware, but we’ll have to see.

I was reading the Cisco Learning Network post “6 Reasons Employers Value Cisco Certifications” and it made me think about my aversion to certification over the last few years. So what’s wrong with certification, and what might be right about it?

What could possibly go wrong?

For one, some people collect certifications the way I collect old computers and soho routers. The cert may be representative of being able to complete a vendor’s exam, but may not reflect feet-on-the-ground (or hands-on-the-keyboard) skills, much less big picture architectural thinking. This was common when we were searching for a full year for a network admin at one job a few years back. No matter how many network certs you have, if you can’t at least give a shot to explaining subnetting, you’re probably not ready for the real world.

Another issue is that most certifications are vendor-specific, and may impart an undue bias toward that vendor over others. I’d like to think this isn’t the case, and a truly good network administrator/architect would know a broad swath of the market and be able to fit technology to an identified and triaged problem/business need, rather than trying to squeeze the business need into a given technology.

But what’s right?

For one, there are different skill levels and foci, and tiered/niched certifications can give a hint as to what level someone is. If I come in to an interview with a CCNA R&S, for example, I probably won’t be asked to provide in-depth explanations of SS7 or 802.11ac. There will always be bad interviewers, like the guy a few years ago who wanted me to explain in depth how BGP worked, after I had said twice that I wasn’t a network engineer and had only worked on LANs. So this isn’t foolproof on either end.

More important to me, now that I’m thinking about the process, is that pursuing a certification gives you a roadmap to study and prepare, and a somewhat finite goal to achieve. I never learned Perl because I didn’t really have a scope or a fixed goal. Making a personal goal to “learn me some networking,” alas, probably won’t get me anywhere.

Having a goal to, say, “take the CCNA DC exam at Cisco Live in May” gives me a framework and a finite goal. I can set aside time every week, study some of the Cisco Learning Network materials, watch some Pluralsight programs with Chris Wahl, and have a fixed time frame for preparation for the exam.

So where do we go from here?

For one, I think that box set of the 200-120 CCNA R&S library will probably sit in the closet for a few more months. It was on sale with an extra coupon at Barnes and Noble last summer, so I don’t feel too bad about it.

I will be plotting out my Cisco Certification Written Exam at Cisco Live in May, as hinted above. I blew off the free exam last year, which was probably good considering I’d had Tech Field Day 9 the week before (Tech Field Day events are great for scrambling the brain, and the 90-100F temperatures were leaning toward poaching my brain along with it).

I’m going to get more involved with Cisco Learning Network, as I’m sure Matt Saunders won’t let me slip on this. Hopefully some of my fellow Cisco Champions will cheer, jeer, prod, or otherwise support me on the journey as well.

And I’ll be sure to share my adventure with you fine readers… feel free to poke at me here if you have suggestions or haven’t heard from me on the certification path in a while.

Do share any certification feedback, suggestions for me, or warnings for other readers… in the comments below.