Cisco C22 M3 “Build” report: From Zero to vSphere in… two days?

Hi folks. The pile of project boxes in my home lab has gotten taller than I am, so when a Twitter follower asked me about running VMware vSphere on one of the systems not too far down in the stack, I took the challenge and said I’d try to get it going to see what I could report back.

Disclosure: While my day job is with Cisco, this computer was purchased out of my own pocket and used no proprietary/employee-only access to software or information. I do not provide end-user support for Cisco gear, nor do I recommend using used/aftermarket gear for production environments.

That system is a now-discontinued Cisco UCS C22 M3S. Yes, C22, not C220. It was an economy variant of the C220, more or less, with a lower cost and lower supported memory capacity as I recall. The one I have features a pair of Intel Xeon E5-2407 v2 processors (quad core 2.4GHz) and 48GB of RAM. The RAID controller is LSI’s 9220-8i, and for now I have a single 73GB hard drive installed because that’s what I found on my bench.

This is a standalone system, even though it’s sitting underneath a UCS 6296 Fabric Interconnect that’s freshly updated as well. I have the two on-board Gigabit Ethernet ports as well as a 4-port Gigabit Ethernet add-on card. And by way of disclosure, while I do work for Cisco and probably could have gotten a better box through work, I bought this one in a local auction out of my own pocket.

Warming up the system

The first thing I needed to do was make sure firmware, management controller, and so forth were up to date and usable. Cisco has long followed the industry standard in servers by making firmware and drivers freely available. I wrote about this back in 2014, when HPE decided to buck the standard, even before I worked for Cisco. You do have to register with a valid email address, but no service contract or warranty is required.

Since I was going to run this machine in standalone mode, I went to the Cisco support site and downloaded the Host Update Utility (HUU) in ISO form.

Updating firmware with the Host Update Utility (HUU) ISO

I loaded up Balena Etcher, a program used to write ISO images and other disk formats to USB flash drives. USB ports are easy to come by on modern computers, but optical drives are not as common. I “burned” the ISO to a flash drive and went to boot it up on the C22.

No luck. I got an error message on screen as the Host Update Utility loaded, referring to Error 906, “firmware copy failed.”

Doing some searching, I found that there were quirks to the bootability of the image. A colleague at Cisco had posted a script to the public community site in 2014, and updated it in 2017, which would resolve this issue. So I brought up my home office Linux box (ironically a HPE Microserver Gen8 that I wrote about in January), copied the script and the iso over, and burned the USB drive again with his script. This time it worked.

Recovering a corrupted BIOS flash image with recovery.cap

Alas, while four of the five firmware components upgraded, the BIOS upgrade was corrupted somehow. Probably my fault, but either way I had to resolve it before I could move forward.

Corrupt bios recovery, before and after

Seemed pretty obvious, and I figured the recovery.cap file would have been copied to the flash drive upon boot, but I figured wrong. You have to extract it from a squashfs archive inside the HUU ISO file. There’s even a ‘getfw’ program in the ISO to do this. Easy, right?

Of course not.

Turns out newer versions of OpenSSL won’t decrypt the filesystem image and extract the needed file, and even my year-out-of-date CentOS 7 box was too new. So I spun up a VM with the original CentOS 7 image and extracted there.

  1. Get the HUU for your system and UCS version (don’t use a C22 BIOS on a C240 or vice versa, for example).
  2. Mount or extract the ISO file
  3. Copy the GETFW/getfw binary out
  4. Unmount the ISO file
  5. ./getfw -b -s <HUU ISO FILE> -d .

This will drop a “bios.cap” file in the current directory. Rename it to “recovery.cap” … put it on a flash drive (plain DOS formatted one is fine), put it into the system, and reset your machine. You’ll go from the first screen with “Could not find a recovery.cap file..” to the second screen transferring to controller. And in a few minutes, your system should be recovered.

Preparing to boot the system

This is the easiest part, in most cases,  but there are a couple of things you may have to modify in the Integrated Management Controller (IMC) and the LSI WebBIOS interface.

Set your boot order. I usually go USB first (so I don’t have to catch the F6 prompt) followed by the PCIe RAID card. The RAID card will only show up if supported and bootable drives are installed though. This can be changed on the fly if you like, but I prefer to do it up front.

Check your RAID controller settings. Follow the BIOS screen instruction for going into WebBIOS (the text interface to configuring the RAID card), and make sure that you have disks presented in virtual drives. I had plugged a UCS drive and a random SSD in and only the UCS drive (a 73GB SAS drive) showed up. It did not appear to the F6 Boot Order menu though, as it was not set bootable in WebBIOS. A few key taps fixed this, and the drive appeared. Again, you can change the boot order after installing, but why not do it first?

Moving forward with VMware installation

This is the easy part, more or less. I went to VMware’s site and grabbed the Cisco custom ISO (which should have current drivers and configurations for Cisco components, especially the RAID controller and network cards). You can also install with the standard vSphere installer if you like.

I burned the 344 MB ISO to a flash drive, finding again that Etcher didn’t like it (complaining not being a bootable ISO) but Rufus did. With a Rufus-burned 8GB drive (choose “yes” to replace menu.c32 by the way), I was able to install the vSphere system and bring it up.

On first install attempt, I did see this message for about a second, and had no drives show up.

Turns out this error warns you that log files are not stored permanently when booting from a USB installation drive, and it was unrelated to the missing drives (which didn’t show up because I originally had an unconfigured SSD and no configured drives installed–see previous section to resolve this).

But when I had the hard drive configured, the install went smoothly.

It is somewhat funny that I’m working with 48GB of RAM and only 60ish GB of storage at the moment, but from here I was able to copy over my OS installation ISOs (8GB over powerline networking made it an overnight job) and bring up my first VM on the new system.

So where do we go from here?

For now, the initial goal of confirming that vSphere will install neatly on a C22 M3 with the 9220-8i RAID controller has been accomplished.

Next up, adding some more storage (maybe SSD if I can find something that will work), maybe bumping the RAM up a bit, and doing something useful with the box. It only draws 80-100 watts under light use, so I’m okay with it being a 24/7 machine, and it’s quiet and in the garage so it shouldn’t scare the neighbors.

If you’re looking to turn up an older Cisco UCS server in your home lab, get familiar with the software center on Cisco.com, as well as the Cisco Community site. Lots of useful information out there as well as on the Reddit /r/homelab site.

Have you rescued old UCS servers for your homelab? Share your thoughts or questions below, or join the conversation on Facebook and Twitter.

 

History of Silicon Valley Indeed: Is Fry’s Electronics Dying?

As most of my readers know, Fry’s Electronics has been a mainstay in Silicon Valley culture since the mid-1980s. It’s spread around the country, from southern California to Texas and Arizona and the Midwest and probably locations I don’t even know about. There’s even one in Las Vegas.

The stores have themes, running from the Wild West theme of the Palo Alto location (the oldest continuously operating Fry’s store), to Las Vegas’s obvious Las Vegas Strip theme. The current (and third) Sunnyvale location is themed History of Silicon Valley, with an oscillator on the front, huge sepia photos of the founding people and events of Silicon Valley, and more.

But perhaps, like Halted and Weird Stuff in the past two years, this piece of Silicon Valley history may be coming to an end.

The first store opened in Sunnyvale in 1985, near the current location of Faultline Brewing (just off 101 and Lawrence Expressway). When I came to California just over 10 years later, it had moved to the “chip” themed building on Kern and Lawrence (white walls with black marks like an integrated circuit; the building is now a Sports Basement). Sometime in the late 1990s or early 2000s they built a new, enormous store at Arques and Santa Trinita, just a block or so from the Kern location. It had parking for hundreds of cars, almost a hundred cash registers, a cafe in the middle of the store, auto electronics installation bays on the side, and all in all it was a formidable experience.

I’ve never seen more than half of the registers in operation, but it’s often been crowded, and I’ve occasionally run into old friends and former colleagues… even those who had long since moved out of the area would come back like pilgrims.

Lately the parking lot has seemed impressively sparse, and fewer shoppers on most visits (except around the holidays and major TV-inducing sporting events). The shelves were usually well stocked, and things were often where you expected them.

But things keep changing.

A Sad Sunnyvale Visit

I went to Fry’s in Sunnyvale last Friday to look for a water filter cartridge. Didn’t think to check stock online first, but once I walked in, I knew I wasn’t going to find it.

The parking lot was the typical empty for a Friday afternoon, maybe a bit more. There were more old guys in the cafe than shoppers in the rest of the store. Lots and lots of empty shelves, far fewer staff than usual, felt like the last days of Orchard Supply Hardware but with fewer people and 10 times the space.

As I walked around, seeing what was left, I got the feeling that they could probably carve the store up… block off half of it and build a new entrance, for an entire second store… and still have plenty of room.

I ended up not finding my water filter cartridge. I did pick up a USB hub and a model train magazine though.

And maybe the strangest thing… As far as I can remember, for the first time in my 22+ years of going to Fry’s (since my friend Ray took me there in 1996 or maybe early 1997), there was nobody at the exit to check my receipt.

What happened to Fry’s?

There’s been a lot of media coverage of this, mostly columnists not entirely unlike me, looking at how the world has changed in the last ten years or so. You usually didn’t go to Fry’s for knowledgeable sales staff (although they’d be happy to read the box to you). Warm inviting environments were not their forte, and the “Fry’s Lending Library” (where you’d buy something to use for a day, or test out something else, before returning it) was definitely not part of their marketing, but a lot of people used that function of their stores, despite the long and drawn-out return process.

You went to Fry’s because if you needed almost anything, from a cable to a capacitor to a chair to a computer to a Coke to a charcoal grill, you could find a variety to choose from. You could pick up a magazine and a candy bar and a soda and a pack of batteries in the long check-out line. And it was almost always good for those things.

The biggest thing that happened to Fry’s was Amazon. And I’d say they started adjusting to Amazon about five years too late.

They acquired Cyberian Outpost in 2002 to build out their e-commerce business, and that went sluggishly. They started doing price-matching to local stores and online stores, but either trained their staff wrong or didn’t train them at all, which meant that even if a 4-port KVM switch was available and in stock 10 miles away, they’d try to match it to an 8-port switch and deny the match (happened to me, and I ended up taking the 10 mile trek).

As time went by, they merged outpost.com into frys.com and made it a better experience, including in-store inventory checking and in-store pickup. They’ve started doing price-matching smoothly and accurately, even to Amazon and Best Buy and Newegg. They even introduced same-day home/office delivery recently.

But the parking lots are more empty than not, the stores are hollow echoes of their past glory, and experts as well as fellow customers are having trouble seeing hope for Fry’s Electronics’s future.

Where do they go from here?

We know the Palo Alto store is closing (due to a lease not being renewed), and there are reports of low stock and low staff around the country leaving many fans and shoppers dubious.

Aside from that, spokesmen for Fry’s say they are rebalancing, restocking for the holidays, adjusting inventory, and otherwise laying in for the long haul. It is true that, from Thanksgiving night through Epiphany, Fry’s does tend to pick up in pace, so it’s entirely possible that the next month will see a restructuring and new life.

At this point, we’ll have to wait and see if Santa brings a new life to Fry’s Electronics this Christmas season.

Where can we go from here?

On the upside, two other mainstays are holding up fairly well.

After an early morning fire in April 2019 destroyed their Sunnyvale location on El Camino, Central Computer (another 1985 Silicon Valley classic) is opening a new Sunnyvale store over near where HRO (and Disk Drive Depot and Action Surplus and the original Fry’s store) used to be. Signage and social media say it should be open by the end of the year. In the meantime, they have locations in Santa Clara, San Mateo, Fremont, and San Francisco.

And after formally going out of business in January 2019 after 54 years, Halted/HSC was bought by Excess Solutions, and they’re settling in nicely at the 7th St and Alma location in San Jose. You’ll find classic Halted stock and some of the classic Halted staff alongside the surplus gear, furniture, office supplies, and components that Excess Solutions has been known for.

If you’re looking for a smaller component source, Anchor Electronics is still around  in an industrial part of Santa Clara.

And assuming Fry’s doesn’t close down their Sunnyvale store before then, you can look forward to the return of the Electronics Flea Market on March 14, 2020, in the side parking lot at Fry’s Sunnyvale.

All may not be lost, but it seems like it sometimes.
What do you think about the state of electronics retail in Silicon Valley and beyond? Share your thoughts in the comments below, or join the conversation on Facebook and Twitter.

Sidebar: Setting up your cryptocurrency wallet

Aside

This is a sidebar post; you may wish to read the main mining rig post first.

Depending on the currency you choose to mine, you may need to create a wallet and/or address to receive the proceeds. The way this works will vary across currencies, but they’re very similar.

Wallets are usually a copy of the blockchain for your currency, along with the keys and address for your own “account” in the currency. It’s sort of an odd configuration if you compare to modern banking, but it serves the dual purpose of tracking your own balance and strengthening the security and scalability of the network. Continue reading

I’ve looked at clouds from both sides now – are they just vapor?

For those of you not of a certain age… a bit of a soundtrack for this post.

 

 

I wrote last month about the “antsle” “personal cloud server,” and a few people on Minds had a brisk but respectful debate over whether it was cloud, and whether there was more to cloud than cloud storage (i.e. Dropbox, Box, Owncloud, OneDrive, Sugarsync, etc).

It got me to thinking about how I’d define “cloud” and why others feel differently. So here’s a bit of a soft-topic consideration for you along the way.

I was first exposed to the buzzword around 2009, when a major PC and IT gear reseller from the midwest was trying to convince me on every call and email thread that I should buy The Cloud(tm). My rep never could tell me why, or what problem it would solve, a common shortcoming of quota-bound sales reps. I think the closest to a justification I ever got was “Just give a try, you’ll be able to tell where you can use it.” And I didn’t.

As the current decade rolled along, anyone running the server side of a client/server model called themselves The Cloud(tm). And of course, Amazon Web Services and other players came along to give their own definitions and market shares to the matter.

Today, at its extreme interpretation, anything not running in focus on your current personal device is probably considered “cloud” by someone. And to be fair to antsle, that’s where they fit in a way.  Continue reading

These 3 hot new trends in storage will blow your mind! Okay, maybe not quite. (2/2)

I’ve attended a couple of Tech Field Day events, and watched/participated remotely (in both senses of the word) in a few more, and each event seems to embody themes and trends in the field covered. Storage Field Day 5 was no exception.

I found a couple of undercurrents in this event’s presentations, and three of these are worth calling out, both to thank those who are following them, and give a hint to the next generation of new product startups to keep them in mind.

This post is the second of a series of two, for your manageable reading pleasure. The first post is here.

Be sure to check out the full event page, with links to presenters and videos of their presentations, at http://techfieldday.com/event/sfd5/

3. The Progressive Effect: Naming Names Is Great, Calling Names Not So Much

Back at the turn of the century, it was common for vendors to focus on their competition in an unhealthy way. As an example, Auspex (remember them) told me that their competitor’s offering of Gigabit Ethernet was superfluous, and that competitor was going out of business within months. I’ll go out on a limb and say this was a stupid thing to say to a company whose product was a wire-speed Gigabit Ethernet routing switch, and, well, you see how quickly Netapp went out of business, right?

At Storage Field Day 5, a couple of vendors presented competitive/comparative analysis of their market segment. This showed a strong awareness of the technology they were touting, understanding of what choices and tradeoffs have to be made, and why each vendor may have made the choices they did.

Beyond that, it can acknowledge the best use for each product, even if it’s the competition’s product. I’ll call this the Progressive Effect, after the insurance company who shows you the competitor’s pricing even if it’s a better deal. If you think your product is perfect for every customer use case, you don’t know your product or the customer very well.

Once again, Diablo Technologies did a comparison specifically naming the obvious competitor (Fusion-io), and it was clear that this was a forward-looking comparison, as you can order a hundred Fusion-io cards and put them into current industry standard servers. That won’t work with most of the servers in your datacenter with the ULLtraDIMMs just yet. But these are products that are likely to be compared in the foreseeable future, so it was useful context and use cases for both platforms were called out.

Solidfire’s CEO Dave Wright really rocked this topic though, tearing apart (in more of an iFixit manner than an Auspex manner) three hyperconverged solutions including his own, showing the details and decisions and where each one makes sense. I suspect most storage company CEOs wouldn’t get into that deep of a dive on their own product, much less the competition, so it was an impressive experience worth checking out if you haven’t already.

There were some rumblings in the Twittersphere about how knowing your competitor and not hiding them behind “Competitor A” or the like was invoking fear, uncertainty, and doubt (FUD). And while it is a conservative, and acceptable, option not to name a competitor if you have a lot of them–Veeam chose this path in their comparisons, for example–that doesn’t mean that it’s automatically deceptive to give a fair and informed comparison within your competitive market.

If Dave Wright had gone in front of the delegates and told us how bad all the competitors were and why they couldn’t do anything right, we probably would’ve caught up on our email backlogs faster, or asked him to change horses even in mid-stream. If he had dodged or danced around questions about his own company’s platform, some (most?) of us would have been disappointed. Luckily, neither of those happened.

But as it stands, he dug into the tech in an even-handed way, definitely adding value to the presentation and giving some insights that not all of us would have had beforehand. In fact, more than one delegate felt that Solidfire’s comparison gave us the best available information on one particular competitor’s product in that space.

 

 

This is a post related to Storage Field Day 5, the independent influencer event being held in Silicon Valley April 23-25, 2014. As a delegate to SFD5, I am chosen by the Tech Field Day community and my travel and expenses are covered by Gestalt IT. I am not required to write about any sponsoring vendor, nor is my content reviewed. No compensation has been or will be received for this or other Tech Field Day post.