About rsts11

Big data integrator/evangelist I suppose. Formerly a deep generalist sysadmin and team lead, still a coffee guru, and who knows what else...

Money Pit: A quick sidebar on my 3D printing (mis-)adventure

You’ve seen the first two parts of my 3D printing series, and I promised part three in the near future.

I decided to postpone the third part a little bit, in order to share a cautionary tale and some printer gore with you. I’m hoping this will be a one-time error, but if you’re like me, you might not think it’s so bad until it shuts down your printing operation for a week or two and potentially costs you $30+ in replacement parts.

First, the gore

Second, the TLDR

(tl;dr means too long;didn’t read. However, I hope you’ll read on.)

Continue reading

Money Pit: 3D Printing Part 2 – First Round of Enhancements

This is one topic in a series of what I’m calling “money pit” projects. To be fair, it’ll be money and time pit topics, and nothing that you’d really have to get a second mortgage on your house to do… but things always get a bit out of hand.

This project is the 3D Printing project. Expect it to be an ongoing series, and I’m hoping to have some friends join the effort and offer their feedback as well.

See the previous part (The Back Story, The Rationale, and The Assembly) for the lead-in to this project. From here we’ll get into the enhancements and early printing.

Our first round of enhancements include:

  • Power protection
  • Metal extruder and printed filament guide
  • PEI print bed
  • Upgraded Bowden tubing

I mentioned OctoPi / OctoPrint in the first installment, so I’ll leave the details out this time other than to say you really should set one of these up. Let me know in the comments if you’d like more details in a future post. 

Continue reading

Money Pit: 3D Printing Part 1 – The Back Story, The Rationale, and The Assembly

This is one topic in a series of what I’m calling “money pit” projects. To be fair, it’ll be money and time pit topics, and nothing that you’d really have to get a second mortgage on your house to do… but things always get a bit out of hand.

This project is the 3D Printing project. The second part is available at First Round of Enhancements and part 3 should be out within a week. 

The Back Story

It all goes back to five or so years ago, when I bought a couple of Banana Pro single board computers from LeMaker in France.The Banana Pro was a Raspberry Pi-inspired board, but with gigabit Ethernet and external SATA on board. Great idea, but they didn’t sell as much as the RPi, so the accessory market was a lot lighter. I think there were 4 cases I found in the past 5 years, many of which were not readily available in the US.

I did order a few cases from China that had a section for the SATA drive, and stocked up on cables for the SATA drives. But I wasn’t too happy with what was out there.I found some of the 3D printer sites where people had built some cases, and thought “someday I’ll get a printer and make some cases.” I said that about every year for 4 years.

Then earlier this year, some more usable cluster kits came onto the used market from the now-defunct rabb.it startup. By “some,” I mean about a thousand of them. (Click on the photo below if you want to buy one of the kits yourself. It is an eBay partner network link but I have no association with the seller other than as a buyer of one cluster kit so far.)Single Board Computer Array with Intel NUC5PPYH and NVIDIA Jetson TK1They each contain ten NUC5PPYB quad-core pentium NUC machines and five NVIDIA Jetson TK1 dev boards. I pondered it for several months (not as long as the printer), finally bought one, and it showed up a week later. (I’ll write more about that project separately, and you can read my friend Stephen Foskett’s Pack Rat series about the rabb.it clusters here.)

About the same time, I broke down and bought a Creality Ender 3 Pro printer from my local geek shop, Central Computers. Central also stocks the Creality-branded filament for $20 per 1kg roll, and they’re about four miles from home. You can also buy directly from Creality, or choose some sellers on Amazon like SainsmartContinue reading

Cisco C22 M3 “Build” report: From Zero to vSphere in… two days?

Hi folks. The pile of project boxes in my home lab has gotten taller than I am, so when a Twitter follower asked me about running VMware vSphere on one of the systems not too far down in the stack, I took the challenge and said I’d try to get it going to see what I could report back.

Disclosure: While my day job is with Cisco, this computer was purchased out of my own pocket and used no proprietary/employee-only access to software or information. I do not provide end-user support for Cisco gear, nor do I recommend using used/aftermarket gear for production environments.

That system is a now-discontinued Cisco UCS C22 M3S. Yes, C22, not C220. It was an economy variant of the C220, more or less, with a lower cost and lower supported memory capacity as I recall. The one I have features a pair of Intel Xeon E5-2407 v2 processors (quad core 2.4GHz) and 48GB of RAM. The RAID controller is LSI’s 9220-8i, and for now I have a single 73GB hard drive installed because that’s what I found on my bench.

This is a standalone system, even though it’s sitting underneath a UCS 6296 Fabric Interconnect that’s freshly updated as well. I have the two on-board Gigabit Ethernet ports as well as a 4-port Gigabit Ethernet add-on card. And by way of disclosure, while I do work for Cisco and probably could have gotten a better box through work, I bought this one in a local auction out of my own pocket.

Warming up the system

The first thing I needed to do was make sure firmware, management controller, and so forth were up to date and usable. Cisco has long followed the industry standard in servers by making firmware and drivers freely available. I wrote about this back in 2014, when HPE decided to buck the standard, even before I worked for Cisco. You do have to register with a valid email address, but no service contract or warranty is required.

Since I was going to run this machine in standalone mode, I went to the Cisco support site and downloaded the Host Update Utility (HUU) in ISO form.

Updating firmware with the Host Update Utility (HUU) ISO

I loaded up Balena Etcher, a program used to write ISO images and other disk formats to USB flash drives. USB ports are easy to come by on modern computers, but optical drives are not as common. I “burned” the ISO to a flash drive and went to boot it up on the C22.

No luck. I got an error message on screen as the Host Update Utility loaded, referring to Error 906, “firmware copy failed.”

Doing some searching, I found that there were quirks to the bootability of the image. A colleague at Cisco had posted a script to the public community site in 2014, and updated it in 2017, which would resolve this issue. So I brought up my home office Linux box (ironically a HPE Microserver Gen8 that I wrote about in January), copied the script and the iso over, and burned the USB drive again with his script. This time it worked.

Recovering a corrupted BIOS flash image with recovery.cap

Alas, while four of the five firmware components upgraded, the BIOS upgrade was corrupted somehow. Probably my fault, but either way I had to resolve it before I could move forward.

Corrupt bios recovery, before and after

Seemed pretty obvious, and I figured the recovery.cap file would have been copied to the flash drive upon boot, but I figured wrong. You have to extract it from a squashfs archive inside the HUU ISO file. There’s even a ‘getfw’ program in the ISO to do this. Easy, right?

Of course not.

Turns out newer versions of OpenSSL won’t decrypt the filesystem image and extract the needed file, and even my year-out-of-date CentOS 7 box was too new. So I spun up a VM with the original CentOS 7 image and extracted there.

  1. Get the HUU for your system and UCS version (don’t use a C22 BIOS on a C240 or vice versa, for example).
  2. Mount or extract the ISO file
  3. Copy the GETFW/getfw binary out
  4. Unmount the ISO file
  5. ./getfw -b -s <HUU ISO FILE> -d .

This will drop a “bios.cap” file in the current directory. Rename it to “recovery.cap” … put it on a flash drive (plain DOS formatted one is fine), put it into the system, and reset your machine. You’ll go from the first screen with “Could not find a recovery.cap file..” to the second screen transferring to controller. And in a few minutes, your system should be recovered.

Preparing to boot the system

This is the easiest part, in most cases,  but there are a couple of things you may have to modify in the Integrated Management Controller (IMC) and the LSI WebBIOS interface.

Set your boot order. I usually go USB first (so I don’t have to catch the F6 prompt) followed by the PCIe RAID card. The RAID card will only show up if supported and bootable drives are installed though. This can be changed on the fly if you like, but I prefer to do it up front.

Check your RAID controller settings. Follow the BIOS screen instruction for going into WebBIOS (the text interface to configuring the RAID card), and make sure that you have disks presented in virtual drives. I had plugged a UCS drive and a random SSD in and only the UCS drive (a 73GB SAS drive) showed up. It did not appear to the F6 Boot Order menu though, as it was not set bootable in WebBIOS. A few key taps fixed this, and the drive appeared. Again, you can change the boot order after installing, but why not do it first?

Moving forward with VMware installation

This is the easy part, more or less. I went to VMware’s site and grabbed the Cisco custom ISO (which should have current drivers and configurations for Cisco components, especially the RAID controller and network cards). You can also install with the standard vSphere installer if you like.

I burned the 344 MB ISO to a flash drive, finding again that Etcher didn’t like it (complaining not being a bootable ISO) but Rufus did. With a Rufus-burned 8GB drive (choose “yes” to replace menu.c32 by the way), I was able to install the vSphere system and bring it up.

On first install attempt, I did see this message for about a second, and had no drives show up.

Turns out this error warns you that log files are not stored permanently when booting from a USB installation drive, and it was unrelated to the missing drives (which didn’t show up because I originally had an unconfigured SSD and no configured drives installed–see previous section to resolve this).

But when I had the hard drive configured, the install went smoothly.

It is somewhat funny that I’m working with 48GB of RAM and only 60ish GB of storage at the moment, but from here I was able to copy over my OS installation ISOs (8GB over powerline networking made it an overnight job) and bring up my first VM on the new system.

So where do we go from here?

For now, the initial goal of confirming that vSphere will install neatly on a C22 M3 with the 9220-8i RAID controller has been accomplished.

Next up, adding some more storage (maybe SSD if I can find something that will work), maybe bumping the RAM up a bit, and doing something useful with the box. It only draws 80-100 watts under light use, so I’m okay with it being a 24/7 machine, and it’s quiet and in the garage so it shouldn’t scare the neighbors.

If you’re looking to turn up an older Cisco UCS server in your home lab, get familiar with the software center on Cisco.com, as well as the Cisco Community site. Lots of useful information out there as well as on the Reddit /r/homelab site.

Have you rescued old UCS servers for your homelab? Share your thoughts or questions below, or join the conversation on Facebook and Twitter.

 

History of Silicon Valley Indeed: Is Fry’s Electronics Dying?

Updated 12/23/2019 and 12/27/2019 – see end of article.

As most of my readers know, Fry’s Electronics has been a mainstay in Silicon Valley culture since the mid-1980s. It’s spread around the country, from southern California to Texas and Arizona and the Midwest and probably locations I don’t even know about. There’s even one in Las Vegas.

The stores have themes, running from the Wild West theme of the Palo Alto location (the oldest continuously operating Fry’s store), to Las Vegas’s obvious Las Vegas Strip theme. The current (and third) Sunnyvale location is themed History of Silicon Valley, with an oscillator on the front, huge sepia photos of the founding people and events of Silicon Valley, and more.

But perhaps, like Halted and Weird Stuff in the past two years, this piece of Silicon Valley history may be coming to an end.

The first store opened in Sunnyvale in 1985, near the current location of Faultline Brewing (just off 101 and Lawrence Expressway). When I came to California just over 10 years later, it had moved to the “chip” themed building on Kern and Lawrence (white walls with black marks like an integrated circuit; the building is now a Sports Basement). Sometime in the late 1990s or early 2000s they built a new, enormous store at Arques and Santa Trinita, just a block or so from the Kern location. It had parking for hundreds of cars, almost a hundred cash registers, a cafe in the middle of the store, auto electronics installation bays on the side, and all in all it was a formidable experience.

I’ve never seen more than half of the registers in operation, but it’s often been crowded, and I’ve occasionally run into old friends and former colleagues… even those who had long since moved out of the area would come back like pilgrims.

Lately the parking lot has seemed impressively sparse, and fewer shoppers on most visits (except around the holidays and major TV-inducing sporting events). The shelves were usually well stocked, and things were often where you expected them.

But things keep changing.

2019-12-22 13.59.06

A Sad Sunnyvale Visit

I went to Fry’s in Sunnyvale last Friday to look for a water filter cartridge. Didn’t think to check stock online first, but once I walked in, I knew I wasn’t going to find it.

The parking lot was the typical empty for a Friday afternoon, maybe a bit more. There were more old guys in the cafe than shoppers in the rest of the store. Lots and lots of empty shelves, far fewer staff than usual, felt like the last days of Orchard Supply Hardware but with fewer people and 10 times the space.

As I walked around, seeing what was left, I got the feeling that they could probably carve the store up… block off half of it and build a new entrance, for an entire second store… and still have plenty of room.

I ended up not finding my water filter cartridge. I did pick up a USB hub and a model train magazine though.

And maybe the strangest thing… As far as I can remember, for the first time in my 22+ years of going to Fry’s (since my friend Ray took me there in 1996 or maybe early 1997), there was nobody at the exit to check my receipt.

What happened to Fry’s?

There’s been a lot of media coverage of this, mostly columnists not entirely unlike me, looking at how the world has changed in the last ten years or so. You usually didn’t go to Fry’s for knowledgeable sales staff (although they’d be happy to read the box to you). Warm inviting environments were not their forte, and the “Fry’s Lending Library” (where you’d buy something to use for a day, or test out something else, before returning it) was definitely not part of their marketing, but a lot of people used that function of their stores, despite the long and drawn-out return process.

You went to Fry’s because if you needed almost anything, from a cable to a capacitor to a chair to a computer to a Coke to a charcoal grill, you could find a variety to choose from. You could pick up a magazine and a candy bar and a soda and a pack of batteries in the long check-out line. And it was almost always good for those things.

The biggest thing that happened to Fry’s was Amazon. And I’d say they started adjusting to Amazon about five years too late.

They acquired Cyberian Outpost in 2002 to build out their e-commerce business, and that went sluggishly. They started doing price-matching to local stores and online stores, but either trained their staff wrong or didn’t train them at all, which meant that even if a 4-port KVM switch was available and in stock 10 miles away, they’d try to match it to an 8-port switch and deny the match (happened to me, and I ended up taking the 10 mile trek).

As time went by, they merged outpost.com into frys.com and made it a better experience, including in-store inventory checking and in-store pickup. They’ve started doing price-matching smoothly and accurately, even to Amazon and Best Buy and Newegg. They even introduced same-day home/office delivery recently.

But the parking lots are more empty than not, the stores are hollow echoes of their past glory, and experts as well as fellow customers are having trouble seeing hope for Fry’s Electronics’s future.

Where do they go from here?

We know the Palo Alto store is closing (due to a lease not being renewed), and there are reports of low stock and low staff around the country leaving many fans and shoppers dubious.

Aside from that, spokesmen for Fry’s say they are rebalancing, restocking for the holidays, adjusting inventory, and otherwise laying in for the long haul. It is true that, from Thanksgiving night through Epiphany, Fry’s does tend to pick up in pace, so it’s entirely possible that the next month will see a restructuring and new life.

At this point, we’ll have to wait and see if Santa brings a new life to Fry’s Electronics this Christmas season.

Where can we go from here?

On the upside, two other mainstays are holding up fairly well.

After an early morning fire in April 2019 destroyed their Sunnyvale location on El Camino, Central Computer (another 1985 Silicon Valley classic) is opening a new Sunnyvale store over near where HRO (and Disk Drive Depot and Action Surplus and the original Fry’s store) used to be. Signage and social media say it should be open by the end of the year. In the meantime, they have locations in Santa Clara, San Mateo, Fremont, and San Francisco.

And after formally going out of business in January 2019 after 54 years, Halted/HSC was bought by Excess Solutions, and they’re settling in nicely at the 7th St and Alma location in San Jose. You’ll find classic Halted stock and some of the classic Halted staff alongside the surplus gear, furniture, office supplies, and components that Excess Solutions has been known for.

If you’re looking for a smaller component source, Anchor Electronics is still around  in an industrial part of Santa Clara.

And assuming Fry’s doesn’t close down their Sunnyvale store before then, you can look forward to the return of the Electronics Flea Market on March 14, 2020, in the side parking lot at Fry’s Sunnyvale.

All may not be lost, but it seems like it sometimes.

What do you think about the state of electronics retail in Silicon Valley and beyond? Share your thoughts in the comments below, or join the conversation on Facebook and Twitter.

December update 1 (Sunnyvale)

With the predictions of increasing stock and even a Black Friday extravaganza, I went back to Fry’s on Black Friday (November 29, pictured above) in the hopes of more signs of life. There were over 20 customers in the store, which is more than my previous visit showed, and I believe they had about 5 registers open (out of 64?).
The fryshelp Twitter account continued to respond promising conversion to the consignment model, and more stock “soon,” but it wasn’t looking promising. And the one item I needed that day, canned compressed air, was nowhere to be found.
I returned the weekend before Christmas, which in years long past would’ve been a crowded experience. I found fewer than 20 patrons, probably half walking out empty-handed.

2019-11-29 10.39.31

They had consistent stock on shopping baskets with trash bags, used to capture rainwater from roof leaks, and a few things were abundant (gaming chairs, signs for their “no credit required” leasing partner), but otherwise it didn’t seem much different. The computer/monitor/printer section was decimated, networking aisle was barren, and there was no sign of change otherwise.
And the Fry’s twitter accounts were slightly more pragmatic but still not really giving hope.
20191222 fryshelp
I’ve heard from friends that this is still happening in other locations around the country, so we’ll have to see if anything changes in 2020.

December Update 2 (San Jose)

After Christmas I went out to check on the Brokaw (San Jose) store, having seen a photo on Instagram that looked less despondent. Initially it didn’t feel as empty, but once I got walking around, it was.

The Apple section had one machine. The computers/monitors/printers section was mostly barren. And lots of shelves were empty. The cafe had an empty (and apparently unplugged) cooler case and one worker. Two registers were open, and they were moving fairly quickly although there was a short line waiting for them.