Need to set up a lightweight VPN to get into your low profile node remotely? Check out Stephen Foskett’s writeup on Zerotier. I’m using it on my Pi nodes to reduce NAT layers.
Many if not most Chia farmers run a full node on their farming / plotting machine. Some larger farms will use the remote harvester model, with a single full node and several machines farming plots on local storage.
If you’re using Flexfarmer from Flexpool, or just want a supplemental node (maybe to speed up your own resyncing, or to supplement decentralization on the Chia network), you might want a dedicated node that doesn’t farm or plot. And for that use case, you don’t really need dual EPYC or AMD Threadripper machines.
In fact, a well-planned Raspberry Pi 4B 4GB or 8GB system, with an external USB drive, will do quite well for this use. If you want to do a few forks as well, or another blockchain full node, a moderately-recent Intel NUC would do quite well for not much more.
So here we’ll look at three builds to get you going. Note that any of these can run a full node plus Flexfarmer if you want, or just a full node.
If you don’t already have Chia software and a full node installed, go ahead and install and sync the node on a full scale PC. it may save you five days of waiting. My original build for this use case was to test the blockchain syncing time from scratch.
Syncing from a semi-optimal Pi 4B from scratch took about 8 days, for what it’s worth. One member of the Chia public Keybase forum reported about 28 hours to sync on an Intel Core i5 12600k.
Caveat: Raspberry Pi boards are a bit more challenging to find and even harder to find anywhere near the frequently-touted $35 price point, or even under $150. And for Chia nodes, you want a minimum of the 4GB Pi 4B (8GB wouldn’t hurt). So while it’s possible to run on older hardware, it’s not recommended.
You might also be able to run on a Pi400 (the Raspberry Pi 4B in a keyboard case, which is much easier to find for $100 or so, complete). I plan to test this soon.
Raspberry Pi with external USB SSD.
This was my initial build, and today it’s running at the Andromedary Instinct providing an accessible full node for about 10-15 watts maximum.
As you saw in my 3D Printing series, after years of pondering a 3D printer, I was finally inspired to buy one when a pile of clusters came up on eBay from the defunct rabb.it video streaming service. In this series, I’ll take you through turning a rabbit door into some useful computing resources.
You can do something similar even after the clusters are sold out; a lot of people have probably bought the clusters and ended up not using them, so you’ll see boards on eBay or local marketplaces… or you can adjust the plans here to other models.
Update December 2021: This post languished in the drafts folder for about a year. I’ve updated links, and I’ll be reporting on some changes since the October 2020 launch of this cluster soon.
Let’s NUC this cluster out
Install memory, SATA cable, and SSD
Upgrade BIOS and set some annoying settings
Install your operating system
Set up central control
Install memory, SATA cable, and SSD
This is the least interesting part of the process, but you’ll need to do it before you can install an OS.
Start by loading the SODIMM of your choice onto the board. If you’re using an SSD like I am, you’ll connect the SATA cable to the black SATA connector next to the front USB stack, and the power cable to the beige connector perpendicular to the SATA connector. If you’re using a standard SD card, plug it into the SD slot to the right (as shown with the MAC address label). If you’re going with netboot (local storage? where we’re going we don’t need local storage!), just connect your network cable.
Upgrade BIOS and set some annoying settings
I created a bootable FreeDOS USB drive with Rufus, a common free software product used to create bootable USB media from ISOs (think Linux, Windows, etc). From there, we get the latest BIOS from Intel’s Download Center and place the file on the bootable drive. (Further BIOS instructions available on Intel Support.)
As of December 2021, 0079 is the latest BIOS, released April 20, 2020. You’ll need to search for NUC5PPYH even though the board’s model is PPYB.
Connect a monitor and keyboard, plug in the bootable drive, and apply power (or just reset the board). Use the F7 key to go into onboard flash update and load the BIOS file from the flash drive, or choose to boot from the flash drive and use the DOS-style flasher from there.
When you’ve done the upgrade and the system has rebooted, go into the BIOS with the F2 key and choose BIOS default values. Then go into the menus to enable all the USB ports (for some reason the default is to enable ports 1-3, leaving physical port 4 and header ports 5-6 disabled) as well as the SATA port if you are using that for storage. I’d also check the boot order (move net boot down in preference or disable outright if you don’t plan to use it). You can choose other settings as desired, and then press F10 to save and reboot.
Install your operating system
The easiest way to roll out the NUC side of the door would be to netboot an installation infrastructure like Cobbler. One of the first things I did when I went to work for the Mouse 10 years ago was setting up Cobbler for a deployment of RHEL 5.5.
Sure enough, Cobbler is still a thing, with very recent updates. I was able to get partway there this time and then, after several dozen runs to the garage and back to power cycle nodes, I gave up and installed from local media.
For CentOS 8, I did a manual install booting from a Rufus-created USB drive, with the SSD installed. I configured my storage and network options by hand, as well as user and root credentials. This left an “anaconda.ks” kickstart file on the installed system, which I copied to a second flash drive.
For the additional systems, I plugged both the CentOS 8 installer and the drive with the kickstart file into the NUCs and booted from USB. I ran into some strange storage issues with the drive not being blank, despite having chosen the kickstart option. Ideally, you would boot from the USB installer, it would find your kickstart config, and just roll out the software without intervention from there.
After that, if your DHCP server doesn’t assign hostnames you like, you can go in and set hostnames with hostnamectl or the like.
Set up central control
If you use a configuration management platform like Ansible, Puppet, Chef, cfengine, or the like, you’ll want to set those up at this point.
I’ve gone with the lightweight method so far, with shared SSH keys from a management host (an Intel NUC with CentOS on it, originally intended to be the cobbler server).
Use ssh-keygen to create your key files, and then ssh-copy-id can be used to push out the keys to your hosts. Then look into a more manageable option.
Where do we go from here?
As I finish this post in December 2021, a year after the original build, I’m looking at going back and making a few changes to the cluster to bring it up as a Kubernetes platform.
With the demise of CentOS as many of us know it, I’m planning to replace the installed OSes with Ubuntu LTS. I’m planning to test out some cryptocurrency cpu-based mining, and run Kubernetes platform(s) on it as well, and bring my second door up to speed (the RAM has been sitting in a box in the living room for a year now).
There’s a chance I’ll even do some lightweight Chia farming, using either bus-powered USB hard drives or some of the extra power connectors from the fused expanders for standard Seagate externals.
For those of you who have bought and built up these doors, what did you do with them? Feel free to share details and blog post links in the comments. I’ll put interesting ones into the body of this post as I see them.
Pricing disclaimer: All prices, availability info, and links are accurate as of the writing of this article (May 3, 2021) unless otherwise noted. Prices vary from day to day and availability does too. Use this info with a grain of salt especially if you are reading this in 2022 or beyond.
Here’s a quick rundown of what was involved in the process.
Feel free to shop in your preferred venues online or locally, or if you already have components, use them. These links are Amazon affiliate links, and if you use them, I get a few bucks to go toward my next hardware adventure. (I bought my NUC and RAM from Central Computer, a local computer store in Silicon Valley, and the NVMe drive came from Amazon.)
Base computer – NUC10i7FNH1 currently $570 at Amazon. You want the i7, and you want the FNH which is the “high” case that holds a 2.5″ drive as well as the m.2.
RAM – 32GB (2x16GB) DDR4 2666 or better SODIMM. Crucial 16GBx2 kit around $182 at Amazon. You can install 64GB, but you probably don’t need it with this processor.
Boot drive – I used a Samsung PM851 that’s not available on Amazon at the moment. Any 2.5″ SATA drive will do, even a HDD. Amazon has the WD Blue SSD 250GB for $45 or 500GB for $60. If you have something else on hand that’s at least 120GB, go ahead and use it, or if you want some internal plot storage, get something bigger.
Plotting drive – 2TB Inland Premium NVMe is popular with its 3200TBW rating, about $240 on Amazon but out of stock for the next week. If you watch your drive life, you can use cheaper NVMe or even SATA m.2 storage. But check the TBW (Total Bytes Written, or Terabytes Written) and warranty for your drive and take that into account.
OS install drive – Get a USB 3.0 drive with 16GB or more space, and use Balena Etcher or Rufus to burn Ubuntu 20.04 LTS to it.I like the Sandisk Ultra 32GB for price point and quality, about $10 at Amazon.
External long-term plot/farm storage – I’ll be using an 8TB external drive in the near term, but you can use whatever you have, even NAS storage.
Bonus: Staging disk. A user on r/chia suggested using a staging drive to copy your final plot file to, so that your plot process ends faster than if it has to be copied to slow disk. You can then automate moving the plot files to your external HDD at your leisure, and get back to plotting again up to an hour faster. For this, you can use an external USB 3.0 or better SSD like the WD My Passport SSD ($150 for 1TB), Crucial X8 ($148 for 1TB), or pretty much any SSD that will hold a batch of your plots (1TB will hold 9 plot files). You can also use a directory on your NVMe drive for this, but make sure you don’t let it fill up.
As you saw in my 3D Printing series, after years of pondering a 3D printer, I was finally inspired to buy one when a pile of clusters came up on eBay from the defunct rabb.it video streaming service.
In this series, I’ll take you through turning a rabbit door into some useful computing resources. You can do something similar even after the clusters are sold out; a lot of people have probably bought the clusters and ended up not using them, or you can adjust the plans here to other models.
The first thing I will put out there is that these are not latest-and-greatest state-of-the-art computers. If you’re looking for a production environment or DDR4 high density memory, keep looking. But if you want an inexpensive modular cluster that’s only about 5 years out of date, there’s hope for you in here.
Update – As of December 2021, I’ve updated some of the links and prices, and updated the description to show that the door clusters have been sold out for some time.
The original cluster
eBay seller “tryc2” has several hundred of these “door clusters” from rabb.it, a now-defunct video streaming service that closed up shop in mid-2019. I call it a “door cluster” as the 42 inch by 17 inch metal plate resembles a door, and gives you an idea of the ease of manipulating and fitting the environment into your home/homelab as it is delivered.
The cluster bundle would have set you back US$300, plus tax where applicable. The seller seems to have had a thousand or so and they were sold out by late 2020. [Updated 2021]
The cluster includes 10 Intel NUC quad-core boards (mine were NUC5PPYB quad-core Pentium; my friend Stephen Foskett got some that were newer NUC6CAYB Celeron boards which took more RAM). These boards feature one DDR3L SODIMM slot (max of 8GB), one SATA port with a non-standard power connector (more on this later), Gigabit Ethernet, HDMI out with a headless adapter (to fool the computer into activating the GPU despite no monitor being connected), four USB ports, and a tiny m.2 slot originally intended for wireless adapters.
In the center of the “door” are five NVIDIA Jetson TK1 boards. These were NVIDIA’s first low-end foray into GPU development, sold to let individuals try out machine learning and GPU computing. There are much newer units, including the Jetson Nano (whose 2GB version is coming this month), if you really want modern AI and GPU testing gear, but these are reasonably capable machines that will run Ubuntu 14 or 16 quite readily. You get 2GB of RAM and a 32GB onboard eMMC module, plus a SATA port and an SD slot as well as gigabit Ethernet.
The infrastructure for each cluster includes a quality Meanwell power supply, a distribution board assembly I haven’t unpacked yet, two automotive-style fuse blocks with power cords going to the 15 computers, and a 16 port Netgear unmanaged Gigabit Ethernet switch. With some modifications, you can run this entire cluster off one power cord and one network cord.
So there is a catch to a $300 15-node cluster. The Jetson nodes are component complete, meaning they have RAM and storage. However, the NUCs are barebones, and you’ll need some form of storage and some RAM.
For the Jetson nodes, you’ll need an older Ubuntu machine and the NVIDIA Jetpack software loader. For the installation host, Ubuntu 14.04 is supported, 16.x should work, and later versions are at your own risk. You’ll also need an Ethernet connection to a network shared with your Ubuntu machine, as well as a MicroUSB connection between your Ubuntu host and the Jetson, to load the official software bundle.
If you do get a cluster bundle with the two-memory-slot NUC boards, you have two options beyond the above. The easy and documented option is to look for 4GB SODIMMs instead of 8GB; you may save a buck or two, or if you’re like me, you may have a box of 4GB SODIMMs from various upgrades and not have to buy anything. The other option is to update your BIOS on the NUC and try out 2x 8GB. For some uses, 16GB will be worth the cost (vSphere or other virtualization clusters). I’d suggest going with a known quantity to update the BIOS to the latest version, and then trying 2x 8GB.
There are two paths to take once you have your gear collected and connected.
How do we lay out the gear?
What do we do with it?
I’ll look at my journey on both paths in upcoming episodes of this series. Spoiler: I’ve 3d-printed stacking plates for both the NUCs and the Jetsons, and am still working on how to mount the remaining pieces so I can e-waste the door piece. And as I write part 1, I still haven’t figured out what to do with the clusters.
Where do we go from here?
If you’ve bought one of these clusters (or more than one), feel free to chime in on the comment section and let me know what you’ve done with it. And stay tuned to this post (or @rsts11 on Twitter or Facebook) for updates on the next installments.
This is one topic in a series of what I’m calling “money pit” projects. To be fair, it’ll be money and time pit topics, and nothing that you’d really have to get a second mortgage on your house to do… but things always get a bit out of hand.
This project is the 3D Printing project. The second part is available at First Round of Enhancements and part 3 should be out within a week.
The Back Story
It all goes back to five or so years ago, when I bought a couple of Banana Pro single board computers from LeMaker in France.The Banana Pro was a Raspberry Pi-inspired board, but with gigabit Ethernet and external SATA on board. Great idea, but they didn’t sell as much as the RPi, so the accessory market was a lot lighter. I think there were 4 cases I found in the past 5 years, many of which were not readily available in the US.
I did order a few cases from China that had a section for the SATA drive, and stocked up on cables for the SATA drives. But I wasn’t too happy with what was out there.I found some of the 3D printer sites where people had built some cases, and thought “someday I’ll get a printer and make some cases.” I said that about every year for 4 years.
Then earlier this year, some more usable cluster kits came onto the used market from the now-defunct rabb.it startup. By “some,” I mean about a thousand of them. (Click on the photo below if you want to buy one of the kits yourself. It is an eBay partner network link but I have no association with the seller other than as a buyer of one cluster kit so far.)They each contain ten NUC5PPYB quad-core pentium NUC machines and five NVIDIA Jetson TK1 dev boards. I pondered it for several months (not as long as the printer), finally bought one, and it showed up a week later. (I’ll write more about that project separately, and you can read my friend Stephen Foskett’s Pack Rat series about the rabb.it clusters here.)
About the same time, I broke down and bought a Creality Ender 3 Pro printer from my local geek shop, Central Computers. Central also stocks the Creality-branded filament for $20 per 1kg roll, and they’re about four miles from home. You can also buy directly from Creality, or choose some sellers on Amazon like Sainsmart. Continue reading →