A quick Chia plotting note and a future searchable tip

Posted on by

2022-10-07: Updated with dual Xeon v4 bladebit.
2023-02-11: Updated with a madmax cuda plotter invalid ndevices error
2023-02-20: Updated with madmax cuda plotting with 128GB RAM 

The backlog is getting more interesting, but in an attempt to compare a Xeon Silver processor to one or two E5-2620v4 processors for some future Chia plotting, I’ve arrived at some benchmarks and a bladebit caveat for the new diskplotter.

The idea is to replace my OG plots with NFT-style plots, while still self-pooling them. At some point I will probably expand my storage again as well. 

Links are to original manufacturer specifications. If you find this document useful, feel free to send me a coffee. It might help with the memory upgrades on one or both machines too.

The systems involved:

System one:

Quick observation: On my Monoprice Stitch power meter, this system goes from about 60W at idle to 160W while plotting with Madmax or Bladebit. Not surprising, but noisy and blowy. 

System two:

Quick observation: This storage is very suboptimal for plotting, but it’s what came with the systems. I will dig into whether I have a larger faster SSD. Unfortunately this system only has USB 2.0 externally, and one low profile PCIe slot, so I’m a bit limited. Might put a 1TB NVMe drive in the PCIe slot though and see how that goes. 

System three (I’ve written about this one before):

  • Dell Precision Workstation T7910
  • Dual Xeon E5-2650Lv4 (each 14c28t)
  • 128GB RAM
  • 4x 1TB Samsung NVMe drives on the Ultra Quad (PCIe 3.0 x4 per drive) in software RAID-0
  • Ubuntu 22.04.1 LTS with current updates as of February 2023

Plotters:

Metrics so far:

System one, Chiapos with 12200MB memory assigned

Time for phase 1 = 10876.922 seconds. CPU (147.640%) Sun Oct 2 19:31:42 2022
Time for phase 2 = 4247.395 seconds. CPU (97.160%) Sun Oct 2 20:42:29 2022
Time for phase 3 = 9153.365 seconds. CPU (95.640%) Sun Oct 2 23:15:03 2022
Time for phase 4 = 635.266 seconds. CPU (97.980%) Sun Oct 2 23:25:38 2022

Total time = 24912.949 seconds. CPU (118.660%) Sun Oct 2 23:25:38 2022

System one, Madmax with -r 10

Phase 1 took 1461.93 sec
Phase 2 took 773.745 sec
Phase 3 took 1241.66 sec, wrote 21866600944 entries to final plot
Phase 4 took 61.6523 sec, final plot size is 108771592628 bytes
Total plot creation time was 3539.07 sec (58.9845 min)

System one, Bladebit with 16GB cache configured

Bladebit plot with 16G cache
Finished Phase 1 in 1744.37 seconds ( 29.1 minutes ).
Finished Phase 2 in 174.39 seconds ( 2.9 minutes ).
Finished Phase 3 in 1501.98 seconds ( 25.0 minutes ).
Finished plotting in 3420.74 seconds ( 57.0 minutes ).

System two with SN_750 NVMe drive (500GB), Bladebit with 24G cache

Finished Phase 1 in 1376.37 seconds ( 22.9 minutes ).
Finished Phase 2 in 148.09 seconds ( 2.5 minutes ).
Finished Phase 3 in 970.59 seconds ( 16.2 minutes ).
Finished plotting in 2495.06 seconds ( 41.6 minutes ).

Gigahorse metrics so far:

System three:

./cuda_plot_k32 -C 5 -n 5 -t /nvme/chia/ -2 /nvme/chia/ -d /plots/gigahorse-cuda/ -c xch1xxxxx -f a00fcxxxxx

Total plot creation time was 380.192 sec (6.33654 min)
Total plot creation time was 336.725 sec (5.61209 min)
Total plot creation time was 355.188 sec (5.9198 min)
Total plot creation time was 374.554 sec (6.24257 min)
Total plot creation time was 388.424 sec (6.47374 min)

The bladebit diskplot quirk:

If you get this error, there’s a good chance you didn’t specify the destination for the plot. 

 Allocating memory
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::Mconstruct null not valid
Aborted (core dumped)

So for example:

./bladebit -n 3 -f <farmerkey> -c <poolcontract> diskplot -t1 /nvme/tmp/ --cache 16G 

would give this error. Unlike the other plotters, it does *not* assume that your temp path is your output path if you only specify the temp path. So you’d use:

./bladebit -n 3 -f <farmerkey> -c <poolcontract> diskplot -t1 /nvme/tmp/ --cache 16G /nvme/plots/

instead. 

The gigahorse cuda plotter error:

With GPU-enhanced plotting now available in released (binary-only) code from Madmax, I decided to throw a modern GPU into my T7910, repair the post-22.04-upgrade mount failures, and give it a try. 

As a reminder from previous posts, this is a dual E5-2650L v4 system with 128GB RAM and 4x 1TB NVMe on the Ultra Quad card. It boots from a 256GB NVMe drive on a PCIe card, and has 4x 8TB SAS drives that don’t seem to be recognized after a few months off. Probably a SATA controller or cable issue, but life goes on. 

So I put one of my RTX 3060 LHR cards in, fixed up the NVMe stuff a bit, and went to run cuda_plotter_k32. It should do the partial memory plot, but alas, I got an error:

Invalid -r | --ndevices, not enough devices: 0

The card showed up in lspci, but then I realized it needed NVIDIA drivers. So I installed the 530 server bundle and tools, and then the plotter worked. 

Alas, the first GPU enhanced plot seems to have wedged the machine against interactive use. Looks like that’s a memory issue that I’ll have to work out, probably by adding memory. 

I will update this with further stats, and maybe make a comparison chart, as testing progresses. I’m also giving serious thought to upgrading the SSD in the dual-E5 machine. 

Obligatory disclosure:

While I work for Supermicro at the time of this writing, the servers and all other elements of my home labs and workspaces are my own and have no association with my employer. This post is my own, and my employer probably doesn’t even remember I have a blog, much less approve of it. 

16 thoughts on “A quick Chia plotting note and a future searchable tip”

  1. Hi Robert, no role for your r2208lt2 quad xeon server? I have to say I’m a bit confuzzled by this. Such a beast is a terrible thing to waste.

    I haven’t gotten my motherboard running yet, but moving in the right direction. I also just picked up a dual lga3647 board which seems to jabe loads of promise.

    Like

    Reply ↓

      • I’ve sourced some cheap 6 pipes heatsinks so hopefully the noise factor will be minimal. I don’t have the Intel chassis. This will be a real Frankenputer when it’s done.

        Like

      • I did just break down and order a set of E5-4620v2 chips for it, and a memory upgrade for my Cascade Lake uniproc box (the E403 mentioned above). There may be news soon on one or both fronts.

        Like

  2. That’s good news. But please understand I wasn’t actually disturbed by the fact that you weren’t currently employing it. That was sort of tongue in cheek. It’s an interesting system. A bare bones unit is available from the UK pretty cheap, but shipping would be a catastrophe. I’ll make the most of what I got here. Got the new baby anyway, arguably a lot more interesting.

    Prices for e5-4600 v2s have come down a lot. I had said to myself the system would have loads of novelty. Who runs internet browser or maybe a game on 32 cores (ok we’ve come a long way since I bought this). To me it’s a curio more then anything. But since I like server hardware, one of these bad boys will be relegated to more or less daily service. The electric bill be damned!

    Like

    Reply ↓

    • I understand completely. I ended up with just under $20/chip shipped for my set of four. It has been just taking up space so far, but since I’ve reduced my power load (less use for GPUs for some reason) so I can warm it up. And I think I have enough RAM to bump it up to 384GB (or at least 192GB).

      Like

      Reply ↓

      • If you should happen to have any PC133 ECC ram laying around, yes you read it right, let me know. Something tells me the answer is a big no. Another server project of mine requires it, albeit an exextremely vintage one.

        Like

  3. It will be so foolish to continue investing in Chia where tens of thousands of dollars investment has not returned even a couple of hundred dollars in almost a year. It is a big failure and if you think not. Then buy everything I have in the name of Chia.

    With thanks and best regards, Syed Ali.

    Confidentiality Notice: This communication and all attachments are only for the person(s) named above. Unless otherwise indicated, it contains information that is confidential, privileged or exempt from disclosure under applicable law. If you are not the person(s) named above, or responsible for delivering it to that person(s), be aware that disclosure, copying, distribution, or use of this communication and all attachments is strictly PROHIBITED. If you have received this communication in error, or are uncertain as to its proper handling, please immediately delete this communication and all attachments and all copies and notify us by replying to this email address or by phone.

    Like

    Reply ↓

      • Is Chia a type of cryptocurrency? If the question overall constirutes the dumbest thing said all day, all I’ll say is rah rah rah rah!.I won the crown! I’ve only skimmed Robert’s blog for the most part. I just resubscribed under a “new” addy, so please be patient with me. His blog attracted me initially because he was the only person I could find that actually owned an Intel S2600LT2HKC4.

        Like

      • O man peep gettin all nasty. Keep it lite boys.

        Ok I educated myself, and was right. Chia or chio is digital curremcy, and utilizes a LISP variant. My goodness is that a blast from the past. Maybe there’s a blockchaim that employs PROLOG out there.

        Like

      • Syed, I’m not the one soiling myself irrelevantly on someone else’s blog. If you invested tens of thousands of dollars, without knowing what Chia was about or paying attention to Bram’s initial and unambiguous advice, maybe soil your own blog first.

        Like

  4. At the time you tested this, I think the Gigahorse plotter was still needing 256GB of RAM.
    There are now options involving shifting some of the temp files onto 1 or 2 SSDs to allow for 128GB or 64GB or RAM usage – at some hit to performance.

    Like

    Reply ↓

    • The one Gigahorse test I did for this blog was on a 128GB RAM system (the T7910). I used the Dell Ultra Quad with four 1TB PCIe 3.0 x4 NVMe drives (Samsung PM961 maybe? Had them for almost two years now) for -t and -2.

      I think I’ve done a couple of 256GB system plots but those machines have very little storage for plots, so I don’t really bother.

      I have plotted on my 64GB desktop (painfully) since then though.

      Like

      Reply ↓

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.