Experimenting with Intel Optane at home with the Intel NUC 7th Generation PC

Welcome back to rsts11 for the summer. We’ve got a lot to cover in the next few weeks.

I haven’t really done a build report in a while, so when I realized I was getting double-dinged for high power usage, I started looking around for ways to save power. One was my desktop PC, which while very nice (with 8 dimm slots and lots of features I don’t use), is using around 250-300W for a 3rd gen core i7 processor.

I decided, based on availability and curiosity, to build out a 7th gen Intel NUC (Next Unit of Computing) PC, which conveniently supports Intel Optane memory. You can read a lot about the Optane technology, but in this application it’s a turbo-charged cache for internal storage. The newer NUCs support it in place of a more conventional m.2/NVMe SSD (used alongside a 2.5″ SSD or HDD), and of course you can use it as an overpriced SSD if you don’t want to use the Optane software.

See my earlier post about an Intel NUC for use with VMware. That NUC is currently running Ubuntu and Splunk for training in the home lab.

I’ll take you through the build manifest and process, and then we’ll look at benchmarks for five configuration permutations.

Build manifest and current prices (July 6, 2018)

  • Intel NUC (NUC7i7BNH) tall mini PC, $450 at Amazon
  • (Optional: NUC kit with preinstalled 16GB Optane module, $489 at Amazon)
  • Intel Optane Memory flash module (16GB $34 – $39 at Amazon, 32GB $58 for Prime members or $72 otherwise at Amazon)
  • Crucial CT2K16G4SFD824A 32GB DDR4 memory kit is currently $310 (it was $172 when I bought it a year and a half ago, ouch).
  • HGST Travelstar 7K1000 1TB 7200rpm SATA drive is $57.
  • Seagate FireCuda 2TB SSHD is $92, with the 1TB version available for $60.
  • Keyboard, mouse, USB flash drive for Windows install, and living room television with HDMI were already in house, but if you’ve read this far, you probably have them and/or know how to choose them. After installation you can use a Logitech Unifying device or a Bluetooth device, but for installation I’d suggest a USB cabled device.
  • Windows 10 Professional can be had for $150 give or take. The actual software can be downloaded from Microsoft but you will need a license key if building a new system without entitlement.

You’re looking at about $1,000 for the full system at today’s prices. If you don’t need 32GB of RAM, stepping down to 16GB should save you at least $100.

Building the NUC

I’ve used Intel NUCs going back to the 3rd gen Intel core processors, so they’re fairly easy to get around. The system I chose, NUC7i7BNH, has the following features:

  • Intel Core i7-7567U processor (2 cores, 4 threads, 3.5-4.0GHz, 4MB cache, 28W TDP)
  • two DDR4 sodimm slots (max 32GB)
  • storage slots:
    • an M.2 (PCIe x4 2242/2280)
    • a 2.5″ SATA (SSD/HDD)
    • a MicroSDXC storage slot exposed on the side (unused by me so far)
  • integrated Intel AC8265 + Bluetooth 4.2 wireless module (unused by me)
  • Gigabit Ethernet
  • Thunderbolt 3 / USB-C (usable for any USB function, Displayport, Thunderbolt 3)
  • four available USB 3.0 ports (including a rapid charge port on the front)
  • two USB 2.0 headers on the board (more on that later)
  • dual microphones for Windows Hello etc
  • two configurable LED indicators on the front panel

I loaded it up with a 2400MHz 32GB DDR4 kit from Crucial. You can get by with 2133, but this was what I had on hand. You can probably get by with less than 32GB, but only you know for sure.

Likewise, I probably did a bit of overkill with the 32GB Optane module, and a 2TB 2.5″ Seagate FireCuda gaming drive (2TB spinning disk plus onboard flash memory). I do tend to keep a lot of downloads locally, including my entire Dropbox folder, and want to have the best acceleration possible.

Notably, the current generation of NUC doesn’t require you to buy a localized power cable; once you have RAM, storage, and a keyboard/mouse/display, you’re good to go.

Getting Optane drivers ready to go

Any of you who built systems in the first decade of SATA will get flashbacks from this, but you will probably need to install additional drivers to make use of Optane. Even with the 1803 release of Windows 10 (downloaded in the last three days from Microsoft), an Intel Rapid Storage Technology (RST) driver is required, as is the Optane application to enable/monitor/disable Optane technology.

Intel does have good documentation (pdf) on this, whether adding Optane to an existing Windows 10 installation, or installing fresh. You obviously need a motherboard and CPU that support the Optane technology. This will be 7th gen or newer Core processors, and motherboards that explicitly call out Optane support (just having an m.2 slot isn’t enough).

Five configuration options

I tested five storage options to get a feel for relatively objective performance.

The benchmark software I used included current versions as of July 5, 2018:

PCMark10 from Underwriters Labs – an application-style benchmark simulating various regular PC tasks
NovaBench from Novawave – a more component-based test that delves into applications a bit, but less than PCMark10
Crystal DiskMark – A pure storage test, using sequential and random read and write tests

Note that the results are for comparison only, and may not be validated with the benchmark sites. They are definitely not endorsed or even known by the benchmark software makers.

The storage options included:

1a. Travelstar 7200RPM 1TB SATA drive, no Optane
1b. Travelstar 7200RPM 1TB SATA drive, Optane enabled
2a. FireCuda SSHD 2TB SATA drive, no Optane
2b. FireCuda SSHD 2TB SATA drive, Optane enabled
3. Optane module itself (Crystal Diskmark test alone)

Tests 1a/1b were performed on the same OS install (fresh with no updates, offline, no network connectivity). Tests 2a/2b/3 were done on a new OS install, again no updates or network connectivity.

Results

Note that benchmarks are often worth less than the paper they’re printed on. I like to use them for comparison within a system over time, or to get some level of comparison between systems, but depending on what you’re doing, your results may be much better or much worse than these.

Novabench tests

Unsurprisingly, CPU, RAM, and GPU tests didn’t show a distinction between the four storage options. Also unsurprisingly, disk performance with Optane was much better than without. I was a bit surprised that unaccelerated writes were better on the standard 7200rpm disk compared to FireCuda.

NovaBench (graph above) showed 2-4x better writes with Optane, and 7-9x better reads, roughly speaking. PCMark10 (graph below) had less shocking results, but Optane improved the 7200rpm disk by 20% on “Essentials,” but only 3% on Digital Content Creation. With FireCuda, Optane only improved Essentials by 5% or so, and the other two tests were similar to the 7200rpm tests

PCMark10 tests

CrystalDiskMark showed similar jumps in performance, with sequential reads going up by 10x and writes 3x. The results for the 4KiB queued tests were absurdly higher, with the worst results being 250x higher on reads and 98x faster on writes.

You’ll notice that on some of the rows, the 7200rpm and FireCuda drives don’t even show up. The random read and writes came in around 1 MB/sec for each, whereas with acceleration via Optane, they came in 93 MB/sec at worst, 621 MB/sec at best.

CrystalDiskMark results

 

I also ran Crystal DiskMark against the Optane disk as an SSD with NTFS filesystem (disabled Optane technology with the app, and then partitioned/formatted with Disk Management in Windows). Effectively this was the result of an NVMe SSD; I would probably not bother with a 32GB SSD for Windows, but I wanted to see how those results would compare with an accelerated HDD.

Turns out, with the exception of the 4KiB Q8T8 read (which was 49% faster than the accelerated FireCuda and 74% faster than the accelerated 7200rpm disk), results were between 2% slower and 9% faster than the accelerated FireCuda.

Give your NUC a hat?

There are two USB 2.0 headers on the NUC motherboard that don’t go anywhere. There are no extra holes in the case to let you use them anywhere either. However, if you don’t plan to stack your devices and want an extra USB option or two, there are companies like GoRite who make feature lids for the 5th generation and later NUCs. Some of the options include a supplemental fan, USB 2.0 ports, audio, video (HDMI, DVI, VGA), external wifi antennas, serial ports, memory card readers, Fast Ethernet, TV tuners, and more. You can also just pick a different color for the lid if that’s all you need.

So where do we go from here?

I’ll be putting some time this weekend into migrating my files and apps to the new system so I can retire/migrate my power-hungry desktop.

The testing I’ve done, while rather generic and not exhaustive, definitely shows that an Optane module will improve your disk i/o performance. An SSD (whether M.2 or 2.5″ or both) will still be the best performance choice, but I was able to get a 2TB SSHD and the 32GB Optane module for less than the price of a good 1TB SSD.

Do you need the 32GB module? Maybe not, but for a larger spinning disk it may be worth it. You’ll see similar performance improvement with a 16GB module, though, so if you need the extra $30 for something else, go for it.

Have you tried Optane yet? If so, let us know what you think in the comments.

Update after two weeks use

I wrote this post, and sat on it for about two weeks so I could see how the Optane acceleration felt during my regular usage pattern over time.

The first couple of days were painful. Switching browser tabs would often lock up, and boot-up was not as fast as a pure SSD. I ordered an M.2 NVMe SSD in case I needed to scrap the Optane experiment, but as time went on, it got more stable.

I decided to disable Optane caching while I synced half a terabyte of Dropbox data. This may have been more superstition than actual benefit, but I figured the caching would not help me on sustained write, and I bought myself another write cycle in the process.

I later realized that part of the degradation may have been from a radical core count reduction. My previous system was a 3rd gen core i7 with 6 cores/12 threads, whereas this one is a 7th gen core i7 with 2 cores/4 threads. RAM is the same quantity (was 8x4GB DDR3, now 2x16GB DDR4), and I’d say the storage is a definite upgrade.

As I wrap up this post, I’ve been running on the Optane system for about 16 days, and it’s quite livable. I did export some of the peripheral connections to an OWC USB-C 10-port dock, which provides some additional charging capability as well as getting some of the USB-A connections off the NUC itself. I’ve plugged my Drobo into the dock, along with my Fitbit Ionic charger and a couple of other items. I could also run a second monitor this way (or even a third), but that would require some more desk space that I’m running short on.

All things considered, it’s a very viable desktop system, and I’m mostly happy with the performance and capacity. It will shortly be mounted on the back of my monitor (to help with airflow and free up some shelf space). And I am a bit tempted to replace the Optane with the ADATA NVMe drive I bought, but I think that may do better in another tiny PC that can’t use Optane (like an HP Elitedesk 800 G2 Mini I have in the home office).

Advertisements

NUC NUC (Who’s there?) VMware lab…

nuc-outside

VMware lab who? VMware lab in pocket-size format!

So in our last installment, I found out that I can upgrade my Shuttle SH67H ESXi servers to support Ivy Bridge processors. If you want to read more about that, feel free to visit my Compact VMware Server At Home post from Winter 2012, and my Upgrading my home VMware lab with Ivy Bridge post from Spring 2013.

The replacement boards came in from Shuttle, and they’ll be going back into the chassis. But as you may have seen at the end of the last post, I discovered the Intel Next Unit of Computing server line. The NUC line current includes three models.

  • DC3217IYE – i3-3217U processor at 1.8 GHZ dual core with 3MB cache), dual HDMI, Gigabit Ethernet at $293 (pictured)
  • DC3217BY – i3-3217U processor, single HDMI, single Thunderbolt,  – no native Ethernet – at $323
  • DCCP847DYE– Celeron 847 (1.1 GHZ dual core with 2MB L3 cache, dual HDMI, Gigabit Ethernet at $172
    (Prices are estimated list from Intel’s site–probably cheaper by a buck or ten at Amazon, Fry’s, Central Computer, or your favorite retailer. Feel free to use my links and help me buy the next one. 🙂 )

nuc-inside

All three have three USB 2.0 ports outside (one front, two rear), as well as two USB headers inside, conceivably useful for a USB flash drive or flash reader. They also have an mSATA-capable Mini-PCIe slot as well as a short mini-PCIe slot suitable for a WiFi/Bluetooth card. And there are two DDR3 SODIMM slots, supporting a reported 16GB of RAM (the processor supports 32GB, but the board/kit do not mention this). They all include VT-x with EPT.

I don’t see the Celeron being that useful for virtualization labs, but these are rather multifunctional for a little 4″ square computer. Imagine putting a broadband modem (3G/4G/Wimax) inside for reasonably potent portable kiosk purposes (VESA mount kit included). A card reader and a DVD burner for saving and sharing (and even editing) photos. Intel’s WiDi wireless display technology is supported as well, if you have a suitable receiver. Or use it with a portable projector for presentations on the go (no more fiddling with display adapters for presentations at your meetings!).

But we’re talking about a VMware lab here.

And let me get this out of the way… this was one of the coolest features of the NUC.


That’s correct, the box has its own sound effects.

Let’s get this party started…

Those of you of my era and inclinations may remember when KITT’s brain was removed and placed in a non-vehicle form factor on the original Knight Rider tv series. When I got ready to RMA my Shuttle motherboards, I was thinking about this sort of effect for a couple of VMs on the in-service ESXi server that was about to have its board sent to southern California. And that’s what I had to do. I couldn’t quite miniaturize the server Orac-style, but  that thought had crossed my mind as well.

So I picked up the DC327IYE unit at Fry’s, got an mSATA 64GB card (Crucial m4 64GB CT064M4SSD3) and a spare low profile USB flash drive (Patriot Autobahn 8GB (PSF8GLSABUSB)) at Central Computers, and took a Corsair 16GB DDR3 Kit (CMSO16GX3M2A1333C9) from my stock. Assembling it took only a few minutes and a jeweler’s screwdriver, and then I was ready to implement ESXi.

I backed up the VMs from the original system using vSphere Client, so that I could re-deploy them later to the NUC. Someday I’ll get Veeam or something better going to actively back up and replicate my VMs, but for the limited persistent use of my cluster (cacti and mediawiki VMs) this was sufficient.

One gotcha: Fixing the NUC network…

I originally tried reusing the 4GB usb drive my existing server was booting from, but it didn’t recognize the Ethernet interface. I installed a fresh 5.0u2 on a new flash drive, and still no luck. I found a post at tekhead.org that detailed slipstreaming the new driver into ESXi’s install ISO. I did so, installed again, and was up and running.

I did have to create a new datastore on the mSATA card — my original server had used a small Vertex 2 SSD from OCZ, which obviously wouldn’t work here. But I was able to upload my backed up OVF files and bring up the VMs very quickly.

And one warning I’ll bring up is that the unit does get a bit warm, and if you use a metal USB flash drive, it will get hot to the touch. My original ESXi lab box used a plastic-shelled USB drive, and I’m inclined to go back to that.

What’s next, Robert?

My next step is going to be bringing central storage back. There is a new HP MicroServer N54L on the market, but my N40L should be sufficient for now–put the 16GB upgrade in and load it up with drives. As those of you who saw my lab post last year know, it was running FreeNAS 8, but I’m thinking about cutting over to NexentaStor Community Edition.

I’ve taken the original Shuttle box and replaced a mid-tower PC with it for my primary desktop. I will probably set the other one up with a Linux of some sort.

And in a week or so I’ll grab a second NUC and build it out as a second cluster machine for the ESXi lab. All five of them are slated to go into my new EXPEDIT shelving thingie in the home office, and I’ll bring you the latest on these adventures as soon as they happen.

Upgrading my home VMware lab (part 1: Ivy Bridge) #rsts11

My most popular post on rsts11 has been my compact VMware server at home post. Thanks to Chris Wahl mentioning me on the VMware forums, and linking from his lab post, I see a dozen visits or more a day to that page.

Imitation is the sincerest form of laziness^wflattery

I have to admit that I’ve been a follower in my use of intriguing lab environments. I got the vTARDIS idea from Simon Gallagher, and built a version of it at work at my last job on a Dell Core 2 Quad workstation under my desk. Then I saw Kendrick Coleman tweet about this new SH67H3 from Shuttle that supported 32GB of non-registered RAM… bought one and put 32GB and an i7-2600S processor into it, as mentioned in the “server at home” post mentioned above.

Now as you may know, the i7-2600 series processors are now a generation behind. Sandy Bridge gave way to Ivy Bridge (the i7-3×00 processors) which are currently easily found at retail. But… SH67H3 v1.0 motherboards don’t support Ivy Bridge. And that’s what was shipping when I bought mine in early 2012.

I found an unbelievable deal on a second SH67H3 open (missing) box at Fry’s in February 2013… let’s just say I spent more on a basic Pentium chip to test it with than I did on the chassis itself. But alas, the second one also had a v1.0 motherboard.

Let’s make the ivy (bridge) grow!

I found sources on the Internets that said a v2.0 board supporting Ivy Bridge was out. I further discovered that Shuttle would trade in your v1.0 board for a v2.0 board for $40. Instructions here at Cinlor Tech’s blog if you’re interested in doing this yourself. Note that you can request the case number through Shuttle’s web-email portal if you prefer this to calling. That’s what I did.

sh67 corpsesI shipped off my two boards in a medium Priority Mail box to Shuttle on the 26th. On the 29th I got confirmation of the return shipment. They should be at my door on April 2nd. I’ll be reinstalling them, and at some point upgrading to the i7-3770s processors on both.

Waitasec, 2600 “S”? 3770 “S”? What’s this all about, then?

Yes, that’s correct. I chose to go with a low power version of the i7-2600 processor a year and change ago. The i7-2600s has a lower base speed than the 2600 or 2600k (unlocked version), 2.8ghz vs 3.4ghz. All three support turbo boost to 3.8ghz though. And the i7-2600s is 65W where the others are 95W.

(Here’s a comparison chart of the three i7-2600 and three i7-3770 processor options via Intel, if you’re curious.)

Other noteworthy differences are on the 2600k, which costs $20 more, but does not support VT-d (directed I/O), vPro management features, or Trusted Execution. VT-d is the only feature of particular concern when you’re building your virtualization lab though. (I’ll admit the VT-d was an accidental discovery–I chose the 2600s more for power savings than anything else). If you’re building a desktop, the “K” model has HD3000 graphics vs HD2000 for the other two chips, by the way.

Now that I’m building a second box, I find that my usual local retail sources don’t have the i7-2600s in stock anymore. I could order one on eBay or maybe find it at Fry’s, but for about the same price I could get the Ivy Bridge version and be slightly future-proofed. Once again, the “S” is the way to go.

The 3770 series run at 3.1ghz (“S”), 3.4ghz (3770), and 3.5ghz (“K”) base speeds, all turbo capable to 3.9ghz. The “S” processor is 65W again, vs only 77W for the other two chips. They all have Intel’s HD4000 integrated graphics and the newer PCIe 3.0 support. They support 1600mhz RAM speeds, vs 1333 top for the previous generation. The “K” processor lacks VT-d, vPro, and Trusted Execution, but does have a nearly $40 premium over the other two chips.

All six of these chips have VT-x including extended page tables (EPT/SLAT), hyperthreading, and enhanced SpeedStep. And they’re all 4 core/8 thread/32gb RAM capable processors that make a great basis for a virtualization environment.

nuc-scaleSo what’s next, Robert?

Well, with two matching machines, I’ll be basically starting from scratch. Time to upgrade the N40L Microserver NAS box to 16GB (Thanks Chris for finding this too!) and probably splitting off a distinct physical storage network for that purpose.

But now, thanks to Marco Broeken’s recent lab rebuild, I’ve been introduced to Intel’s Next Unit of Computing (NUC), so tune in soon for my experience with my first NUC system. Sneak peek of the ESXi splash screen and the actual unit here… stay tuned!