Micro-S2D

Come along for a journey trying to run S2D on two micro PCs — and why you really shouldn't.

Table of Contents

Preface

I know we all hate a long story before the recipe, but this one has some real explaining to do. Lucky you, I have a table of contents and you can skip over my woes.

First, this is not what this project was supposed to be and if I could do it all over again, I wouldn’t. I chose the Minisforum MS-01 to make a 2-node S2D cluster due to the NVMe slots, dual 20g thunderbolt, and dual 10g SFP+. I was hoping to use the dual 20g thunderbolt ports for storage & live migration, and the 10g X710 ports for Management & Compute. I even picked up some QNAP cards so I could do 5x NVMe per node.

Unfortunately, I could not get the thunderbolt ports to work for SMB Multichannel or Network ATC, so using them for storage was off the table. They can however come in handy as a way to RDP from one node to the other when messing with network changes, so they’re only nearly useless. Next problem, it took me months to get the X710s to behave. You really do need the latest firmware and drivers. Even with all of that, I can’t recommend the X710s for any storage traffic.

Additionally, I started off with a big lot of enterprise 1.92TB M.2 PM983 NVMe drives. The MS-01s refused to boot into an OS with the PM983s, regardless of which slot, how many drives, etc., because they can’t provide enough power to the M.2 slots to handle enterprise 22110 drives. I ended up getting Samsung 9100 Pros that are great for gaming, but really fall short for server workloads. They do not have Power Loss Prevention capabilities, so each write needs to get flushed all the way to NAND before S2D counts the write as complete. Because of this, once the 9100 Pro’s SLC cache fills up, the cluster basically falls over. High queues on the NVMe drives cause CPU waits, and the interrupts on an underpowered CPU start to grow exponentially. Intel E-cores don’t help here, since any storage or interrupt work that lands on them runs far slower than on the six P-cores.

So I eventually settled on putting a Mellanox ConnectX-4 Lx NIC in each node for the storage intent and put Compute and Management traffic on the X710. That of course meant no QNAP card, and only 3x SSD per node.

One more problem: these nodes have no real airflow for the NICs, and they started to overheat. I did some custom soldering and 3D printing, and actually ended up with some great cooling results on the NIC chip when using passive DACs.

Let me emphasize, nearly every part of this is unsupported, and even if it were, it doesn’t work all that great compared to real servers. I wouldn’t even recommend this for a homelab, unless you want a crash course in the deepest troubleshooting S2D has to offer.

The MS-01 nodes racked in two clusters, HV01/HV02 and HV03/HV04

Hardware Setup

Hardware

First up, the hardware I used:

Part Model
CPU i9-12900H (14c/20t)
RAM 2x 48GB DDR5 Crucial
Boot SSD Random old NVMe
Data SSD 2x Samsung 9100 Pro 1TB
NIC Mellanox ConnectX-4 Lx

Step 1 is of course to install everything. When it comes to SSD placement, make sure to put the boot drive in the slot furthest to the right. That slot only runs at PCIe 3.0x2 so we don’t want to use that for a real data drive. Also, make sure that the M.2/U.2 switch is set to M.2, or you’re going to have a really bad day.

Two MS-01s opened up, showing the M.2 SSD slots

Two MS-01s with a Mellanox ConnectX-4 Lx NIC installed

Updated NIC cooler:

3D-printed blower shroud over the ConnectX-4 Lx NIC chip, powered by leads soldered to the card

BIOS

Once they’re all put together, onto the BIOS Setup.

  • All of my nodes shipped with BIOS 1.26, which seemed to work fine for me, but if you bought used I’d make sure both nodes are on the same version.
  • If you bought used, go ahead and reset to factory defaults. If you bought new, maybe do it anyways. I’ve seen some weird things with these guys.
  • Advanced menu.
    • Trusted Computing:

      • Enable SHA384 and SM3_256 PCR Banks

        BIOS Trusted Computing page with the SHA256, SHA384 and SM3_256 PCR banks enabled

    • Onboard Devices Settings:

      • DVMT Pre-Allocated: 48M

      • Aperture Size: 128MB

        BIOS Onboard Devices Settings page with DVMT Pre-Allocated at 48M and Aperture Size at 128MB

    • ACPI Settings:

      • Restore (Restory?) on AC Power Loss: Last State

        BIOS ACPI Settings page with Restore on AC Power Loss set to Last State

    • HW Monitor & Smart Fan:

      • Set all fans to “Full Mode”. These boxes are still going to run plenty warm, and at full speed they’re still quieter than my other servers.

        BIOS PC Health Status page with every fan set to Full Mode

AD prep

Lots of this is optional, and environmental based. But this is what I did ahead of time to make things easier.

  • Create OU for HV Clusters, with a sub OU for this particular cluster.
  • Create and link GPO to allow RDP and Remote Powershell.
  • Create and link GPO to configure W32TM (NTP) settings.
  • Create and link GPO to use Delivery Optimization on the local network.
  • Create and link GPO to disable Interactive Logon CTRL+ALT+DEL
  • Create and link GPO to allow necessary Windows Firewall rules.
    • RDP
    • ICMP (Ping)
    • File and Printer Sharing (SMB)
    • WinRM
    • WMI
    • Delivery Optimization
    • Performance Logs and Alerts
    • Virtual Machine Monitoring

Initial OS Setup

I used the absolute latest version of Windows Server vNext, but feel free to use the retail release of Windows Server 25 if you’d like.

  • Prep Windows Server on a USB drive
  • Boot, and install Windows Server to the boot NVMe.
    • If you did vNext, you can use the public vNext activation key “2KNJJ-33Y9H-2GXGX-KMQWH-G6H67”
  • Set an admin password, and login.
  • Do a Rename-Computer and restart 
    Rename-Computer -NewName HV01 -Restart
    Rename-Computer -NewName HV01 -Restart
  • Join the Domain
    • Move the computer objects to the dedicated OU now, not later.
  • Restart the computer 
    Restart-Computer
    Restart-Computer
  • Set the timezone. 
    Set-TimeZone -Id "Central Standard Time"
    Set-TimeZone -Id "Central Standard Time"
  • Set minimum processor state to 25% on the High Performance plan 
    Powercfg -setacvalueindex 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c 54533251-82be-4824-96c1-47b60b740d00 893dee8e-2bef-41e0-89c6-b55d0929964c 25
    Powercfg -setacvalueindex 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c 54533251-82be-4824-96c1-47b60b740d00 893dee8e-2bef-41e0-89c6-b55d0929964c 25
  • Set Power Plan to High Performance (activating it after the change makes it take effect) 
    Powercfg -setactive 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c
    Powercfg -setactive 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c

Drivers

  • Download the Mellanox WinOF-2 drivers
    • You need the WinOF-2 driver, and use the latest 26.x version, not the 3.x LTS.
  • Download the Mellanox ConnectX-4 Lx firmware for your specific card
    • I know it’s a pain to figure out the part number, but you do want the latest firmware.
  • Download the MS-01 driver package
    • Scroll down to MS-01, expand “Drivers & Downloads” then click on the “Driver Package” link
    • This package actually has lots of fun unsupported stuff in it. It has an i226 driver that will actually install on Windows Server 2025 if you want to try that.
  • Download the Intel X710 WS2025 driver package
    • We are downloading this for the NVM Firmware, DO NOT INSTALL THE X710 DRIVERS.
    • You may have to scroll down for the “Intel® Network Adapter Driver for Windows Server 2025*” package
  • Install Mellanox drivers
  • Install Chipset Drivers (MS-01 #1)
  • Install Graphics Drivers (MS-01 #4)
  • DO NOT INSTALL THE SERVER 2025 X710 DRIVERS FROM INTEL. I tried all sorts of versions, they just don’t work. Let Windows Update give you the 1.10.128 version. From further testing, it appears that Intel’s Windows Server 2022 package works, and the Server 2025 package does not.
  • Update the X710 Firmware (NVMUpdate included in the X710 driver package)
  • Optionally install i226, ME, Etc.
  • Restart the computer 
    Restart-Computer
    Restart-Computer

Host Network Prep

This part is pretty important, and will help set you up for Failover Clustering health checks, and Network ATC Configuration.

  • Don’t forget to manually set the switchless IPs
NIC HV01 HV02 VLAN
vManagement 10.10.50.21/24 10.10.50.22/24 50
CX4-1 10.71.1.21/24 10.71.1.22/24 700
CX4-2 10.71.2.21/24 10.71.2.22/24 700

Cluster IP: 10.10.50.20

Rename Network Adapters (Sometimes optional, but nice)

You will need to substitute the proper -Name and -NewName depending on the order in which your adapters got named by default. Windows names NICs kind of randomly, as you can see in my screenshot. It is important to name the ports on the card by their MAC address order. 

Get-NetAdapter | Sort MacAddress

Rename-NetAdapter -Name 'Ethernet' -NewName 'CX4-1'
Rename-NetAdapter -Name 'Ethernet 2' -NewName 'CX4-2'
Rename-NetAdapter -Name 'Ethernet 3' -NewName 'X710-1'
Rename-NetAdapter -Name 'Ethernet 4' -NewName 'X710-2'

#Node1
New-NetIPAddress -IPAddress 10.71.1.21 -InterfaceAlias "CX4-1" -Prefixlength 24
New-NetIPAddress -IPAddress 10.71.2.21 -InterfaceAlias "CX4-2" -Prefixlength 24
New-NetIPAddress -IPAddress 10.10.50.21 -InterfaceAlias "X710-1" -Prefixlength 24 -DefaultGateway 10.10.50.1
#Node2
New-NetIPAddress -IPAddress 10.71.1.22 -InterfaceAlias "CX4-1" -Prefixlength 24
New-NetIPAddress -IPAddress 10.71.2.22 -InterfaceAlias "CX4-2" -Prefixlength 24
New-NetIPAddress -IPAddress 10.10.50.22 -InterfaceAlias "X710-1" -Prefixlength 24 -DefaultGateway 10.10.50.1

Get-NetAdapter -Name 'X710-1' | Set-DnsClientServerAddress -ServerAddresses ("10.10.50.51","10.10.50.52","10.10.50.53")
Get-NetAdapter | Sort MacAddress

Rename-NetAdapter -Name 'Ethernet' -NewName 'CX4-1'
Rename-NetAdapter -Name 'Ethernet 2' -NewName 'CX4-2'
Rename-NetAdapter -Name 'Ethernet 3' -NewName 'X710-1'
Rename-NetAdapter -Name 'Ethernet 4' -NewName 'X710-2'

#Node1
New-NetIPAddress -IPAddress 10.71.1.21 -InterfaceAlias "CX4-1" -Prefixlength 24
New-NetIPAddress -IPAddress 10.71.2.21 -InterfaceAlias "CX4-2" -Prefixlength 24
New-NetIPAddress -IPAddress 10.10.50.21 -InterfaceAlias "X710-1" -Prefixlength 24 -DefaultGateway 10.10.50.1
#Node2
New-NetIPAddress -IPAddress 10.71.1.22 -InterfaceAlias "CX4-1" -Prefixlength 24
New-NetIPAddress -IPAddress 10.71.2.22 -InterfaceAlias "CX4-2" -Prefixlength 24
New-NetIPAddress -IPAddress 10.10.50.22 -InterfaceAlias "X710-1" -Prefixlength 24 -DefaultGateway 10.10.50.1

Get-NetAdapter -Name 'X710-1' | Set-DnsClientServerAddress -ServerAddresses ("10.10.50.51","10.10.50.52","10.10.50.53")

Get-NetAdapter sorted by MAC address, before and after renaming the ports CX4-1, CX4-2, X710-1 and X710-2

Cluster Configuration

  • First, we’re going to check that the data drives are ready to be pooled. At this point, you will have to run this on each node. 
    Get-PhysicalDisk
    Get-PhysicalDisk

Get-PhysicalDisk on one node: the boot drive shows CanPool False, the two data SSDs CanPool True

  • If any of your drives are listed as “CanPool=False” then you can run Microsoft’s command to clean them.
  • Add Roles and Features. You 100% want to do this from powershell, not server manager. 
    Install-WindowsFeature -Name "Hyper-V", "Failover-Clustering", "Data-Center-Bridging", "RSAT-Clustering-PowerShell", "Hyper-V-PowerShell", "FS-FileServer", "NetworkATC" -IncludeAllSubFeature -IncludeManagementTools -Restart
    Install-WindowsFeature -Name "Hyper-V", "Failover-Clustering", "Data-Center-Bridging", "RSAT-Clustering-PowerShell", "Hyper-V-PowerShell", "FS-FileServer", "NetworkATC" -IncludeAllSubFeature -IncludeManagementTools -Restart
  • FCM Validation:

    • Open Failover Cluster Manager. If you did server core, you will need to do this from a management machine.

    • In the top right, click “Validate Configuration”

      Failover Cluster Manager with Validate Configuration in the Actions pane

    • It will take a few minutes to validate, and provide a report. There may be some errors or warnings, and you should view the report and check them all.

      The cluster validation summary report

    • Assuming everything else in the validation report looks good, click the “Create the cluster now using the validated nodes” box, and click finish.

    • Fill in the name and IP you’d like to provide the cluster, click next a couple of times, and click create.

    • From the cluster overview page, go to “Cluster Core Resources”, and right click on the “Server Name” object with your cluster’s name. Go to properties, Select the network address and click edit, and make sure everything looks OK. Optionally also select “publish PTR records” and click OK again.

      Cluster Name properties in Failover Cluster Manager, showing the cluster IP address and the Publish PTR records option

Network ATC

  • This section assumes you are using the same exact network setup as me, but this will hopefully at least give you a good starting point to understand ATC. You only have to do this on one node, and it will apply to the whole cluster.
  • First, create the intent 
    $MgmtAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$MgmtAdapterPropertyOverrides.NetworkDirect = 0 
$MgmtAdapterPropertyOverrides.JumboPacket = 1514 
$StorAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$StorAdapterPropertyOverrides.JumboPacket = 9014
#RoCEv2=4, iWARP=1
$StorAdapterPropertyOverrides.NetworkDirectTechnology = 4
$StorageOverride = New-NetIntentStorageOverrides 
$StorageOverride.EnableAutomaticIPGeneration = $false

Add-NetIntent -Name Compute_Management -Management -Compute -AdapterName X710-1, X710-2 -ManagementVlan 50 -AdapterPropertyOverrides $MgmtAdapterPropertyOverrides

Add-NetIntent -Name Storage -Storage -AdapterName CX4-1, CX4-2 -AdapterPropertyOverrides $StorAdapterPropertyOverrides -StorageOverrides $Storageoverride
    $MgmtAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$MgmtAdapterPropertyOverrides.NetworkDirect = 0 
$MgmtAdapterPropertyOverrides.JumboPacket = 1514 
$StorAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$StorAdapterPropertyOverrides.JumboPacket = 9014
#RoCEv2=4, iWARP=1
$StorAdapterPropertyOverrides.NetworkDirectTechnology = 4
$StorageOverride = New-NetIntentStorageOverrides 
$StorageOverride.EnableAutomaticIPGeneration = $false

Add-NetIntent -Name Compute_Management -Management -Compute -AdapterName X710-1, X710-2 -ManagementVlan 50 -AdapterPropertyOverrides $MgmtAdapterPropertyOverrides

Add-NetIntent -Name Storage -Storage -AdapterName CX4-1, CX4-2 -AdapterPropertyOverrides $StorAdapterPropertyOverrides -StorageOverrides $Storageoverride
  • Next, we need to get the status and make sure it’s succeeded. DO NOT CONTINUE UNTIL THIS IS COMPLETE 
    Get-NetIntentStatus
    Get-NetIntentStatus

Get-NetIntentStatus output with every intent showing Success

  • If you’re looking for some alternate Network ATC setup configurations, Lee has a great blog post here, and I’ve written my own here: Network ATC 101
  • If you’re feeling really adventurous, and need to get the X710s to work for storage, Network ATC will fail because they do not support RDMA. This is the same problem you’d have trying to use Network ATC in a VM or with any other non-RDMA NICs. Here are the necessary overrides to make it work. DO NOT DO THIS IF YOU HAVE RDMA CAPABLE NICS. 
    $Override = New-NetIntentAdapterPropertyOverrides
$Override.JumboPacket = "9014"
$Override.NetworkDirect = $false

Add-NetIntent -Name Storage -Storage -AdapterName X710-1, X710-2 -AdapterPropertyOverrides $Override
    $Override = New-NetIntentAdapterPropertyOverrides
$Override.JumboPacket = "9014"
$Override.NetworkDirect = $false

Add-NetIntent -Name Storage -Storage -AdapterName X710-1, X710-2 -AdapterPropertyOverrides $Override

Set Up Cluster Aware Updating (CAU)

  • First step here is often overlooked, and VERY important. Give the Cluster Name Object (CNO) full control of its OU!
  • These are the parameters I chose to use for this lab cluster. I expect most of you to somewhat modify these settings. 
    $Parameters = @{
    ClusterName = 'HV-CLUS20'
    DaysOfWeek = 'Monday', 'Friday'
    WeeksOfMonth = '1', '2', '3', '4'
    MaxFailedNodes = '0'
    MaxRetriesPerNode = '3'
    RebootTimeoutMinutes = '30'
    SuspendClusterNodeTimeoutMinutes = '30'
    SuspendRetriesPerNode = '2'
    WaitForStorageRepairTimeoutMinutes = '60'
    RequireAllNodesOnline = $true
    AttemptSoftReboot = $true
    EnableFirewallRules = $true
    Force = $true
    }
Add-CauClusterRole @Parameters 
    $Parameters = @{
    ClusterName = 'HV-CLUS20'
    DaysOfWeek = 'Monday', 'Friday'
    WeeksOfMonth = '1', '2', '3', '4'
    MaxFailedNodes = '0'
    MaxRetriesPerNode = '3'
    RebootTimeoutMinutes = '30'
    SuspendClusterNodeTimeoutMinutes = '30'
    SuspendRetriesPerNode = '2'
    WaitForStorageRepairTimeoutMinutes = '60'
    RequireAllNodesOnline = $true
    AttemptSoftReboot = $true
    EnableFirewallRules = $true
    Force = $true
    }
Add-CauClusterRole @Parameters
  • Then, we’ll check that it’s working. 
    Get-CauClusterRole
    Get-CauClusterRole

Get-CauClusterRole output with the CAU role Online

  • Assuming it says Online, and the settings look right, you should be good. I did however run into an odd issue a few days after deployment. My CAU status went to Offline. Thankfully, super easy to check, and super easy to fix. 
    Get-CauClusterRole
Enable-CauClusterRole
    Get-CauClusterRole
Enable-CauClusterRole

Get-CauClusterRole showing the CAU role Offline, then Enable-CauClusterRole bringing it back Online

Set Cluster Witness

  • Since this is a 2-node cluster, we definitely want a cluster witness. This is super easy, you just need to point it at an SMB share that both nodes can write to. 

    Set-ClusterQuorum -Cluster HV-CLUS20 -FileShareWitness \\SERVER-1\HV-CLUS20 -Credential (Get-Credential)
Get-ClusterQuorum -Cluster HV-CLUS20
    Set-ClusterQuorum -Cluster HV-CLUS20 -FileShareWitness \\SERVER-1\HV-CLUS20 -Credential (Get-Credential)
Get-ClusterQuorum -Cluster HV-CLUS20

  • You should then check that the cluster witness shows as “online” in FCM.

    Failover Cluster Manager showing the file share witness Online

  • Or better yet, use a Cloud Witness! 

    Set-ClusterQuorum -CloudWitness -AccountName <StorageAccountName> -AccessKey <StorageAccountAccessKey>
    Set-ClusterQuorum -CloudWitness -AccountName <StorageAccountName> -AccessKey <StorageAccountAccessKey>

Set Up S2D

  • Finally the fun part! 
    Enable-ClusterStorageSpacesDirect -PoolFriendlyName 'MICRO-S2D' -Verbose
    Enable-ClusterStorageSpacesDirect -PoolFriendlyName 'MICRO-S2D' -Verbose
  • And check that the storage pool is online and healthy. 
    Get-StoragePool
    Get-StoragePool

Get-StoragePool output with the pool healthy

  • Or if you want more details: 
    Get-StoragePool -IsPrimordial $false | FL
    Get-StoragePool -IsPrimordial $false | FL

Full details of the S2D storage pool from Get-StoragePool

  • Validate that the ClusterPerformanceHistory volume has been created. This can take up to ~15 minutes. 
    Get-Volume
    Get-Volume

Get-Volume output including the ClusterPerformanceHistory volume

S2D Volume Creation

  • First step here is really to understand how much space is available in your pool.
  • Since I used 2x1TB SSD per node, I’m going to “lose” one drive worth of capacity per node to the recommended reserve, then I’m using 2-way mirroring across the two nodes, leaving me with just under 1TB usable, but I don’t want to run at 100% right out of the gate, so I’m going with a couple of 300GB CSVs (Cluster Shared Volumes) to start. 
    $StoragePool = Get-Storagepool -IsPrimordial $False
New-Volume -StoragePool $StoragePool -FriendlyName CSV1 -FileSystem CSVFS_ReFS -Size 300GB -ResiliencySettingName "Mirror" -ProvisioningType "Fixed"
New-Volume -StoragePool $StoragePool -FriendlyName CSV2 -FileSystem CSVFS_ReFS -Size 300GB -ResiliencySettingName "Mirror" -ProvisioningType "Fixed"
    $StoragePool = Get-Storagepool -IsPrimordial $False
New-Volume -StoragePool $StoragePool -FriendlyName CSV1 -FileSystem CSVFS_ReFS -Size 300GB -ResiliencySettingName "Mirror" -ProvisioningType "Fixed"
New-Volume -StoragePool $StoragePool -FriendlyName CSV2 -FileSystem CSVFS_ReFS -Size 300GB -ResiliencySettingName "Mirror" -ProvisioningType "Fixed"

Hyper-V Tweaks

  • Create (my) standard folder structure. 
    New-Item -Path C:\ClusterStorage\CSV1\VMs -ItemType Directory
New-Item -Path C:\ClusterStorage\CSV1\VHDs -ItemType Directory
New-Item -Path C:\ClusterStorage\CSV1\ISOs -ItemType Directory
    New-Item -Path C:\ClusterStorage\CSV1\VMs -ItemType Directory
New-Item -Path C:\ClusterStorage\CSV1\VHDs -ItemType Directory
New-Item -Path C:\ClusterStorage\CSV1\ISOs -ItemType Directory
  • Set default locations for VM Creation. 
    Get-ClusterNode | Foreach { Set-VMHost -ComputerName $_.Name -VirtualMachinePath 'C:\ClusterStorage\CSV1\VMs' }
Get-ClusterNode | Foreach { Set-VMHost -ComputerName $_.Name -VirtualHardDiskPath 'C:\ClusterStorage\CSV1\VHDs' }
    Get-ClusterNode | Foreach { Set-VMHost -ComputerName $_.Name -VirtualMachinePath 'C:\ClusterStorage\CSV1\VMs' }
Get-ClusterNode | Foreach { Set-VMHost -ComputerName $_.Name -VirtualHardDiskPath 'C:\ClusterStorage\CSV1\VHDs' }
  • Increase Failover Cluster load balancer aggressiveness (Yes, this is actually the valid powershell way to set this…) 
    (Get-Cluster).AutoBalancerLevel = 2
    (Get-Cluster).AutoBalancerLevel = 2
  • Set maximum parallel migrations. Don’t ask me why this isn’t covered with all the other things we’ve set. 
    (get-Cluster).MaximumParallelMigrations = 3
    (get-Cluster).MaximumParallelMigrations = 3

Double checks

I had plenty of weird things going on with WAC Network ATC on this deployment. I blame it on using the latest vNext version of Windows Server, latest preview version of WAC, and some other unnamed new features. This is basically a list of good things to double check.

  • Double check that Network ATC extension works in WAC. Mine did not, and I had to add a different extension feed to get a newer version of Network ATC than what was in the default feed.
    • WAC > Settings > Extensions > Feeds > Add https://aka.ms/sme-extension-feed
    • Install the newest Network ATC Extension.
  • Double check Network ATC Global Overrides. I had some funky stuff with my Network ATC Powershell module, so I double checked everything looked good in WAC.

    • WAC > Cluster > Network ATC Cluster Settings

      Network ATC cluster settings in Windows Admin Center

  • Check SMB Multichannel 
    Get-SmbMultichannelConnection -SmbInstance SBL
    Get-SmbMultichannelConnection -SmbInstance SBL

Cluster tuning

  • Set CPU Scheduler to the newer Core Scheduler.

    • Powershell: 

      bcdedit /set hypervisorschedulertype Core
      bcdedit /set hypervisorschedulertype Core

    • WAC > Cluster > Settings > General

      Windows Admin Center cluster settings with the Hyper-V core scheduler selected

  • Enable In-Memory read cache. I have enough RAM, so I’m happy to give some up to in-memory read cache. This is basically what every linux ZFS system is doing with ARC.

    • WAC > Cluster > Settings > In-memory Cache

      In-memory cache settings in Windows Admin Center

  • Set Storage Spaces Repair Speed

    • WAC > Cluster > Settings > Storage Spaces and Pools

      Storage Spaces repair speed setting in Windows Admin Center

Handy tasks

  • Getting rid of removed adapters showing up as partitioned in FCM

    • The docs claim that you can use Add-ClusterExcludedAdapter commands to fix this, but I have never been able to get them to work. Here’s the actual fix:
    • Open Regedit, navigate to “Computer\HKEY_LOCAL_MACHINE\Cluster\NetworkInterfaces" and delete all references to the removed adapter. Having good friendly names will help here.

  • Affinity rules vs Preferred Owners

    • Affinity rules can be configured in WAC or Powershell, and allow you to try and keep two or more VMs together or apart. For most production VMs, this is what you probably want to use. You can also enable “soft anti-affinity” so if you don’t have enough nodes in the cluster to keep them apart, it will keep them running. Perfect use case for this is having two domain controllers on a 3-node cluster.
    • Preferred Owner allows you to specify which node in the cluster you’d like the role (VM) to run on. This can have some interesting use cases. In my case, I want to keep DC1 on HV01, and DC2 on HV02 so I chose to use preferred owners. Preferred owner does however have some… unintuitive logic to it. If you think you want to use this, you should read the doc thoroughly.

  • Ubuntu VM optimizations 

    sudo apt update
sudo apt install linux-azure linux-image-azure linux-headers-azure linux-tools-common linux-cloud-tools-common linux-tools-azure linux-cloud-tools-azure
sudo apt full-upgrade
sudo apt install openssh-server
sudo reboot
    sudo apt update
sudo apt install linux-azure linux-image-azure linux-headers-azure linux-tools-common linux-cloud-tools-common linux-tools-azure linux-cloud-tools-azure
sudo apt full-upgrade
sudo apt install openssh-server
sudo reboot

  • Expand a virtual disk & volume

    • Pipe the disk into Resize-VirtualDisk — don’t hand the object to -FriendlyName, which takes a string. Doing that stringifies the object and you get the very unhelpful No MSFT_VirtualDisk objects found with property 'FriendlyName' equal to 'MSFT_VirtualDisk (ObjectId = ...)'.
    • Grow the virtual disk first, then query the partition. Get-PartitionSupportedSize reports SizeMax equal to the current size until there’s actually free space behind it.
    • If the volume is a CSV, run the partition resize on the node that owns it (Get-ClusterSharedVolume) — from any other node the resize fails. 
      $Size = 800GB #EX: 400GB, 3TB, etc
$VirtualDisk = Get-VirtualDisk Images1
$VirtualDisk | Resize-VirtualDisk -Size $Size
$Partition = $VirtualDisk | Get-Disk | Get-Partition | Where PartitionNumber -Eq 2
$Partition | Resize-Partition -Size ($Partition | Get-PartitionSupportedSize).SizeMax
      $Size = 800GB #EX: 400GB, 3TB, etc
$VirtualDisk = Get-VirtualDisk Images1
$VirtualDisk | Resize-VirtualDisk -Size $Size
$Partition = $VirtualDisk | Get-Disk | Get-Partition | Where PartitionNumber -Eq 2
$Partition | Resize-Partition -Size ($Partition | Get-PartitionSupportedSize).SizeMax

Resize-VirtualDisk growing the virtual disk

Resize-Partition extending the partition to its new maximum size

Day 2: Living With It

Building the cluster was most of the fun, but a homelab cluster you can’t see is a homelab cluster that’s quietly on fire. A few follow-ups since this post originally went up:

  • CPU temps without a BMC. The MS-01 has no management controller, so getting a thermal reading takes an in-band ring-0 driver — and that turned into a whole saga (a crash-loop shipping memory to the cloud, because of course it did). Write-up here: Reading CPU Temps Without a BMC.
  • Actually monitoring the cluster. The stock Windows monitoring is blind to node/quorum/CSV state and real S2D latency. Closing that gap — plus the ReFS-redirected-CSV trap and the item timeout Get-ClusterPerf needs — is here: Monitoring S2D with Zabbix.
  • Plan on re-pasting. Once I had temps graphed, one node sat ~12 °C hotter than its identical twin under the same load — textbook thermal-paste pump-out. A re-paste dropped it from ~90 °C to ~78 °C. If you run these little boxes hard, the factory paste does not love sustained load.

The End

Ultimately, these nodes were more trouble than they were worth, and I replaced them with real servers and U.2 drives. It was an interesting project and a good journey, but not one I’d repeat.