Tuning S2D for the Workload

Preface

When I walk into a customer’s S2D cluster to tune it, the question usually isn’t “what did they get wrong?” It’s “what does this workload need, and is the cluster set up to give it that?” A cluster that’s right for 500 VDI desktops isn’t right for a handful of SQL VMs, and the defaults don’t know which one you have.

These are some settings I like to check on S2D clusters (Windows Server or Azure Local) before changing anything. Each one either quietly costs performance or sends you chasing a problem that isn’t there, and every one should be measured before and after you touch it. Get monitoring set up first, so you have a baseline before you start turning knobs.

1. Size the CSV cache by what a miss costs

The CSV in-memory read cache ((Get-Cluster).BlockCacheSize) is RAM each node sets aside to cache reads. The usual advice is either “give it more” or “it’s useless on all-flash.” Both miss the point: the value of a cache hit is the cost of the read it avoided.

I’ve run the same cache on two kinds of all-NVMe clusters with very different results:

On the consumer drives, a miss could land in an SLC-cache or garbage-collection stall, so every hit saved real latency and the cache earned a bigger allocation. On enterprise drives with power-loss protection, a miss costs tens of microseconds, so a bigger cache just takes RAM away from VMs.

The workload matters as much as the drives. The cache only pays off when the same blocks are read again. The VMs that benefit are read-heavy ones whose hot data fits in the cache and isn’t already cached inside the guest. Database VMs with their own buffer pools keep their hot reads in guest RAM and get little from it.

Check the miss cost, not the hit rate. A low hit rate on its own isn’t a reason to grow the cache: it often just means the cluster is idle, or the working set is far bigger than any cache you could give it. Look at:

• \Cluster CSV File System(_total)\Read Latency: if reads are already at flash speed, there’s nothing left to save.
• Miss IOPS (ClusterNode.CsvCache.Iops.Read.Miss from Get-ClusterPerf): if misses are single digits per second, the cache isn’t what’s holding you back.

The cache is read-only, so it does nothing for write latency. If writes are slow, look at the drives (see Monitoring S2D with Zabbix for narrowing down latency by layer).

2. Minroot: give the host what it needs, and no more

Minroot (hypervisorrootproc) limits the host OS (the root partition) to a set number of virtual processors. It doesn’t reserve anything for the host: guest VMs still run on every logical processor, including the ones the host’s VPs run on. It only caps how much of the box the host can use at once.

Why it helps: the host runs the storage stack, the cluster service and every management agent. Without minroot the host gets a VP for every logical processor, so its work can land on any core, and with the core scheduler a core running a host VP can’t run a guest VP at that moment. Guests wait. With minroot the host’s work runs on fewer cores, so it gets in the guests’ way less often.

What it did on my clusters:

• On a cluster running without minroot, guest VP wait per dispatch was roughly 3× higher than with it. Setting it brought wait straight back down.
• On my cluster with lightly loaded hosts, going from minroot 40 to 24 cut root VP wait by roughly 25–45 % and S2D write p95 by about 7 %. Going from 24 to 20 trimmed write p95 another 6–8 % and p99 about 13 %.
• My cluster with busier hosts (much higher host CPU) stayed at 24. Its host partition has more work to do, and 20 vs 24 never gave a clean answer.

The right number depends on how much the host itself has to do, so test it instead of copying mine.

Setting it

# set the number of root (host) LPs, then drain and reboot the node
bcdedit /set hypervisorrootproc 24

# after the reboot: confirm what the host actually got, and what the box has
(Get-Counter '\Hyper-V Hypervisor Root Partition(*)\Virtual Processors').CounterSamples.CookedValue
(Get-Counter '\Hyper-V Hypervisor\Logical Processors').CounterSamples.CookedValue

Roll it one node at a time like any other reboot.

Under minroot, Task Manager, Win32_Processor and \Processor(_Total) only see the host’s VPs. A node that looks 60 % busy in Task Manager can be 10 % busy as a whole box. Measure whole-box CPU from the hypervisor counters (see Monitoring S2D with Zabbix).

Finding the optimal number

Watch these before and after each change:

How I test:

• Change one thing. Step down (e.g. 40 → 24 → 20) and stop when host CPU or write latency starts to climb.
• Compare the same clock hours, before and after, and skip the first hour after each reboot.
• Let each change soak for at least 24 hours before deciding it’s better. Workloads shift from hour to hour and day to day, so one good afternoon doesn’t prove anything. Compare full days, and the longer you can leave it, the more you can trust the result.
• Ignore VP wait at idle. See the next section.

3. CPU power: wake times and BIOS settings

Core wake times skew VP wait

When a VP is dispatched onto a core sleeping in a deep C-state, it pays the time that core takes to wake up, and that lands in wait per dispatch. At low load, wake time is most of what you’re measuring, not contention. Idle values differ by roughly 10× between platforms for this reason alone: around 10,000–13,000 ns on servers with deep C-states enabled, versus 1,000–2,000 ns on hardware that wakes faster.

• Compare VP wait under load, before and after a minroot change, not at idle.
• Don’t change two things at once. Changing BIOS C-states from Disabled to Autonomous roughly doubled guest VP wait on one of my clusters with no other change. Mix that into a minroot test and you’ll credit or blame the wrong setting.
• If wake latency matters more than power for the workload, look at the BIOS, not just the Windows power plan.

BIOS settings that help or hurt

Names vary by vendor, and every one of these trades power for latency, so decide which matters more for the workload first. The right answer also changes by CPU generation: settings that helped on one platform did nothing, or hurt, on another. Treat these as levers to test, not a profile to copy.

2nd-gen Xeon Scalable (my lab)

Minimum processor state in Windows belongs with these. Lowering it saved power and cost nothing measurable on a quiet cluster, but cost write latency on a busier one, because busy hosts wake and ramp cores far more often. Don’t generalise a “no cost” result from a quiet cluster to a busy one.

I ended up keeping the power savings, because latency was already well inside my targets.

Xeon 6

Xeon 6 is far more sensitive. Two changes come before any BIOS work:

• Disable C-states, in the BIOS or from Windows:

powercfg /setacvalueindex SCHEME_CURRENT SUB_PROCESSOR IDLEDISABLE 1
powercfg /setactive SCHEME_CURRENT

• Keep each storage NIC’s work on its own socket. Socket-to-socket latency is high, so RSS processing on the far socket hurts every I/O (not an issue on single-socket servers):

Get-NetAdapterRss -Name <storage NICs> | Format-List Name, NumaNode, BaseProcessorNumber, MaxProcessorNumber
# pin RSS to processors on the NIC's own node
Set-NetAdapterRss -Name <NIC> -NumaNode <node> -BaseProcessorNumber <first LP on that node> -MaxProcessorNumber <last LP>

Those two alone cut average write latency under load to roughly a third.

Then the BIOS, in two levels:

With both levels applied, latency on a Xeon 6 cluster improved anywhere from about 2× to well over 10× depending on the percentile, with the biggest gains in the tail.

Notice that ASPM and power management go opposite ways between the two generations. That’s the point of this whole section: measure on the hardware and workload in front of you.

4. Live migration: the right network, transport and concurrency

Live migration is the one workload that can saturate the storage network on purpose. Get it wrong and a patch-night drain either crawls or hurts storage latency for every VM on the cluster.

Which network it uses

Live migration inside a cluster should run over the RDMA storage networks, never management.* Check which cluster networks it can use:

Get-ClusterNetwork | Format-Table Name, Role, Address, Metric
Get-ClusterResourceType 'Virtual Machine' | Get-ClusterParameter MigrationExcludeNetworks, MigrationNetworkOrder
Get-VMHost | Select-Object UseAnyNetworkForMigration

• Exclude management. Otherwise a single storage link failure quietly moves migrations onto the slower management network.
• Watch for networks the cluster adds on its own. Failover Clustering auto-enrolls any new subnet (iSCSI vNICs, a backup network) with Role = Cluster, which also makes it a migration candidate. Set those to Role = None.
• UseAnyNetworkForMigration should be False on cluster nodes.

* The exception is cluster-to-cluster (shared-nothing) live migration. Storage networks are usually non-routed, and in a switchless mesh they physically can’t reach another cluster, so a move between clusters has to go over a routed network, normally management or compute. Tools like Windows Admin Center (vMode) enable UseAnyNetworkForMigration on the source host to make that work. Plan for it: expect it to be slower than in-cluster moves, check the bandwidth on that path, and set UseAnyNetworkForMigration back to False afterwards if you don’t move VMs between clusters regularly.

Switched vs switchless

• Switched: every storage NIC reaches every node, so SMB Multichannel spreads one migration across all of them. Migration shares those same links with S2D traffic from every node, which is why the SMB bandwidth limit (below) matters most here.
• Switchless (direct-connect mesh): a migration between two nodes can only use the links between those two nodes. In a 3-node mesh with two links per pair, any single migration gets two of the four storage ports. Parallel migrations to different nodes use different links, so concurrency is effectively free up to the number of storage paths.

Transport (performance option)

Get-VMHost | Select-Object VirtualMachineMigrationPerformanceOption. Network ATC picks this for you; on a hand-built cluster, check it.

Concurrency and bandwidth

# two separate caps -- the cluster one decides drains and CAU
Get-VMHost | Select-Object MaximumVirtualMachineMigrations, MaximumStorageMigrations
(Get-Cluster).MaximumParallelMigrations

# SMB bandwidth reserved for live migration, so it can't starve storage
Get-SmbBandwidthLimit

• Concurrency: I set one migration per storage path per node: 2 on a 2-node switchless cluster, 4 on a 3-node mesh. On a switched fabric, start lower and raise it while watching storage latency during a drain. Never leave it at 1. One migration that gets stuck, or just crawls (a large VM, or a busy one dirtying memory faster than it can be copied), then holds up every VM queued behind it in the drain.
• Set both caps. Set-VMHost -MaximumVirtualMachineMigrations is per node; (Get-Cluster).MaximumParallelMigrations is what the cluster actually orchestrates during a drain, failover or Cluster-Aware Updating. Setting only the host value looks right in Get-VMHost and still moves one VM at a time.
• With Network ATC, set it through ATC. ATC defaults to one concurrent migration and calculates its own SMB bandwidth limit. Change them with a global override, or ATC’s drift remediation can put its values back:

$o = New-NetIntentGlobalClusterOverrides
$o.MaximumVirtualMachineMigrations = 2
# $o.MaximumSMBMigrationBandwidthInGbps = 10    # only if you need to change ATC's calculated limit
Set-NetIntent -GlobalClusterOverrides $o
Get-NetIntentStatus -GlobalOverrides

Raise the drain timeout for large-memory hosts. When Cluster-Aware Updating (which also drives Azure Local updates) drains a node, it only waits ShutdownTimeoutInMinutes for the VMs to move. When that runs out, the remaining live migrations are switched to quick migrations while the node is already restarting, so those VMs go down, and in the worst case their data is damaged. The update gets blamed, but it’s the timeout. On Hyper-V hosts Windows sizes this from physical memory, about 100 minutes per 64 GB with a floor of 20 (my 256 GB nodes read 399), so on most clusters it’s already sensible. This is always worth double-checking.

(Get-Cluster).ShutdownTimeoutInMinutes          # check the current value
(Get-Cluster).ShutdownTimeoutInMinutes = 1440   # 24 hours

A long timeout costs nothing when drains are fast; it only matters when a drain is slow, which is exactly when you want it to keep live migrating.

Gotcha: drains fail with event 21111 and “may be busy or in a state where it cannot be moved”. That message is generic. The real reason is only in the Microsoft-Windows-FailoverClustering/Diagnostic log, and on a busy drain it’s often “migration limit reached” at the destination. That’s normal backpressure: the cluster retries until they all move. Don’t raise the limit just to quiet the event. If the limit in that message is lower than what you configured, re-apply the setting by writing a different value first (set 1, then 2); setting the same value it already reports is a no-op.

5. “RDMA enabled” isn’t “RDMA working”

Get-NetAdapterRdma showing Enabled = True only means it’s switched on. RoCE fails quietly: SMB falls back to TCP, or a bad PFC config drops packets and latency climbs. Check it’s actually clean:

# SMB Direct actually in use: query S2D's SBL instance (the default instance returns nothing
# on a cluster node). An RDMA-capable pair with channels up but RdmaConnectionCount 0 is TCP fallback.
Get-SmbMultichannelConnection -SmbInstance SBL |
    Format-Table ServerName, ClientRdmaCapable, ServerRdmaCapable, CurrentChannels, RdmaConnectionCount, TcpConnectionCount

# PFC on the SMB Direct priority only (usually 3), plus an ETS bandwidth reservation
Get-NetQosFlowControl
Get-NetQosTrafficClass

# health counters should stay at 0
Get-Counter '\RDMA Activity(*)\RDMA Completion Queue Errors',
            '\Network Interface(*)\Packets Received Discarded'

• PFC enabled on priority 3 only, not on every priority.
• An ETS traffic class for SMB Direct (e.g. about 50 % of bandwidth).
• Zero RDMA completion-queue errors and zero discards on the storage adapters.

Gotcha: only count discards on the adapters that carry SMB Direct. Compute and management uplinks drop broadcast and flooded traffic by design, and you’ll waste an afternoon chasing it.

6. Defender: trust the automatic exclusions, then check the gaps

A lot of guides tell you to add C:\ClusterStorage and the VM file types to Defender’s exclusions. On Windows Server 2016 and later and on Azure Local, Defender already does this automatically: the Hyper-V role excludes VHD/VHDX, AVHDX, VMCX, VMRS, VSV, ISO and RCT files plus vmms.exe and vmwp.exe, and the File and Storage Services role excludes C:\ClusterStorage (Microsoft’s list).

Gotcha: you can’t see them. Automatic exclusions don’t appear in Get-MpPreference, and MpCmdRun -CheckExclusion reports those paths as “not excluded”. On Azure Local even custom exclusions are hidden from local admins. Don’t add exclusions just because the list looks empty.

What to actually check:

$p = Get-MpPreference
$p.DisableAutoExclusions          # must be False -- some hardening baselines turn it on
$p.ScanParameters                 # 1 = quick scan (default), 2 = full scan
$p.ScanScheduleDay                # 0 = every day, 8 = never
(Get-MpComputerStatus).FullScanEndTime

• Automatic exclusions only apply to real-time protection. A scheduled full scan ignores them and will walk every VHDX on every CSV. Keep the scheduled scan as a quick scan.
• If automatic exclusions are disabled, or you run third-party AV, add the exclusions yourself from Microsoft’s Hyper-V host list. Use the file-type exclusions: on my cluster a custom C:\ClusterStorage path exclusion didn’t match files on the CSVs, which Defender sees as \Device\CSVVolumeN\…, but the .vhdx extension exclusion did.

Takeaway

Tune for the workload in front of you, and measure before you change anything. Every one of these looked like either a problem or a free win until the numbers said otherwise.