Monitoring S2D with Zabbix

Preface

The stock “Windows by Zabbix agent” template is great at OS-level stuff (services, disks, CPU, memory, NICs) and blind to the part that matters on a hyperconverged cluster: the cluster and the storage. It will tell you a filesystem is filling up. It won’t tell you a node got evicted, the witness is gone, a CSV went block-redirected, or volume latency just tripled.

This is how I close that gap on S2D clusters running both Windows Server 2025 and Azure Local, plus the gotchas that cost me time. It’s the same on any hardware.

The plan

What the stock template doesn’t cover:

1. Cluster control plane — node membership, quorum/witness, failed resources, CSV online + redirected state.
2. Storage performance — volume latency percentiles, IOPS, CSV cache hit rate, and latency at each layer of the stack for troubleshooting.
3. Pool capacity — free space above the drive-repair reserve.
4. Host CPU and scheduling latency — the whole box (not just the host OS), and how long VMs wait for a core.
5. The VMs — per-VM state, health, CPU, and memory pressure.
6. Things that degrade quietly — drive wear and errors, RDMA health, Network ATC intent status.

The thresholds throughout are the ones I run and recommend as a starting point; tune them to your hardware and workload. Everything runs through the Zabbix Agent 2 already on the nodes: UserParameters (PowerShell) and the agent’s built-in perf counters. No extra agents.

Layer 1: the cluster control plane

The cluster cmdlets return cluster-wide data from any node, so you only want to collect it once per cluster, not once per node. Don’t pin it to a “control node”, though: if that node is down, you lose cluster monitoring exactly when you need it.

Create a Zabbix host for the cluster itself, with the cluster IP as its agent interface. The cluster IP is a cluster resource that follows whichever node owns the core group, and the agent on every node listens on all local addresses, so a poll to the cluster IP is answered by the current owner. Collection survives any single node going down, with nothing to re-point. Per-node signals (event logs, each node’s VMs, NIC counters) stay on the node hosts.

Gotcha: this only works with passive checks. An active check is keyed off the agent’s configured Hostname=, which is the node name, so data for the cluster host silently never arrives. Make every item on the cluster host a passive Zabbix agent item (or a dependent item off one).

The collector itself is a small PowerShell script behind a UserParameter (cluster[*]):

"nodesdown"     { @(Get-ClusterNode      | ?{$_.State -ne 'Up'}).Count }
"quorumbad"     { @(Get-ClusterResource  | ?{$_.ResourceType -match 'Witness' -and $_.State -ne 'Online'}).Count }
"resfailed"     { @(Get-ClusterResource  | ?{$_.State -eq 'Failed'}).Count }
"csvoffline"    { @(Get-ClusterSharedVolume | ?{$_.State -ne 'Online'}).Count }
"csvblockredir" { @(Get-ClusterSharedVolume | Get-ClusterSharedVolumeState |
                    ?{"$($_.BlockRedirectedIOReason)" -ne 'NotBlockRedirected'} |
                    Select -Expand Name -Unique).Count }

Alert on nodesdown > 0, quorumbad > 0, csvoffline > 0, and so on.

Gotcha: ReFS CSVs are always “FileSystemRedirected”, and that’s normal

Symptom: every ReFS CSV reports this, all the time:

1StateInfo                    : FileSystemRedirected
2FileSystemRedirectedIOReason : FileSystemReFs
3BlockRedirectedIOReason      : NotBlockRedirected

Cause: ReFS CSVs run filesystem-redirected by design; metadata operations go to the coordinator node. It isn’t a fault.

Fix: never alert on the filesystem-redirect flag. Alert on block redirection (BlockRedirectedIOReason ≠ NotBlockRedirected), which means a node lost its direct path to storage and is sending all its I/O over the network. That’s what csvblockredir above keys on.

Layer 2: storage performance

Live latency and IOPS: native counters

For per-minute graphs, use the Cluster CSV File System perf counters. The agent reads them directly (no script), and the _total instance saves you enumerating volumes:

1perf_counter_en["\Cluster CSV File System(_total)\Read Latency",60]
2perf_counter_en["\Cluster CSV File System(_total)\Write Latency",60]
3perf_counter_en["\Cluster CSV File System(_total)\Reads/sec",60]
4perf_counter_en["\Cluster CSV File System(_total)\Writes/sec",60]

The latency counters are in seconds; multiply by 1000 in preprocessing for milliseconds.

Percentiles and CSV cache hit rate: Get-ClusterPerf

An average hides the spikes VMs actually feel, so alert on write p95/p99. S2D’s own performance history (Get-ClusterPerf) has the volume latency series and the CSV in-memory read cache hits and misses. One script on the cluster host turns the last hour into percentiles and a hit rate:

$series = 'Volume.Latency.Read','Volume.Latency.Write','Volume.IOPS.Total',
          'ClusterNode.CsvCache.Iops.Read.Hit','ClusterNode.CsvCache.Iops.Read.Miss'
$h  = Get-Cluster | Get-ClusterPerf -ClusterSeriesName $series -TimeFrame LastHour
$by = $h | Group-Object { ("$($_.MetricId)" -split ',')[0] } -AsHashTable -AsString

function Pctl($vals, $p) {
  $v = @($vals | Sort-Object); if (-not $v) { return -1 }
  $v[[math]::Min($v.Count - 1, [math]::Floor($v.Count * $p / 100))]
}
$wr   = $by['Volume.Latency.Write'].Value                  # seconds
$hit  = ($by['ClusterNode.CsvCache.Iops.Read.Hit'].Value  | Measure-Object -Sum).Sum
$miss = ($by['ClusterNode.CsvCache.Iops.Read.Miss'].Value | Measure-Object -Sum).Sum

@{
  wr_p95_ms    = [math]::Round((Pctl $wr 95) * 1000, 3)
  wr_p99_ms    = [math]::Round((Pctl $wr 99) * 1000, 3)
  cache_mb     = (Get-Cluster).BlockCacheSize
  cache_hitpct = if ($hit + $miss) { [math]::Round($hit / ($hit + $miss) * 100, 1) } else { -1 }
} | ConvertTo-Json -Compress

• Compute the hit rate from summed hits and misses, not by averaging a per-sample hit-rate series. Quiet samples with a handful of I/Os read 0 % or 100 % and drag an average around.
• The CSV cache is worth watching. On an enterprise all-flash cluster I typically see it between 60 and 95 % over a day. If it drops, check whether BlockCacheSize changed or the working set outgrew it.

Gotcha: set the item timeout explicitly. Get-ClusterPerf takes a few seconds, and the first time I tried it every item came back “Timeout occurred while gathering data.” It runs fine as LocalSystem; the problem is that in Zabbix 7 the effective timeout for an agent item is the server-side item timeout (3 s by default), not the agent’s own Timeout= setting. Set timeout=30s on every master item that runs PowerShell. This applies to every slow collector in this post, not just this one.

Gotcha: the “IO” counters read zero on ReFS

Symptom: your IOPS graph is a flat line at zero.

Cause: the IO Read… / IO Write… “direct” counters read zero on a ReFS CSV, because ReFS runs redirected.

Fix: use the top-level Reads/sec / Writes/sec / Read Latency / Write Latency counters above, which roll up direct and redirected I/O.

Also skip the local \PhysicalDisk\Avg. Disk sec/* counters for S2D health. The real I/O path is the storage bus layer / virtual disk, so local physical-disk latency isn’t what your VMs see.

Narrowing it down: latency at each layer of the stack

A volume latency number tells you that storage is slow, not where. S2D latency is a stack, and every layer has its own counter (all in seconds):

How Cluster Disk Counters works: every node reports every drive in the pool. Instances are named <owning node's cluster Id><3 digits>, so on node Id 2 its own drives are 2001, 2002 and so on ((Get-ClusterNode -Name <node>).Id). The counters measure the drive I/O issued by the node you run them on: Local: latency covers its own drives and Remote: latency covers other nodes’ drives, which cleanly splits drive latency from network latency. One catch: with ReFS, CSVs are filesystem-redirected, so only the CSV owner nodes issue drive I/O. A node that owns no CSV reads 0 on every drive, even with VMs running on it. That’s normal, so run this on the CSV owners (Get-ClusterSharedVolume shows them). (\SMB Client Shares(*blockshare$) also has per-peer latency, but its instances are named by the peer’s link-local IPv6 address, which changes when the peer restarts.)

Mirrored writes go to several drives in parallel, so the layers don’t add up neatly, but comparing them narrows things down fast:

• VM high, CSV normal: the problem is inside the VM or its virtual disk (a long checkpoint chain, an in-guest queue), not the cluster.
• CSV high, both drive layers normal: check the Redirected … counters and block redirection (Layer 1). I/O is taking the long way round.
• Other nodes’ drives high, while each node’s own drives are normal: the network. Check for RDMA fallback and errors (Layer 6), PFC/ETS config, or a congested link.
• One drive high, its peers normal: that drive. Cross-check its reliability counters (Layer 6).
• All drives high: the pool is saturated, or the drives can’t keep up with sustained writes (consumer drives without power-loss protection fall off a cliff once their SLC cache fills).

To check all of them at once during an incident, run this on each CSV owner node (a 10-second sample, in ms):

# Snapshot write latency at each layer of the S2D stack on this node (ms, ~10 s average)
$id = (Get-ClusterNode -Name $env:COMPUTERNAME).Id      # this node's drives are <Id><3 digits>
$s  = (Get-Counter -SampleInterval 2 -MaxSamples 5 -Counter @(
        '\Hyper-V Virtual Storage Device(*)\Latency'
        '\Cluster CSV File System(*)\Write Latency'
        '\Cluster Disk Counters(*)\Remote: Write Latency'
        '\Cluster Disk Counters(*)\Local: Write Latency')).CounterSamples
function Row($label, $samples) {
  $v = $samples | Group-Object InstanceName |
       ForEach-Object { ($_.Group | Measure-Object CookedValue -Average).Average * 1000 }
  if (-not $v) { return '{0,-36} (no instances)' -f $label }
  $m = $v | Measure-Object -Average -Maximum
  '{0,-36} avg {1,7:N3} ms   worst {2,7:N3} ms' -f $label, $m.Average, $m.Maximum
}
Row '1 VM virtual disks (VHDX)'        ($s | ? { $_.Path -like '*virtual storage device(*' })
Row '2 CSV file system'                ($s | ? { $_.Path -like '*csv file system(*' -and $_.InstanceName -ne '_total' })
Row '3 Other nodes'' drives (network)'  ($s | ? { $_.Path -like '*remote: write latency' -and $_.InstanceName -notmatch "^($id\d{3}|_total)$" })
Row '4 This node''s drives'            ($s | ? { $_.Path -like '*local: write latency'  -and $_.InstanceName -match "^$id\d{3}$" })

Look at the worst column; idle virtual disks pull the averages down. A healthy all-flash cluster looks something like this, with the network hop dominating and each node’s own drives well under 0.1 ms:

1 VM virtual disks (VHDX)            avg   0.214 ms   worst   0.796 ms
2 CSV file system                    avg   0.403 ms   worst   0.624 ms
3 Other nodes' drives (network)      avg   0.348 ms   worst   0.426 ms
4 This node's drives                 avg   0.062 ms   worst   0.068 ms

For ongoing monitoring, put the _Total Local and Remote read/write latencies on every node as plain items, and discover every drive on every node with perf_instance_en.discovery["Cluster Disk Counters"] (excluding _Total). Don’t filter discovery to each node’s own drives: a node that owns no CSV issues no I/O, so its drives would go unwatched. With every drive discovered everywhere, each drive is covered from whichever nodes own CSVs, wherever the CSVs move. A per-drive trigger for “well above this node’s overall average, and above a floor” catches a single failing drive without firing on idle noise or the normal local/remote gap.

Layer 3: pool capacity

“How full is the pool” is the wrong question. The useful one is: how much can I still allocate before I eat the drive-repair reserve? That reserve is free space S2D needs to repair a failed drive in place, and Microsoft’s rule is one capacity drive per server, up to 4:

$pool  = Get-StoragePool -IsPrimordial $false
$cap   = $pool | Get-PhysicalDisk | Where-Object Usage -ne 'Journal'   # capacity drives only
$nodes = @(Get-StorageFaultDomain -Type StorageScaleUnit).Count
$total = ($cap | Measure-Object Size -Sum).Sum
$alloc = ($cap | Measure-Object AllocatedSize -Sum).Sum
$reserve = [math]::Min($nodes, 4) * ($cap | Measure-Object Size -Maximum).Maximum

@{ capacity_tb      = [math]::Round($total / 1TB, 2)
   free_above_res_tb = [math]::Round(($total - $alloc - $reserve) / 1TB, 2) } | ConvertTo-Json -Compress

• Alert when free_above_res_tb drops below 0 (the reserve is being consumed; Average) and below 10 % of capacity (thin; Warning).
• Exclude cache/journal devices. They hold no data, and counting them makes the pool look bigger than it is.

Gotcha: don’t alert on “can the cluster absorb a node loss”. It sounds stricter, but S2D doesn’t rebuild a node that goes down: it waits for the node to come back. Reserving space for a full-node rebuild describes a repair that never happens. On a 2-node or 3-node 3-way-mirror cluster that check is structurally false forever, so it pages on a correctly built cluster while telling you nothing about the drive-repair case that actually matters.

Layer 4: host CPU and scheduling latency

The host’s real CPU isn’t system.cpu.util

On a Hyper-V host, system.cpu.util and \System\Processor Queue Length only measure the root partition (the host OS). With minroot the host partition is capped at a handful of virtual processors, so the host OS can look busy while the box is mostly idle, or calm while every core is busy running VMs.

For the whole box, use the hypervisor’s counters:

1perf_counter_en["\Hyper-V Hypervisor Logical Processor(_Total)\% Total Run Time",60]
2perf_counter_en["\Hyper-V Hypervisor\Virtual Processors",60]
3perf_counter_en["\Hyper-V Hypervisor\Logical Processors",60]

Gotcha: _Total divides by the wrong thing. On the Hyper-V Hypervisor Logical Processor counter set, _Total is the sum across logical processors divided by the number of virtual processors, not logical processors. On my nodes it over-read whole-box CPU by 1.3× to 2×, and the error moves every time a VM starts, stops or migrates (it changes the VP count). Fix it with a calculated item:

1last(//perf_counter_en["\Hyper-V Hypervisor Logical Processor(_Total)\% Total Run Time",60])
2  * last(//perf_counter_en["\Hyper-V Hypervisor\Virtual Processors",60])
3  / last(//perf_counter_en["\Hyper-V Hypervisor\Logical Processors",60])

That’s true whole-box CPU %. Multiply by the logical processor count and divide by 100 for “cores busy”, which compares cleanly across clusters with different core counts. % Guest Run Time from the same counter set has the same bias.

Scheduling latency: CPU Wait Time Per Dispatch

Utilization doesn’t tell you whether VMs are waiting for a core. CPU Wait Time Per Dispatch does: it’s how long a virtual processor sits in the hypervisor’s run queue before it gets a physical core (Hyper-V’s version of VMware’s “CPU Ready”), in nanoseconds:

1perf_counter_en["\Hyper-V Hypervisor Virtual Processor(_Total)\CPU Wait Time Per Dispatch",60]
2perf_counter_en["\Hyper-V Hypervisor Root Virtual Processor(_Total)\CPU Wait Time Per Dispatch",60]

The first is guest VMs; the second is the host OS. Track the VP:LP ratio (virtual ÷ logical processors) alongside it as a capacity signal: warn above 2:1, more urgently above 3:1.

Gotcha: at idle, wait per dispatch measures power management, not contention. When a VP is dispatched onto a core sleeping in a deep C-state, it pays the wake-up time. Idle values differ by roughly 10× between platforms for that reason alone: around 10,000–13,000 ns on enterprise servers with deep C-states enabled, versus 1,000–2,000 ns on a whitebox. So:

• Gate the alert on load: avg(guest wait,30m) > 15000 and avg(whole-box CPU,30m) > 50. High wait on an idle host is noise.
• Set thresholds per platform, not fleet-wide.
• If idle wait is high on a host you care about, look at the BIOS power profile (deep C-states) before anything else.

Layer 5: the VMs

Use LLD (low-level discovery) so VMs are picked up automatically as they’re created or live-migrate around the cluster.

To avoid spawning PowerShell per VM per item, use one master item that returns a JSON blob for every VM on the node, and dependent item prototypes that pull each value out:

# master item: hyperv.vm.data  -> one Get-VM call, JSON array of every VM
foreach ($v in Get-VM) {
  [pscustomobject]@{
    name          = $v.Name
    id            = $v.VMId.Guid
    state         = $v.State        # Running / Off / Paused / Saved / Critical
    cpu           = $v.CPUUsage
    memassignedmb = [math]::Round($v.MemoryAssigned/1MB)
    memdemandmb   = [math]::Round($v.MemoryDemand/1MB)
    # demand / assigned -> memory pressure %
    mempressure   = [math]::Round($v.MemoryDemand / $v.MemoryAssigned * 100)
  }
}

Each dependent prototype extracts its field with a JSONPath keyed on the discovered VM ID, e.g. $[?(@.id=='{#VMID}')].mempressure.first(). One PowerShell run per cycle feeds CPU, memory, state, heartbeat and checkpoint-age items for every VM.

Memory pressure (demand ÷ assigned) is the most useful of these. The first thing it caught was a Kubernetes control-plane VM wanting more than twice the memory it was assigned (200 %+ pressure). That one turned out to be an AKS-on-Azure-Local node the platform redeploys monthly, so hand-tuning it would just get wiped, but it explained why that node felt heavy.

Layer 6: things that degrade quietly

Layers 1–5 catch things that broke or are running hot. The scarier failures creep: a drive wearing out, RoCE dropping packets and limping along on TCP, a Network ATC intent that half-applied. This is another collector using the same UserParameter + master-JSON pattern as the VM layer.

Drive reliability, not just “healthy”

Get-PhysicalDisk’s HealthStatus flips after things are already bad. Get-StorageReliabilityCounter gives the leading indicators: wear %, uncorrected read/write errors, and max latency, per drive:

Get-StoragePool -IsPrimordial $false | Get-PhysicalDisk | ForEach-Object {
  $r = $_ | Get-StorageReliabilityCounter
  [pscustomobject]@{ wear=$r.Wear; rderr=$r.ReadErrorsUncorrected
                     wrerr=$r.WriteErrorsUncorrected; rdlat=$r.ReadLatencyMax }
}

• The scan is slow (several seconds across a dozen NVMe, plus PowerShell’s cold start). Set the item timeout to 30 s (see the timeout gotcha in Layer 2) and poll it every 30 minutes; wear doesn’t change by the minute.
• Baseline each platform before alerting on latency. These are max-latency counters and the floor is platform-specific: enterprise nodes peaked in the 10–20 ms range, while a whitebox cluster on consumer NVMe spiked past 100 ms under load. Alert against each platform’s own normal, or one cluster will cry wolf on the other’s numbers.

Is RDMA actually working?

When RDMA breaks it doesn’t announce itself: it falls back to TCP, or bad PFC config drops packets and latency craters. Watch the two counters that should always read zero: RDMA completion-queue errors and discards on the storage NICs.

• Find the storage NICs programmatically: RDMA-enabled physical adapters that are not members of a vSwitch/SET team. The team uplinks carry compute traffic and drop broadcast/flood frames by design, so their discards don’t matter.
• Check SMB Multichannel for TCP fallback. Get-SmbMultichannelConnection -SmbInstance SBL shows each storage connection’s RDMA and TCP channel counts; any TCP channels on an RDMA-capable pair means fallback. Two gotchas:
  • Without -SmbInstance, it returns nothing on a cluster node, so a fallback check that leaves it out reports clean forever.
  • Read the SBL instance only. It’s a full mesh (connections = peers × storage subnets), so every node should read the same. The CSV instance only has connections toward nodes that currently own a CSV, so its counts move on every CSV failover.
• Alert on change, not the absolute. These counters are cumulative since boot (one of my nodes carried well over a hundred discards from a link negotiation months earlier). A > 0 rule alarms forever; change() > 0 fires only on new drops and clears when they stop.

Network ATC: trust, but verify

Network ATC can report an intent as Failed or stuck Pending while the adapter config looks perfect. If you don’t watch Get-NetIntentStatus, you find out when a node reboots and its networking doesn’t come back the way you left it. One item counts unhealthy intents, and a text item carries the detail:

# Count ONLY terminal trouble. In-flight states during ATC's drift-check are healthy.
$bad = Get-NetIntentStatus | ?{
  $_.ConfigurationStatus -match 'Fail' -or $_.ProvisioningStatus -match 'Fail' -or
  [int]$_.RetryCount -gt 0 -or "$($_.Error)".Trim()
}
@{ failed = @($bad).Count
   detail = ($bad | %{ "$($_.IntentName)@$($_.Host):$($_.ConfigurationStatus) retry=$($_.RetryCount)" }) -join '; ' }

Gotcha: don’t alert on “not Success”. ATC re-provisions every intent on a ~15-minute drift-check cycle, and mid-cycle an intent briefly shows Provisioning or Validating. A 5-minute poll will occasionally land in that window and fire an alert that’s cleared by the time you look.

Fix: count only terminal trouble (a *Fail* status, a non-empty Error, or RetryCount > 0, as in the snippet above), and debounce the trigger: min(/HOST/s2d.atc.failed,#3)>0 requires three consecutive bad polls (~15 min). The debounce also absorbs the brief errors ATC logs at boot before the cluster service is fully up. For fixing a stuck intent, see Network ATC 101.

Keep these deep-health signals at dashboard severity, not pager severity. A worn drive or a creeping discard count is a “look at it this week” item. The only things in this layer that should page are an unhealthy disk or uncorrected errors.

Collector gotchas

A few things that make a collector silently wrong rather than visibly broken:

• One missing counter set fails the whole Get-Counter call. If you request paths from several counter sets in one call and any set has no instances, PDH fails all of them. On Azure Local with SDN, \Hyper-V Virtual Network Adapter(*) doesn’t exist, so a combined call returns nothing at all. Call Get-Counter once per counter set (not once per path; that’s much slower) and report which sets were missing.
• An absent field is not a zero. If a value is legitimately absent sometimes (the VM is off, the counter set doesn’t exist on this platform), set the dependent item’s JSONPath error handler to discard. “Set value to 0” graphs fake silence, and “unsupported” buries real problems in noise. Changing a prototype doesn’t fix items already discovered; delete them and let discovery recreate them.
• Don’t Move-Item a script into place from C:\Windows\Temp. A move keeps the source’s permissions, and the Temp ACL drops BUILTIN\Users. The node’s own agent (LocalSystem) still works, so nothing looks broken, but a cluster-scoped collector that reads the script on peer nodes gets “Access is denied”. Copy over the existing file instead, or icacls <file> /reset.

Alerting

Collecting all this is half the job. The other half is making alerts trustworthy enough that nobody mutes them: gate by severity, suppress known-benign noise, and add a short delay so transient blips clear themselves before anyone gets paged.

Treat every threshold in this post as a starting point. Latency, scheduling wait and capacity margins all vary with hardware, power settings and workload, so watch a week or two of your own data and adjust before you let anything page.

Where it lands

One dashboard that answers “is the cluster healthy?”: node and quorum state, write p95 and CSV cache hit rate, free space above the repair reserve, whole-box CPU and VM scheduling latency, per-VM CPU and memory pressure, and hardware temps from the BMC (or in-band if you don’t have one: see Reading CPU Temps Without a BMC).

A dashboard is only as honest as its items. My per-CSV “usage %” table silently left out an entire volume, because it matched CSVs by name and that one was named differently on the other cluster. A missing row looks exactly like a healthy one. Every so often, reconcile dashboards against ground truth (Get-ClusterSharedVolume, Get-VirtualDisk).

If you take a few things from this: don’t alert on ReFS filesystem redirection, set item timeouts explicitly for anything that runs PowerShell, measure the whole box, not the host OS (and correct _Total), gate scheduling-latency alerts on load, and discover your VMs instead of hand-listing them.