Network ATC 101

What Network ATC actually does, how to build intents for converged and switchless clusters, and how to un-stick it when it fails

Table of Contents

Preface

Network ATC is awesome, but you really need to have some understanding before you jump in, so let’s see if I can help out a little bit. Let’s start off with everyone’s first question “What does Network ATC stand for?” Well officially Network ATC is not an acronym for anything, however some of the imagery published includes airplanes and one of its primary functions is to direct traffic. Read into that as you will.

Network ATC is going to help deploy consistent networking across all hosts in a cluster, and prevent configuration drift. Awesome! But, it also needs every host to be configured the same. Think of this as infrastructure as code for host networking.

The basics

Pre-Reqs

  • Windows Server 2025 Datacenter, or Azure Local 2311.2 or newer.
  • All NICs in an intent must be the same Make, Model, Speed, etc.
  • All NICs in an intent must be named identically across all nodes in the cluster.
  • The following roles must be installed: Network ATC, Hyper-V, Failover Clustering, Data Center Bridging, FS-SMBBW

Intent types

There are really three intent types commonly used. An intent must have at least one type, but can have multiple.

  • Compute: The compute intent type creates a Hyper-V switch for VM traffic.
  • Management: Also creates a Hyper-V switch, but creates a virtual management adapter for the Host OS.
  • Storage: This is going to be for SMB/RDMA traffic. Primarily for S2D storage and Live Migration between nodes.

Now, I said you can mix and match but there are some limits. You can only have one management intent, and you can only have one storage intent. You can have “unlimited” compute intents, but you’re obviously going to hit some limits with how many physical NICs you can get in a node.

There is also a “stretch” intent type for S2D stretched clusters, but that’s in a bit of a funny spot right now, so I’ll pretend that doesn’t exist.

Many people are still doing “fully converged” intents, with all three traffic types in a single intent. This normally works, but you’re going to end up relying on QoS to properly prioritize storage, compute, and management traffic between your nodes. With modern PCIe Gen5 NVMe drives you can hit 100Gbps off a single drive, so on any high-performance cluster you’re going to want a decent bit of bandwidth allocated for that storage intent. If at all possible keep your storage intent separate, but I have no concerns doing management and compute on the same intent. You can specify a VLAN tag for your management traffic when you create the intent.

Overrides

Network ATC has its defaults for everything out of the box, but they may not perfectly fit your environment.

  • Global Overrides configure things like “Maximum VM Migrations” and “Live Migration Performance Selection”
  • Adapter Property Overrides configure things like RDMA Protocol, Jumbo Packet size, etc.
  • Storage Overrides configure things like Automatic IP Generation
  • QoS Policy Overrides configure… you guessed it, QoS Policies

Renaming Network Adapters

As I said, all adapters across the cluster need to be named the same, so if it helps you can pretty easily rename your adapters. You will need to substitute the proper -NewName depending on the order in which your adapters got named by default. You can see in this screenshot that the adapters on this node got added in a different order.

Get-NetAdapter | Sort MacAddress

Rename-NetAdapter -Name 'Ethernet' -NewName 'CX4-1'
Rename-NetAdapter -Name 'Ethernet 2' -NewName 'CX4-2'
Rename-NetAdapter -Name 'Ethernet 3' -NewName 'X710-1'
Rename-NetAdapter -Name 'Ethernet 4' -NewName 'X710-2'
Get-NetAdapter | Sort MacAddress

Rename-NetAdapter -Name 'Ethernet' -NewName 'CX4-1'
Rename-NetAdapter -Name 'Ethernet 2' -NewName 'CX4-2'
Rename-NetAdapter -Name 'Ethernet 3' -NewName 'X710-1'
Rename-NetAdapter -Name 'Ethernet 4' -NewName 'X710-2'

Get-NetAdapter sorted by MAC address, before and after renaming the ports CX4-1, CX4-2, X710-1 and X710-2

Commands and examples

Command Basics

Here are the basic PowerShell commands for Network ATC.

Get-NetIntent
#Get-NetIntent will provide the configuration information of the current intents

Get-NetIntentStatus
#Get-NetIntentStatus will provide status of the intents applying to each node in the cluster

Get-NetIntent -GlobalOverrides
#Get-NetIntent -GlobalOverrides will provide information about any global overrides that are configured. This info is not included in the base "Get-NetIntent"

Add-NetIntent
#Add-NetIntent will create a new Network ATC Intent

Remove-NetIntent
#Remove-NetIntent will... remove a Network ATC Intent

Set-NetIntent
#Set-NetIntent will change the configuration of an existing Intent

Set-NetIntentRetryState
#When things get messed up, and your intent fails to apply, this is how you make it retry.
Get-NetIntent
#Get-NetIntent will provide the configuration information of the current intents

Get-NetIntentStatus
#Get-NetIntentStatus will provide status of the intents applying to each node in the cluster

Get-NetIntent -GlobalOverrides
#Get-NetIntent -GlobalOverrides will provide information about any global overrides that are configured. This info is not included in the base "Get-NetIntent"

Add-NetIntent
#Add-NetIntent will create a new Network ATC Intent

Remove-NetIntent
#Remove-NetIntent will... remove a Network ATC Intent

Set-NetIntent
#Set-NetIntent will change the configuration of an existing Intent

Set-NetIntentRetryState
#When things get messed up, and your intent fails to apply, this is how you make it retry.

Basic Fully-Converged (Storage, Compute, Management)

Add-NetIntent -Name ConvergedIntent -Management -Compute -Storage -AdapterName CX4-1, CX4-2
Add-NetIntent -Name ConvergedIntent -Management -Compute -Storage -AdapterName CX4-1, CX4-2

2-node “switchless”, Management on VLAN 50

$MgmtAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$MgmtAdapterPropertyOverrides.NetworkDirect = 0 
$MgmtAdapterPropertyOverrides.JumboPacket = 1514 
$StorAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$StorAdapterPropertyOverrides.JumboPacket = 9014
#RoCEv2=4, iWARP=1
$StorAdapterPropertyOverrides.NetworkDirectTechnology = 4
$StorageOverride = New-NetIntentStorageOverrides 
$StorageOverride.EnableAutomaticIPGeneration = $false

Add-NetIntent -Name Management_Compute -Management -Compute -AdapterName X710-1, X710-2 -ManagementVlan 50 -AdapterPropertyOverrides $MgmtAdapterPropertyOverrides

Get-NetIntent

Add-NetIntent -Name Storage -Storage -AdapterName CX4-1, CX4-2 -AdapterPropertyOverrides $StorAdapterPropertyOverrides -StorageOverrides $StorageOverride
$MgmtAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$MgmtAdapterPropertyOverrides.NetworkDirect = 0 
$MgmtAdapterPropertyOverrides.JumboPacket = 1514 
$StorAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$StorAdapterPropertyOverrides.JumboPacket = 9014
#RoCEv2=4, iWARP=1
$StorAdapterPropertyOverrides.NetworkDirectTechnology = 4
$StorageOverride = New-NetIntentStorageOverrides 
$StorageOverride.EnableAutomaticIPGeneration = $false

Add-NetIntent -Name Management_Compute -Management -Compute -AdapterName X710-1, X710-2 -ManagementVlan 50 -AdapterPropertyOverrides $MgmtAdapterPropertyOverrides

Get-NetIntent

Add-NetIntent -Name Storage -Storage -AdapterName CX4-1, CX4-2 -AdapterPropertyOverrides $StorAdapterPropertyOverrides -StorageOverrides $StorageOverride

3-node “switchless”, Management on VLAN 50

Manually setting IPs

3-node switchless can be a little bit more complex. You have to maintain multiple storage networks that are different across the nodes.

param
 (
     [Parameter(Mandatory = $true)]
     [Int]
     $NodeID
 )

 $SwitchlessNodeID = ${NodeID} #Update to the current Node number being configured.
 $SwitchlessClusterNodes = 3 #Edit with number of nodes in the cluster.

 ##############################################
 #Setup Storage Network for Switchless Topology
 ##############################################
 $StorageSubnet = '10.71.0.0'
 $SingleStorageIPAddress = @('10.71.1','10.71.2','10.71.3','10.71.4','10.71.5','10.71.6')
 $DualStorageIPAddress = @('10.71.11','10.71.12','10.71.13','10.71.14','10.71.15','10.71.16')
 $StorageAddressPrefix = 29
 $supportedAdapters = @("Mellanox", "QLogic", "E810")
 $StorageAdapter = Get-NetAdapter | Where InterfaceDescription -Match ($supportedAdapters -Join "|") | ? Status -like Up | sort Name | Get-NetAdapterHardwareInfo | ? Slot -GE 1 | Sort-Object Slot,Function
 if ( $StorageAdapter ) {
     Write-Output 'These adapters will be used for storage (dependent on cluster size):'
     Write-Output $($StorageAdapter | Format-Table Name,Description,Slot,Function)
     Pause
 } else {
     throw 'No RDMA Storage Adapters found!'
 }

 $SingleStorageIPAddress = $SingleStorageIPAddress | ForEach-Object { if (($_).Substring(($_).Length -2) -match $SwitchlessNodeID) { $_ } }
 $DualStorageIPAddress = $DualStorageIPAddress | ForEach-Object { if (($_).Substring(($_).Length -2) -match $SwitchlessNodeID) { $_ } }
 $SingleStorageIPAddress = $SingleStorageIPAddress | ForEach-Object { $_ + '.' + $SwitchlessNodeID }
 $DualStorageIPAddress = $DualStorageIPAddress | ForEach-Object { $_ + '.' + $SwitchlessNodeID }
 $StorageSubnet = $StorageSubnet.Split('.')[0] + '.' + $StorageSubnet.Split('.')[1]
 $SingleStorageIPAddress = $SingleStorageIPAddress | ForEach-Object { $_.Replace('10.71',$StorageSubnet) }
 $DualStorageIPAddress = $DualStorageIPAddress | ForEach-Object { $_.Replace('10.72',$StorageSubnet) }
 Write-Output "Storage IP Addresses: $(($SingleStorageIPAddress)[0..($SwitchlessClusterNodes -2)]) ($(($DualStorageIPAddress )[0..($SwitchlessClusterNodes -2)]))"
 Pause
 ##

 #################################
 ## Assign IPs for Storage Network
 #################################
 if ( ($SwitchlessClusterNodes -1) -le $StorageAdapter.Count ) {
     Write-Output 'Configuring Single-Link Full Mesh Switchless Networks'
     for ($i=0;$i -lt ($SwitchlessClusterNodes -1);$i++) {
     Write-Output "Adapter: $(($StorageAdapter)[$i].Description) Name: $(($StorageAdapter)[$i].Name) IP: $($SingleStorageIPAddress[$i])"
     $null = New-NetIPAddress -InterfaceAlias ($StorageAdapter)[$i].Name -IPAddress $SingleStorageIPAddress[$i] -PrefixLength $StorageAddressPrefix -Verbose
     }
     if ( ($SwitchlessClusterNodes -1)*2 -le $StorageAdapter.Count ) {
         Write-Output 'Configuring Dual-Link Full Mesh Switchless Networks'
         $n = $SwitchlessClusterNodes -1
         for ($i=0;$i -lt ($SwitchlessClusterNodes -1);$i++) {
             Write-Output "Adapter: $(($StorageAdapter)[$n].Description) Name: $(($StorageAdapter)[$n].Name) IP: $($DualStorageIPAddress[$i])"
             $null = New-NetIPAddress -InterfaceAlias ($StorageAdapter)[$n].Name -IPAddress $DualStorageIPAddress[$i] -PrefixLength $StorageAddressPrefix -Verbose
             $n++
         }
     }
 } else {
     throw "Not enough Storage NICs available based on cluster size of $SwitchlessClusterNodes"
 }
param
 (
     [Parameter(Mandatory = $true)]
     [Int]
     $NodeID
 )

 $SwitchlessNodeID = ${NodeID} #Update to the current Node number being configured.
 $SwitchlessClusterNodes = 3 #Edit with number of nodes in the cluster.

 ##############################################
 #Setup Storage Network for Switchless Topology
 ##############################################
 $StorageSubnet = '10.71.0.0'
 $SingleStorageIPAddress = @('10.71.1','10.71.2','10.71.3','10.71.4','10.71.5','10.71.6')
 $DualStorageIPAddress = @('10.71.11','10.71.12','10.71.13','10.71.14','10.71.15','10.71.16')
 $StorageAddressPrefix = 29
 $supportedAdapters = @("Mellanox", "QLogic", "E810")
 $StorageAdapter = Get-NetAdapter | Where InterfaceDescription -Match ($supportedAdapters -Join "|") | ? Status -like Up | sort Name | Get-NetAdapterHardwareInfo | ? Slot -GE 1 | Sort-Object Slot,Function
 if ( $StorageAdapter ) {
     Write-Output 'These adapters will be used for storage (dependent on cluster size):'
     Write-Output $($StorageAdapter | Format-Table Name,Description,Slot,Function)
     Pause
 } else {
     throw 'No RDMA Storage Adapters found!'
 }

 $SingleStorageIPAddress = $SingleStorageIPAddress | ForEach-Object { if (($_).Substring(($_).Length -2) -match $SwitchlessNodeID) { $_ } }
 $DualStorageIPAddress = $DualStorageIPAddress | ForEach-Object { if (($_).Substring(($_).Length -2) -match $SwitchlessNodeID) { $_ } }
 $SingleStorageIPAddress = $SingleStorageIPAddress | ForEach-Object { $_ + '.' + $SwitchlessNodeID }
 $DualStorageIPAddress = $DualStorageIPAddress | ForEach-Object { $_ + '.' + $SwitchlessNodeID }
 $StorageSubnet = $StorageSubnet.Split('.')[0] + '.' + $StorageSubnet.Split('.')[1]
 $SingleStorageIPAddress = $SingleStorageIPAddress | ForEach-Object { $_.Replace('10.71',$StorageSubnet) }
 $DualStorageIPAddress = $DualStorageIPAddress | ForEach-Object { $_.Replace('10.72',$StorageSubnet) }
 Write-Output "Storage IP Addresses: $(($SingleStorageIPAddress)[0..($SwitchlessClusterNodes -2)]) ($(($DualStorageIPAddress )[0..($SwitchlessClusterNodes -2)]))"
 Pause
 ##

 #################################
 ## Assign IPs for Storage Network
 #################################
 if ( ($SwitchlessClusterNodes -1) -le $StorageAdapter.Count ) {
     Write-Output 'Configuring Single-Link Full Mesh Switchless Networks'
     for ($i=0;$i -lt ($SwitchlessClusterNodes -1);$i++) {
     Write-Output "Adapter: $(($StorageAdapter)[$i].Description) Name: $(($StorageAdapter)[$i].Name) IP: $($SingleStorageIPAddress[$i])"
     $null = New-NetIPAddress -InterfaceAlias ($StorageAdapter)[$i].Name -IPAddress $SingleStorageIPAddress[$i] -PrefixLength $StorageAddressPrefix -Verbose
     }
     if ( ($SwitchlessClusterNodes -1)*2 -le $StorageAdapter.Count ) {
         Write-Output 'Configuring Dual-Link Full Mesh Switchless Networks'
         $n = $SwitchlessClusterNodes -1
         for ($i=0;$i -lt ($SwitchlessClusterNodes -1);$i++) {
             Write-Output "Adapter: $(($StorageAdapter)[$n].Description) Name: $(($StorageAdapter)[$n].Name) IP: $($DualStorageIPAddress[$i])"
             $null = New-NetIPAddress -InterfaceAlias ($StorageAdapter)[$n].Name -IPAddress $DualStorageIPAddress[$i] -PrefixLength $StorageAddressPrefix -Verbose
             $n++
         }
     }
 } else {
     throw "Not enough Storage NICs available based on cluster size of $SwitchlessClusterNodes"
 }

Intent Creation

$Mgmt_compute_nics = @('NIC1','NIC2') 
$storage_nics = @('SLOT 3 Port 1','SLOT 3 Port 2','SLOT 4 Port 1','SLOT 4 Port 2')
$storage_vlan =@(700)
$Mgmt_Vlan=50
$MgmtAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$MgmtAdapterPropertyOverrides.NetworkDirect = 0 
$MgmtAdapterPropertyOverrides.JumboPacket = 1514 
$StorAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$StorAdapterPropertyOverrides.JumboPacket = 9014
#RoCEv2=4, iWARP=1
$StorAdapterPropertyOverrides.NetworkDirectTechnology = 4
$StorageOverride = New-NetIntentStorageOverrides 
$StorageOverride.EnableAutomaticIPGeneration = $false

Add-NetIntent -Name Management_Compute -Management -Compute -AdapterName $Mgmt_compute_nics -ManagementVlan $Mgmt_Vlan -AdapterPropertyOverrides $MgmtAdapterPropertyOverrides

Add-NetIntent -Name Storage -Storage -AdapterName $Storage_Nics -StorageVLANs $Storage_Vlan -AdapterPropertyOverrides $StorAdapterPropertyOverrides -StorageOverrides $StorageOverride
$Mgmt_compute_nics = @('NIC1','NIC2') 
$storage_nics = @('SLOT 3 Port 1','SLOT 3 Port 2','SLOT 4 Port 1','SLOT 4 Port 2')
$storage_vlan =@(700)
$Mgmt_Vlan=50
$MgmtAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$MgmtAdapterPropertyOverrides.NetworkDirect = 0 
$MgmtAdapterPropertyOverrides.JumboPacket = 1514 
$StorAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides 
$StorAdapterPropertyOverrides.JumboPacket = 9014
#RoCEv2=4, iWARP=1
$StorAdapterPropertyOverrides.NetworkDirectTechnology = 4
$StorageOverride = New-NetIntentStorageOverrides 
$StorageOverride.EnableAutomaticIPGeneration = $false

Add-NetIntent -Name Management_Compute -Management -Compute -AdapterName $Mgmt_compute_nics -ManagementVlan $Mgmt_Vlan -AdapterPropertyOverrides $MgmtAdapterPropertyOverrides

Add-NetIntent -Name Storage -Storage -AdapterName $Storage_Nics -StorageVLANs $Storage_Vlan -AdapterPropertyOverrides $StorAdapterPropertyOverrides -StorageOverrides $StorageOverride

Global Override example

$ClusterOverride = New-NetIntentGlobalClusterOverrides
$ClusterOverride.EnableNetworkNaming = $True
$ClusterOverride.EnableLiveMigrationNetworkSelection = $True
$ClusterOverride.EnableVirtualMachineMigrationPerformanceSelection = $False
$ClusterOverride.VirtualMachineMigrationPerformanceOption = "SMB"
$ClusterOverride.MaximumVirtualMachineMigrations = "4"
Set-NetIntent -GlobalClusterOverrides $ClusterOverride
$ClusterOverride = New-NetIntentGlobalClusterOverrides
$ClusterOverride.EnableNetworkNaming = $True
$ClusterOverride.EnableLiveMigrationNetworkSelection = $True
$ClusterOverride.EnableVirtualMachineMigrationPerformanceSelection = $False
$ClusterOverride.VirtualMachineMigrationPerformanceOption = "SMB"
$ClusterOverride.MaximumVirtualMachineMigrations = "4"
Set-NetIntent -GlobalClusterOverrides $ClusterOverride

Using non-RDMA NICs

If you are trying to use non-RDMA NICs, Network ATC will fail to create a storage intent because RDMA is required. This is the same problem you’d have trying to use Network ATC in a VM. Here are the necessary overrides to make it work. DO NOT DO THIS IF YOU HAVE RDMA-CAPABLE NICS.

$Override = New-NetIntentAdapterPropertyOverrides
$Override.JumboPacket = "9014"
$Override.NetworkDirect = $false

Add-NetIntent -Name Storage -Storage -AdapterName pNIC01, pNIC02 -AdapterPropertyOverrides $Override
$Override = New-NetIntentAdapterPropertyOverrides
$Override.JumboPacket = "9014"
$Override.NetworkDirect = $false

Add-NetIntent -Name Storage -Storage -AdapterName pNIC01, pNIC02 -AdapterPropertyOverrides $Override

For more setup configurations, Lee has a great blog post here.

Testing and troubleshooting

General Status

Get-NetIntentStatus
Get-NetIntentStatus

Get-NetIntentStatus output with every intent showing Success

If you have more than a handful of hosts, you may prefer this formatting:

Get-NetIntentStatus | Select IntentName,Host,ConfigurationStatus,ProvisioningStatus,RetryCount,Error,LastSuccess,LastUpdated,LastConfigApplied |ft
Get-NetIntentStatus | Select IntentName,Host,ConfigurationStatus,ProvisioningStatus,RetryCount,Error,LastSuccess,LastUpdated,LastConfigApplied |ft

Check the status of Global Overrides, which do not really display in Get-NetIntentStatus

Get-NetIntentStatus -GlobalOverrides
Get-NetIntentStatus -GlobalOverrides

  • If “Get-NetIntentStatus -GlobalOverrides” comes back with the Error “WindowsFeatureNotInstalled”, run this command then configure settings from WAC.
    Remove-NetIntent -GlobalOverrides
    Remove-NetIntent -GlobalOverrides

Retry provisioning

If something gets messed up, ATC will try to provision the intent three times. If your intent has a failure as seen here you can use Set-NetIntentRetryState to restart provisioning.

get-netintentstatus -name management_compute | Select IntentName, Error, ConfigurationStatus
Set-NetIntentRetryState -ClusterName ($CLUS) -Name ($IntentName)
get-netintentstatus -name management_compute | Select IntentName, Error, ConfigurationStatus
Set-NetIntentRetryState -ClusterName ($CLUS) -Name ($IntentName)

Get-NetIntentStatus showing the intent ProvisioningFailed on every node, then Set-NetIntentRetryState resetting the first node

Set-NetIntentRetryState on the remaining nodes, and the intent moving through Pending and Validating to Success

Retry refused: “InProgress goal states may not be retried”

Symptom: Get-NetIntentStatus shows ConfigurationStatus: Failed on a node, but Set-NetIntentRetryState refuses with:

InProgress goal states may not be retried. Current goal state status for intent storage on host <NODE> is Pending
InProgress goal states may not be retried. Current goal state status for intent storage on host <NODE> is Pending

ATC has used up its three automatic attempts, so it won’t retry on its own, and the retry cmdlet thinks a provision is still in flight.

1. Find the real error. ProvisioningFailed is generic. The actual exception is in the ATC Admin log:

Get-WinEvent -LogName 'Microsoft-Windows-Networking-NetworkATC/Admin' -MaxEvents 25 |
    Format-List TimeCreated, Id, LevelDisplayName, Message
Get-WinEvent -LogName 'Microsoft-Windows-Networking-NetworkATC/Admin' -MaxEvents 25 |
    Format-List TimeCreated, Id, LevelDisplayName, Message
For example, PropertyBag value null for Speed means an adapter briefly reported no link speed (a port re-negotiating) while ATC was setting properties. A transient blip like that is enough to fail all three attempts. Make sure the cause is gone before you retry.

2. Don’t re-issue the intent to reset it. Set-NetIntent or Update-NetIntentAdapter with the same adapters does not trigger provisioning on a failed host. ATC says so itself: “Running Set-NetIntent does not trigger provisioning on a failed host. Please run Set-NetIntentRetryState…” To confirm a node is stuck, compare the version you want with what each node applied:

# the version you WANT applied:
(Get-AllNetIntents)['storage'].ResourceContentVersion
# what each node ACTUALLY applied (LastAppliedRequestVersion + Status per node):
Get-NetIntentAllGoalStates
# the version you WANT applied:
(Get-AllNetIntents)['storage'].ResourceContentVersion
# what each node ACTUALLY applied (LastAppliedRequestVersion + Status per node):
Get-NetIntentAllGoalStates

3. Reset it with the cluster parameter set, not -ComputerName. On a cluster member, -ComputerName reads the node’s local goal-state store, which is where the stale “Pending” lives. -ClusterName with -NodeName reads the cluster store, which correctly shows “Failed” and can be reset:

# Not this on a cluster member (local store, sees "Pending", refuses):
# Set-NetIntentRetryState -Name storage -ComputerName <NODE>

Set-NetIntentRetryState -Name storage -ClusterName <CLUSTER> -NodeName <NODE> -Wait
# Not this on a cluster member (local store, sees "Pending", refuses):
# Set-NetIntentRetryState -Name storage -ComputerName <NODE>

Set-NetIntentRetryState -Name storage -ClusterName <CLUSTER> -NodeName <NODE> -Wait
Once it resets, RetryCount drops to 0 and the node re-provisions at the current intent version.

Run these from a console or RDP session on a node, not Enter-PSSession. Anything that modifies a cluster-scoped intent (Update-NetIntentAdapter, Set-NetIntent, the cluster-scoped retry reset) validates against every node over CIM. A remote session can’t pass your credentials on (Kerberos double hop), so you get Cannot connect to CIM server. Access is denied.

“Successfully updated intent” doesn’t mean your override applied

Symptom: Set-NetIntent prints “Successfully updated intent” and Get-NetIntentStatus is green on every node, but the adapter settings didn’t change.

Common cause: the override variable you passed is $null, usually from a typo. PowerShell doesn’t error on an undefined variable by default, so ATC re-applies the intent with no override and still reports success:

$MgmtAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides
$MgmtAdapterPropertyOverrides.NetworkDirect = 1
$MgmtAdapterPropertyOverrides.NetworkDirectTechnology = 4   # RoCEv2

# Bug: built ...Overrides (plural), passed ...Override (singular, never assigned = $null)
Set-NetIntent -Name compute_management -AdapterPropertyOverrides $MgmtAdapterPropertyOverride
$MgmtAdapterPropertyOverrides = New-NetIntentAdapterPropertyOverrides
$MgmtAdapterPropertyOverrides.NetworkDirect = 1
$MgmtAdapterPropertyOverrides.NetworkDirectTechnology = 4   # RoCEv2

# Bug: built ...Overrides (plural), passed ...Override (singular, never assigned = $null)
Set-NetIntent -Name compute_management -AdapterPropertyOverrides $MgmtAdapterPropertyOverride

Fix: read the override back after every Set-NetIntent and confirm your values landed:

(Get-NetIntent -Name compute_management).AdapterAdvancedParametersOverride |
    Select-Object NetworkDirect, NetworkDirectTechnology, JumboPacket
(Get-NetIntent -Name compute_management).AdapterAdvancedParametersOverride |
    Select-Object NetworkDirect, NetworkDirectTechnology, JumboPacket

Related:

  • Enable RDMA on any intent whose adapters support it, not just storage. RDMA on the management/compute adapters speeds up the SMB traffic that runs over them, like cluster-to-cluster live migration and copying ISOs to the hosts. Only turn it off (NetworkDirect = 0) on adapters that can’t do RDMA.
  • NetworkDirectTechnology values: 1 = iWARP, 2 = InfiniBand, 3 = RoCE, 4 = RoCEv2. RoCEv2 also needs the physical fabric configured for lossless traffic (PFC/ETS on that traffic class); it isn’t a host-only setting.

Double checks, extra validation, and tidbits

  • I’ve seen plenty of weird things going on with WAC Network ATC. This is basically a list of good things to double check.

  • Double check that Network ATC extension works in WAC. Mine did not, and I had to add a different extension feed to get a newer version of Network ATC than what was in the default feed.

    • WAC > Settings > Extensions > Feeds > Add https://aka.ms/sme-extension-feed
    • Install the newest Network ATC Extension.
  • Double check Network ATC Global Overrides.

    • WAC > Cluster > Network ATC Cluster Settings

      Network ATC cluster settings in Windows Admin Center

  • Check SMB Multichannel

    Get-SmbMultichannelConnection -SmbInstance SBL
    Get-SmbMultichannelConnection -SmbInstance SBL

  • ATC re-applies every intent about every 15 minutes (its drift check). Two consequences:

    • The Operational log (Microsoft-Windows-Networking-NetworkATC/Operational) logs “successfully provisioned / configuration drift remediated” every cycle. That’s boilerplate, not proof something was broken. Check the .../Admin log to see whether ATC actually changed anything.
    • If you monitor Get-NetIntentStatus, you’ll sometimes catch an intent mid-cycle in Provisioning or Validating. Don’t alert on “not Success”. Alert on a *Fail* status, a non-empty Error, or RetryCount > 0, sustained over a couple of polls. More in Monitoring S2D with Zabbix.