k8s-aks-troubleshooting

7 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

• skills/k8s-aks-troubleshooting/SKILL.md14 KBb5cbb9d2b8c9upstream ↗

  ---
  name: k8s-aks-troubleshooting
  description: >-
    Use when a problem is specific to Azure Kubernetes Service (AKS): a cluster or node pool
    in "Failed" state, an upgrade or scale that fails ("UpgradeFailed", "violate the pod's
    disruption budget", "QuotaExceeded", "AllocationFailed"), a LoadBalancer Service stuck
    at <pending> ("SyncLoadBalancerFailed", "AuthorizationFailed"), an Azure Disk or Files
    PVC that fails ("ProvisioningFailed", "volume node affinity conflict", "Multi-Attach
    error", "Mount error(13)"), workload identity errors ("AADSTS700016", "AADSTS70021"),
    CoreDNS edits that AKS reverts, "error dialing backend", Azure Policy denials, or API
    server access (authorized IP ranges, private clusters, kubelogin, "User does not have
    access to the resource in Azure"). Investigates with kubectl and read-only az commands,
    and proposes fixes. Reads first and asks before changing anything.
  license: Apache-2.0
  metadata:
    verified-with: "aks, kubernetes 1.33"
    verified-on: "2026-10-25"
  ---
  
  # AKS troubleshooting
  
  Find out why something that depends on Azure fails in an AKS cluster, and propose a fix.
  **Read first, ask before changing.** Run the investigation commands freely: kubectl reads
  and the read-only `az` commands below (`show`, `list`, `get`). Run a command under an "Ask
  first" line only after the user approves that exact command; if your credentials do not
  allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
  Every `az` change (update, upgrade, role assignments, anything that writes) is a proposed
  fix.
  
  An AKS cluster lives in two places. Kubernetes objects are in the cluster; the cluster
  itself, its **node pools** (virtual machine scale sets), load balancers, public IPs, disks
  and identities are Azure resources. The cluster resource is in the user's resource group;
  the resources AKS manages for it are in a separate **node resource group** (by default
  `MC_<resource group>_<cluster>_<region>`). Many AKS failures show in Kubernetes only as a
  short event, while the real error is in Azure: in the cluster's `provisioningState`, in
  the latest operation, or in the activity log.
  
  The commands use placeholders: `my-resource-group` and `my-cluster` (the cluster),
  `my-node-resource-group`, `my-nodepool`, `my-namespace`, `my-app`, `my-pvc`, `my-pod`,
  `my-node` and `my-identity`. Replace them with the real names. The `az` commands need an
  Azure CLI login (`az account show` tells which subscription it uses). Never read Secrets,
  and do not print tokens, keys or client secrets from any source.
  
  ## When to use
  
  - The cluster or a node pool shows `Failed`, or an upgrade, scale or update failed.
  - A LoadBalancer Service has no `EXTERNAL-IP`, or traffic through it fails.
  - A PVC on an Azure Disk or Azure Files class stays `Pending`, or a pod cannot mount it.
  - A pod using workload identity cannot get an Azure token.
  - A change to the `coredns` ConfigMap disappears, `kubectl logs` or `exec` fail with
    `error dialing backend`, or Azure Policy denies a pod.
  - kubectl cannot reach or log in to the cluster.
  
  Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill;
  if it is not installed, keep going with the steps here and tell the user which skill
  would help:
  
  - A pod that crashes for its own reasons (its logs, probes, OOMKilled):
    `k8s-workload-troubleshooting`. A node that is NotReady or full, or pods that do not
    fit: `k8s-node-troubleshooting`.
  - A PVC problem that is not about Azure (no StorageClass, access modes, a full volume):
    `k8s-storage-troubleshooting`.
  - A Service with no endpoints, or a NetworkPolicy: `k8s-network-troubleshooting`. The
    Corefile itself: `k8s-coredns-troubleshooting`.
  - A `Forbidden` that comes from Kubernetes RBAC (no Azure suffix):
    `k8s-rbac-troubleshooting`. Webhook or admission errors that are not Azure Policy:
    `k8s-api-troubleshooting`; Gatekeeper constraints themselves:
    `k8s-gatekeeper-troubleshooting`.
  - Before an upgrade, a check of everything in the cluster that would block or break it
    (PDBs, unmanaged pods, webhooks, deprecated APIs, add-ons): `k8s-upgrade-readiness`.
  
  ## Investigate
  
  **1. Find the cluster and its state.** If the user did not name it, list the clusters
  the login can see, then read the one in question:
  
  ```bash
  az aks list -o table
  az aks show -g my-resource-group -n my-cluster --query "{state:provisioningState, power:powerState.code, version:kubernetesVersion, nodeResourceGroup:nodeResourceGroup, network:networkProfile.networkPlugin, networkMode:networkProfile.networkPluginMode, outbound:networkProfile.outboundType, private:apiServerAccessProfile.enablePrivateCluster, authorizedIpRanges:apiServerAccessProfile.authorizedIpRanges, azureRbac:aadProfile.enableAzureRbac, localAccountsDisabled:disableLocalAccounts, oidcIssuer:oidcIssuerProfile.issuerUrl, workloadIdentity:securityProfile.workloadIdentity.enabled}" -o json
  ```
  
  **2. Node pools and recent operations.** A `Failed` state is about the last operation, not
  the running workloads:
  
  ```bash
  az aks nodepool list -g my-resource-group --cluster-name my-cluster --query "[].{name:name, mode:mode, state:provisioningState, power:powerState.code, count:count, min:minCount, max:maxCount, autoscale:enableAutoScaling, vmSize:vmSize, zones:availabilityZones, version:currentOrchestratorVersion, maxSurge:upgradeSettings.maxSurge}" -o table
  az monitor activity-log list -g my-resource-group --offset 3d --status Failed --query "[].{time:eventTimestamp, operation:operationName.value, status:status.value, message:properties.statusMessage}" -o json
  az aks operation show-latest -g my-resource-group -n my-cluster
  ```
  
  `az aks operation show-latest` needs the `aks-preview` extension; if it is missing, the
  activity log has the same error.
  
  **3. The Kubernetes side.**
  
  ```bash
  kubectl get nodes -L kubernetes.azure.com/agentpool,topology.kubernetes.io/zone,node.kubernetes.io/instance-type
  kubectl events -A --types=Warning
  kubectl get pods -n kube-system -o wide
  ```
  
  **4. Follow the symptom** to its reference, which has the commands for that area:
  
  | Symptom | Read |
  |---|---|
  | Cluster or node pool `Failed`, upgrade or scale errors, quota and VM capacity, autoscaler, node auto-repair, versions | [references/nodepools-upgrades.md](references/nodepools-upgrades.md) |
  | LoadBalancer Service `<pending>` or unreachable, `SyncLoadBalancerFailed`, static IPs, internal load balancers, health probes, SNAT | [references/load-balancers.md](references/load-balancers.md) |
  | Azure Disk or Azure Files PVC `Pending` or not mounting, zones, attach limits; workload identity and `AADSTS` errors | [references/storage-identity.md](references/storage-identity.md) |
  | CoreDNS customization, `error dialing backend`, Azure Policy, other managed add-ons; API server reachability, Entra ID login, Azure RBAC | [references/addons-access.md](references/addons-access.md) |
  
  ## Read the results
  
  - **`provisioningState` is `Failed` while nodes and pods run normally**: the last Azure
    operation on the cluster failed; the cluster itself usually keeps working. The activity
    log's `statusMessage` names the cause, often nested: an outer code such as
    `ResourceOperationFailure`, `UpgradeVMSSAgentPoolFailed` or `CreateOrUpdateExtensionFailed`
    wraps the real one in `details`. **`Update failed for this resource, as there is a
    conflicting operation in progress`** means two operations overlapped (for example an
    add-on update during a scale); nothing else needs fixing. Fix the real cause first, then
    reconcile the cluster (see Proposed fix). A reconcile does not fix an unresolved quota,
    policy, network or capacity error.
  - **`(UpgradeFailed) Drain node ... failed when evicting pod ... Cannot evict pod as it
    would violate the pod's disruption budget`**: a PodDisruptionBudget allows 0
    disruptions; see [references/nodepools-upgrades.md](references/nodepools-upgrades.md).
  - **`QuotaExceeded`**, **`AllocationFailed`**, **`ZonalAllocationFailed`**,
    **`SkuNotAvailable`**: Azure could not create the virtual machines, the subscription's
    quota or the region's capacity, not Kubernetes.
  - **A LoadBalancer Service stays `<pending>` with `SyncLoadBalancerFailed`**: the event's
    message is the Azure error from the cloud provider. **`AuthorizationFailed ... does not
    have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read'
    over scope '.../subnets/my-subnet' or the scope is invalid`** on an internal load
    balancer usually means that **subnet does not exist** (a typo in
    `azure-load-balancer-internal-subnet`): the cluster identity can read only the subnets
    it was granted, so a missing one looks like missing permission. Check the subnet exists
    before granting any role.
  - **A PVC `Pending` with `ProvisioningFailed ... RESPONSE 400 ... InvalidParameter`**: Azure
    refused the disk itself, for example **`The value 40960 of parameter 'disk.diskSizeGb' is
    out of range. The value must be between '1' and '32767', inclusive.`** (an Azure Disk is
    at most 32 TiB). The message after `ERROR CODE` is Azure's; the rest is the CSI driver.
  - **A workload identity pod logs `AADSTS700016: Application with identifier '<client ID>'
    was not found in the directory`**: the client ID in the ServiceAccount annotation
    `azure.workload.identity/client-id` belongs to no identity in that tenant. **`AADSTS70021:
    No matching federated identity record found for presented assertion`**: the identity
    exists, but no federated credential matches the token's issuer and subject
    (`system:serviceaccount:my-namespace:my-app`). See
    [references/storage-identity.md](references/storage-identity.md).
  - **`Error from server: error dialing backend: dial tcp <node ip>:10250: i/o timeout`** on
    `kubectl logs` or `exec`, while `get` works: the API server cannot reach the kubelet
    through the konnectivity tunnel; see
    [references/addons-access.md](references/addons-access.md).
  - **`... User does not have access to the resource in Azure. Update role assignment to
    allow access.`**: the cluster uses Azure RBAC for Kubernetes, and the caller's Azure role
    assignments, not Kubernetes RoleBindings, decide.
  
  ## Proposed fix
  
  Show each fix to the user before running it: what it changes and how to undo it. Every
  change to an Azure resource is a proposed fix the user runs; make it in the cluster's
  source of truth (Bicep, Terraform, ARM templates, a pipeline) when there is one, since a
  later deployment would undo a change made by hand. Kubernetes objects such as Services,
  PVCs and ServiceAccounts belong in the workload's manifests or chart.
  
  After the cause of a `Failed` state is fixed, a reconcile re-applies the cluster's current
  configuration without changing it:
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  az aks update -g my-resource-group -n my-cluster --yes
  # Undo: none needed; it re-applies the existing configuration
  ```
  
  For a node pool in `Failed`:
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  az aks nodepool update -g my-resource-group --cluster-name my-cluster -n my-nodepool
  # Undo: none needed; it re-applies the existing node pool configuration
  ```
  
  Never start another operation while one is still running (`az aks operation show-latest`
  shows `InProgress`), and never edit the scale sets, load balancers or disks in the node
  resource group by hand: AKS owns them, reverts such changes, and they are not supported.
  The fixes for each area, with their undo, are in the references.
  
  ## Sources
  
  Checked against AKS with Kubernetes 1.33 (a private cluster with Azure CNI, Entra ID and
  Azure RBAC, workload identity and the Azure Policy add-on) and the Microsoft Learn pages
  below, as of October 2026:
  
  - [Troubleshoot AKS upgrade, scaling and failed-state errors](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/availability-performance/cluster-node-virtual-machine-failed-state),
    [PodDrainFailure](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/create-upgrade-delete/error-code-poddrainfailure),
    [QuotaExceeded](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/error-codes/quota-exceeded-error)
    and [ZonalAllocationFailed and AllocationFailed](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/error-codes/zonalallocation-allocationfailed-error).
  - [Supported Kubernetes versions](https://learn.microsoft.com/azure/aks/supported-kubernetes-versions),
    [upgrade options](https://learn.microsoft.com/azure/aks/upgrade-options),
    [upgrade rules and version skew](https://learn.microsoft.com/azure/aks/upgrade-aks-control-plane#kubernetes-version-upgrade-rules),
    [node auto-repair](https://learn.microsoft.com/azure/aks/node-auto-repair) and
    [cluster autoscaler](https://learn.microsoft.com/azure/aks/cluster-autoscaler).
  - [Standard load balancer](https://learn.microsoft.com/azure/aks/configure-load-balancer-standard),
    [internal load balancer](https://learn.microsoft.com/azure/aks/internal-lb),
    [static public IP](https://learn.microsoft.com/azure/aks/static-ip) and
    [SNAT](https://learn.microsoft.com/azure/aks/troubleshoot-source-network-address-translation).
  - [Storage concepts](https://learn.microsoft.com/azure/aks/concepts-storage),
    [Azure Disk mount failures](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/storage/fail-to-mount-azure-disk-volume)
    and [Azure Files mount failures](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/storage/fail-to-mount-azure-file-share).
  - [Workload identity](https://learn.microsoft.com/azure/aks/workload-identity-overview).
  - [CoreDNS customization](https://learn.microsoft.com/azure/aks/coredns-custom),
    [tunnel connectivity](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/connectivity/tunnel-connectivity-issues),
    [API server connection issues](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/connectivity/troubleshoot-cluster-connection-issues-api-server),
    [kubelogin](https://learn.microsoft.com/azure/aks/kubelogin-authentication) and
    [Azure RBAC for Kubernetes](https://learn.microsoft.com/azure/aks/manage-azure-rbac).
  - Messages quoted from the test cluster: the internal load balancer `AuthorizationFailed`
    for a missing subnet, the Azure Disk `InvalidParameter` size error, `AADSTS700016` from
    a workload identity pod, and a `Failed` cluster after `CreateOrUpdateExtensionFailed`
    with `conflicting operation in progress`.
  

• skills/k8s-aks-troubleshooting/references/addons-access.md6.0 KBe7906bf1d093upstream ↗

  # Managed add-ons and API server access
  
  AKS runs and updates a set of components itself: CoreDNS, konnectivity-agent,
  metrics-server, the CSI drivers, the workload identity webhook, Azure Policy (Gatekeeper)
  and the monitoring agents (`ama-logs`, `ama-metrics`), mostly in `kube-system`. AKS puts
  them back to its own configuration, so changes go through the documented hooks below, not
  through edits of their objects.
  
  ## CoreDNS
  
  AKS owns the `coredns` ConfigMap and reverts edits to it. Customization goes in the
  ConfigMap `coredns-custom` in `kube-system`: keys ending in `.server` add server blocks
  (for example a stub domain), keys ending in `.override` add plugins to the default block.
  
  ```bash
  kubectl get configmap coredns-custom -n kube-system -o yaml
  kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
  kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50 --prefix
  ```
  
  - **An edit to the `coredns` ConfigMap is gone after a while**: expected; move it to
    `coredns-custom`.
  - **A `coredns-custom` change has no effect**: the key does not end in `.server` or
    `.override`, or CoreDNS was not restarted (it reads `coredns-custom` at start). A syntax
    error there crashes the CoreDNS pods like a Corefile error:
    `k8s-coredns-troubleshooting`.
  
  ## Logs, exec and the konnectivity tunnel
  
  The API server reaches the kubelets (for `logs`, `exec`, `port-forward` and webhooks that
  run in the cluster) through `konnectivity-agent` pods in `kube-system`, unless the cluster
  uses API Server VNet Integration.
  
  ```bash
  kubectl get pods -n kube-system -l app=konnectivity-agent -o wide
  kubectl logs -n kube-system -l app=konnectivity-agent --tail=50 --prefix
  ```
  
  - **`Error from server: error dialing backend: dial tcp <node ip>:10250: i/o timeout`**,
    or `logs` and `exec` failing while `get` works: the tunnel is down or the API server
    cannot reach the node. Usual causes: the konnectivity-agent pods are not Running, a
    network security group or firewall blocks the agent's outbound connection to the API
    server (TCP 443, or 9000 on older clusters), an egress firewall that strips the TLS
    ALPN extension, or SNAT exhaustion. Often only some nodes are affected.
  
  ## Azure Policy
  
  The Azure Policy add-on runs Gatekeeper; its constraints come from policy assignments in
  Azure and are named `azurepolicy-...`.
  
  ```bash
  kubectl events -n my-namespace --types=Warning
  kubectl get constraints
  az aks show -g my-resource-group -n my-cluster --query addonProfiles.azurepolicy -o json
  ```
  
  - **`admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-...]
    ...`**: an Azure Policy assignment in `deny` mode. Constraints in `dryrun` (audit) only
    report. Editing the constraint in the cluster does not last: the add-on rewrites it from
    Azure within about 15 minutes. The fix is in the workload (meet the policy) or in Azure
    (an exemption or a changed assignment), by whoever owns the policy.
  
  ## API server access
  
  ```bash
  az aks show -g my-resource-group -n my-cluster --query "{fqdn:fqdn, privateFqdn:privateFqdn, private:apiServerAccessProfile.enablePrivateCluster, authorizedIpRanges:apiServerAccessProfile.authorizedIpRanges, aad:aadProfile.managed, azureRbac:aadProfile.enableAzureRbac, localAccountsDisabled:disableLocalAccounts}" -o json
  kubectl auth whoami
  kubectl auth can-i list pods -n my-namespace
  ```
  
  - **`Unable to connect to the server: dial tcp <ip>:443: i/o timeout`**:
    - With `authorizedIpRanges` set, the client's public IP must be in the list.
    - A private cluster (`enablePrivateCluster: true`) answers only inside its network or
      peered networks, and its `privateFqdn` must resolve to a private IP there.
    - Otherwise a firewall or proxy on the client's side.
  - **`kubelogin is not installed`**, or a login prompt in a non-interactive tool: the
    kubeconfig uses Entra ID through the `kubelogin` exec plugin; it needs `kubelogin` and,
    for scripts, a non-interactive login mode. The user fixes their kubeconfig; never fetch
    credentials yourself.
  - **`You must be logged in to the server (Unauthorized)`**: the token expired or belongs
    to another tenant; the user logs in again.
  - **`Error from server (Forbidden): ... User does not have access to the resource in
    Azure. Update role assignment to allow access.`**: the cluster uses Azure RBAC for
    Kubernetes (`azureRbac: true`). The caller needs an Azure role such as **Azure Kubernetes
    Service RBAC Reader**, **Writer** or **Admin** on the cluster or a namespace scope;
    `az role assignment list` shows what they have. Without that suffix it is Kubernetes
    RBAC: `k8s-rbac-troubleshooting`.
  - **`Getting static credential isn't allowed because this cluster is set to disable local
    accounts`**: admin credentials are off (`disableLocalAccounts: true`); use Entra ID
    credentials.
  
  ## Proposed fix
  
  Add a client's IP to the authorized ranges. The flag replaces the whole list, so give the
  current ranges plus the new one:
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  az aks update -g my-resource-group -n my-cluster --api-server-authorized-ip-ranges 203.0.113.0/24,198.51.100.10/32
  # Undo: az aks update -g my-resource-group -n my-cluster --api-server-authorized-ip-ranges 203.0.113.0/24
  ```
  
  Give a user read access to one namespace through Azure RBAC (someone allowed to assign
  roles runs it; `my-cluster-id` is the cluster's resource ID from `az aks show --query id`):
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  az role assignment create --assignee [email protected] --role "Azure Kubernetes Service RBAC Reader" --scope my-cluster-id/namespaces/my-namespace
  # Undo: az role assignment delete --assignee [email protected] --role "Azure Kubernetes Service RBAC Reader" --scope my-cluster-id/namespaces/my-namespace
  ```
  
  Restart CoreDNS after changing `coredns-custom`:
  
  > Ask first: the agent runs this only after the user approves it.
  
  ```bash
  kubectl rollout restart deployment/coredns -n kube-system
  # Undo: none needed; the restart only recreates the pods with the same template
  ```
  

• skills/k8s-aks-troubleshooting/references/load-balancers.md6.0 KBa466740e03bbupstream ↗

  # LoadBalancer Services
  
  For a Service of `type: LoadBalancer`, the Azure cloud provider (part of the AKS control
  plane) adds a frontend, rules and a health probe to the load balancer `kubernetes`
  (public) or `kubernetes-internal` (internal) in the node resource group, and puts the
  frontend's IP in the Service's `EXTERNAL-IP`. Annotations under
  `service.beta.kubernetes.io/azure-...` change what it builds. Errors appear as
  `SyncLoadBalancerFailed` events on the Service, with the Azure error inside.
  
  ## Investigate
  
  ```bash
  kubectl get service my-app -n my-namespace -o wide
  kubectl describe service my-app -n my-namespace
  kubectl get service my-app -n my-namespace -o jsonpath='{.metadata.annotations}{"\n"}{.spec.externalTrafficPolicy}{"\n"}{.spec.loadBalancerSourceRanges}{"\n"}'
  kubectl get endpointslices -n my-namespace -l kubernetes.io/service-name=my-app
  ```
  
  The cluster's network and the Azure side (`my-node-resource-group` from `az aks show
  --query nodeResourceGroup`):
  
  ```bash
  az aks show -g my-resource-group -n my-cluster --query "{nodeResourceGroup:nodeResourceGroup, outbound:networkProfile.outboundType, loadBalancerProfile:networkProfile.loadBalancerProfile, identity:identity.type, principal:identity.principalId}" -o json
  az aks nodepool list -g my-resource-group --cluster-name my-cluster --query "[].{name:name, subnet:vnetSubnetId}" -o table
  az network lb list -g my-node-resource-group -o table
  az network lb frontend-ip list -g my-node-resource-group --lb-name kubernetes-internal -o table
  az network lb probe list -g my-node-resource-group --lb-name kubernetes -o table
  az network public-ip list -g my-node-resource-group -o table
  ```
  
  For an internal load balancer, list the subnets of the cluster's virtual network (its
  resource group and name are in the node pool's `vnetSubnetId`), and the roles of the
  cluster identity:
  
  ```bash
  az network vnet subnet list -g my-vnet-resource-group --vnet-name my-vnet --query "[].{name:name, prefix:addressPrefix}" -o table
  az role assignment list --assignee my-principal-id --all --query "[].{role:roleDefinitionName, scope:scope}" -o table
  ```
  
  ## Read the results
  
  - **`EXTERNAL-IP <pending>` with `EnsuringLoadBalancer` and `SyncLoadBalancerFailed`
    repeating**: read the Azure error inside the event (`ERROR CODE: ...` and its `message`).
  - **`AuthorizationFailed ... does not have authorization to perform action
    'Microsoft.Network/virtualNetworks/subnets/read' over scope '.../subnets/my-subnet' or the
    scope is invalid`** with `azure-load-balancer-internal-subnet` set: compare the subnet
    name with the virtual network's subnets. **If the subnet does not exist**, the annotation
    has a typo or names a subnet of another network; this is not a permission problem, and
    granting a role would not fix it. If the subnet exists, the cluster identity really lacks
    `Microsoft.Network/virtualNetworks/subnets/join/action` and `subnets/read` on it (usually
    through **Network Contributor**); `az role assignment list` shows what it has.
  - **`AuthorizationFailed ... 'Microsoft.Network/publicIPAddresses/read'`** or **`user
    supplied IP address was not found`** with `azure-pip-name` or `loadBalancerIP`: the
    public IP is not in the node resource group, so the Service must name its resource
    group with `service.beta.kubernetes.io/azure-load-balancer-resource-group` (the group's
    **name**, not its ID), and the cluster identity needs Network Contributor there.
  - **A frontend IP exists but traffic does not arrive**:
    - No endpoints: `k8s-network-troubleshooting`.
    - `externalTrafficPolicy: Local`: the health probe is HTTP `/healthz` on the Service's
      `healthCheckNodePort`, and only nodes with a ready pod pass.
    - With `Cluster`, an HTTP app's probe uses `/` unless
      `service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path` names a path
      that returns 200; a probe that fails on every node drops all traffic.
    - `loadBalancerSourceRanges` or `azure-allowed-ip-ranges` limit the source addresses in
      the network security group; the client must be in them.
  - **Outbound connections fail or time out under load**, with `outboundType:
    loadBalancer`: SNAT ports may be exhausted (each outbound IP gives 64,000 ports, shared
    by the nodes). `InvalidLoadBalancerProfileAllocatedOutboundPorts`: the requested ports
    per node do not fit the number of nodes and IPs.
  - Do not edit the `kubernetes` load balancer, its rules or the network security group in
    the node resource group by hand: the cloud provider rewrites them on the next sync.
  
  ## Proposed fix
  
  Fix the Service's annotation in the workload's manifest or chart, for example the right
  subnet name, and apply it there. The cloud provider retries on its own; the Service gets
  its IP within a minute or two. Changing the annotation on the live Service shows at once
  whether that fixes it; the next deployment from the source overwrites it either way:
  
  > Ask first: the agent runs this only after the user approves it.
  
  ```bash
  kubectl annotate service my-app -n my-namespace service.beta.kubernetes.io/azure-load-balancer-internal-subnet=my-subnet --overwrite
  # Undo: kubectl annotate service my-app -n my-namespace service.beta.kubernetes.io/azure-load-balancer-internal-subnet=my-missing-subnet --overwrite
  ```
  
  Only when the subnet or public IP exists and the identity really lacks access, grant the
  cluster identity Network Contributor on that scope (someone with rights to assign roles
  runs it):
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  az role assignment create --assignee my-principal-id --role "Network Contributor" --scope my-subnet-id
  # Undo: az role assignment delete --assignee my-principal-id --role "Network Contributor" --scope my-subnet-id
  ```
  
  SNAT exhaustion: more outbound IPs, or a NAT gateway (a larger change, planned with the
  network owner):
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  az aks update -g my-resource-group -n my-cluster --load-balancer-managed-outbound-ip-count 2
  # Undo: az aks update -g my-resource-group -n my-cluster --load-balancer-managed-outbound-ip-count 1
  ```
  

• skills/k8s-aks-troubleshooting/references/nodepools-upgrades.md8.1 KBa553c63cdb4bupstream ↗

  # Node pools, upgrades, capacity and versions
  
  Each node pool is a virtual machine scale set in the node resource group. AKS changes it
  through operations (create, scale, upgrade, update); one that fails leaves the pool, and
  often the cluster, in `provisioningState: Failed` until the next successful operation.
  
  ## Investigate
  
  ```bash
  az aks nodepool show -g my-resource-group --cluster-name my-cluster -n my-nodepool --query "{state:provisioningState, power:powerState.code, count:count, min:minCount, max:maxCount, autoscale:enableAutoScaling, vmSize:vmSize, zones:availabilityZones, version:currentOrchestratorVersion, nodeImage:nodeImageVersion, maxSurge:upgradeSettings.maxSurge, maxUnavailable:upgradeSettings.maxUnavailable}" -o json
  az aks operation show-latest -g my-resource-group -n my-cluster --nodepool-name my-nodepool
  az monitor activity-log list -g my-resource-group --offset 3d --status Failed --query "[].{time:eventTimestamp, operation:operationName.value, message:properties.statusMessage}" -o json
  az aks get-upgrades -g my-resource-group -n my-cluster -o table
  az aks nodepool get-upgrades -g my-resource-group --cluster-name my-cluster --nodepool-name my-nodepool -o table
  az aks show -g my-resource-group -n my-cluster --query "{autoUpgrade:autoUpgradeProfile, autoscalerProfile:autoScalerProfile}" -o json
  ```
  
  Kubernetes side of an upgrade, a drain or a scale:
  
  ```bash
  kubectl get pdb -A
  kubectl get nodes -L kubernetes.azure.com/agentpool,kubernetes.azure.com/node-image-version
  kubectl events -A --types=Warning
  kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml
  ```
  
  Capacity and quota, when VMs cannot be created (`my-region` is the cluster's location from
  `az aks show`):
  
  ```bash
  az vm list-usage -l my-region -o table
  az vm list-skus -l my-region --size Standard_D4s_v5 --zone -o table
  ```
  
  ## Read the results
  
  Failed operations:
  
  - The activity log's `statusMessage` holds the error as JSON. Read the innermost
    `details[].code` and `message`: `UpgradeVMSSAgentPoolFailed`, `ScaleVMSSAgentPoolFailed`
    and `ResourceOperationFailure` only say which operation failed.
  - **`Operation is not allowed because there's an in progress ... operation`**
    (`OperationNotAllowed`), or **`conflicting operation in progress`**: another operation
    ran at the same time. Wait until `az aks operation show-latest` shows it finished, then
    reconcile.
  - **`AKSOperationPreempted`**: a newer operation replaced this one.
  - **`RequestDisallowedByPolicy`**: an Azure Policy assignment on the subscription or
    resource group blocked the change (for example required tags). **`ResourceLocked`**: a
    delete or read-only lock on the node resource group.
  
  Upgrades:
  
  - **`(UpgradeFailed) Drain node aks-my-nodepool-... failed when evicting pod my-pod failed
    with Too Many Requests error ... Cannot evict pod as it would violate the pod's disruption
    budget.. PDB debug info: my-namespace/my-pod blocked by pdb my-pdb with 0 unready pods.`**
    and the event `Eviction blocked by Too Many Requests (usually a pdb)`: `kubectl get pdb
    -A` shows `ALLOWED DISRUPTIONS 0` for that budget, usually a `minAvailable` equal to the
    replica count, or a single replica with `minAvailable: 1`. **`Evict blocked by
    conflicting disruption budgets`**: two PDBs select the same pod.
  - **`UpgradeBlockedOnDeprecatedAPIUsage`**: workloads still call APIs that the target
    version removes; the upgrade is held until they stop or the user overrides it.
  - **No upgrade target in `get-upgrades`**, or **`OperationNotAllowed: Upgrading Kubernetes
    version ... is not allowed`**: a cluster on community support takes minor versions one
    step at a time (for example 1.33 to 1.34, then 1.35); only long-term-support clusters
    may skip to a higher LTS version. `az aks get-versions -l my-region -o table` lists
    what AKS supports in the region now; a version that has left standard support runs only
    with paid long-term support, and AKS will upgrade unsupported clusters itself (see the
    supported versions page in Sources for the current dates).
  - **`NodePoolMcVersionIncompatible ... Minor version of node pool cannot be more than 3
    versions less than control plane's version`**: since Kubernetes 1.28 a node pool may be
    up to three minor versions behind the control plane, and never ahead of it. Upgrade the
    old pool first, then the control plane.
  
  Capacity:
  
  - **`QuotaExceeded`**: **`Operation could not be completed as it results in exceeding
    approved standardDSv3Family Cores quota ... Current Limit: ..., Current Usage: ...,
    Additional Required: ...`**: compare with `az vm list-usage`. An upgrade needs extra
    room for the surge nodes (`maxSurge`, by default 10% on newer pools).
  - **`ZonalAllocationFailed`** (`We do not have sufficient capacity for the requested VM size
    in this zone`), **`AllocationFailed`**, **`OverconstrainedAllocationRequest`**: the region
    or zone has no capacity for that size right now. **`SkuNotAvailable`**: `az vm list-skus`
    shows `NotAvailableForSubscription` under restrictions for that size or zone.
  - **`SubnetIsFull`** or `InsufficientSubnetSize`: the node subnet (Azure CNI without
    overlay also gives every pod a subnet IP) has no free addresses for new nodes.
  
  Autoscaler and nodes:
  
  - The cluster autoscaler runs in the AKS control plane, so it has no pod. Its state is in
    the ConfigMap `cluster-autoscaler-status`, and its decisions are events from
    `cluster-autoscaler` (`NotTriggerScaleUp` on a pending pod, with the reason per node
    pool: a max count reached, a taint or selector no pool matches). A pool at `max` does
    not grow.
  - A manual scale of a pool with the autoscaler on fails with **`Cannot scale cluster
    autoscaler enabled node pool`**.
  - Node auto-repair: an unhealthy (NotReady) node is rebooted, then reimaged, then
    redeployed by AKS, with events from `aks-auto-repair` (`NodeRebootStart`,
    `NodeReimageStart`, `NodeRedeployStart`, and `...Error` when one fails). A node that
    keeps coming back is a node problem: `k8s-node-troubleshooting`.
  
  ## Proposed fix
  
  A PodDisruptionBudget that blocks the drain: give the workload more replicas than the
  budget's `minAvailable` (a budget with `maxUnavailable: 0` blocks every eviction whatever
  the count), or relax the budget in its manifest or chart, then retry the upgrade. Note the current
  replica count first (`kubectl get deployment my-app -n my-namespace`); raising it is
  reversible:
  
  > Ask first: the agent runs this only after the user approves it.
  
  ```bash
  kubectl scale deployment/my-app --replicas=3 -n my-namespace
  # Undo: kubectl scale deployment/my-app --replicas=2 -n my-namespace (the count you noted before)
  ```
  
  Then retry the upgrade to the same target version; AKS continues from where it stopped:
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  az aks nodepool upgrade -g my-resource-group --cluster-name my-cluster -n my-nodepool --kubernetes-version my-target-version
  # Undo: none; a node pool cannot be downgraded. my-target-version is the version the failed upgrade aimed at (az aks nodepool get-upgrades)
  ```
  
  Not enough room for surge nodes (quota or capacity): upgrade with fewer extra nodes (as
  below), or with `--max-surge 0` and `--max-unavailable 1`, which takes a node out at a time
  instead. Note the current `maxSurge` from `az aks nodepool show` first:
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  az aks nodepool update -g my-resource-group --cluster-name my-cluster -n my-nodepool --max-surge 1
  # Undo: the same command with the previous --max-surge from az aks nodepool show
  ```
  
  A quota increase is requested in the Azure portal (Quotas) by someone with rights on the
  subscription. A different VM size means a new node pool: AKS cannot change the size of an
  existing pool's VMs in place. After the cause is fixed, reconcile the pool or cluster as
  in SKILL.md.
  
  A cluster autoscaler maximum that is too low:
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  az aks nodepool update -g my-resource-group --cluster-name my-cluster -n my-nodepool --update-cluster-autoscaler --min-count 3 --max-count 6
  # Undo: the same command with the previous --min-count and --max-count from az aks nodepool show
  ```
  

• skills/k8s-aks-troubleshooting/references/storage-identity.md8.7 KB385c4c054871upstream ↗

  # Azure storage and workload identity
  
  ## Azure Disk and Azure Files
  
  AKS ships CSI drivers and StorageClasses for both. Azure Disk classes (`managed-csi`,
  `managed-csi-premium`, `managed-csi-premium-v2` from 1.35, and the older `default` and
  `managed-premium`) give `ReadWriteOnce` disks that attach to one node at a time, and use
  `WaitForFirstConsumer`, so a disk is created only once a pod is scheduled. Azure Files
  classes (`azurefile-csi`, `azurefile-csi-premium`) give SMB or NFS shares that many pods
  can mount (`ReadWriteMany`). On multi-zone clusters the disk classes use zone-redundant
  storage (ZRS) since 1.29; older or custom classes with LRS disks are zonal.
  
  ### Investigate storage
  
  ```bash
  kubectl get pvc my-pvc -n my-namespace
  kubectl describe pvc my-pvc -n my-namespace
  kubectl describe pod my-pod -n my-namespace
  kubectl get storageclass
  kubectl get storageclass managed-csi -o yaml
  kubectl get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,HANDLE:.spec.csi.volumeHandle,ZONE:.spec.nodeAffinity.required.nodeSelectorTerms
  kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type
  kubectl get csinode my-node -o jsonpath='{.spec.drivers}'
  kubectl logs -n kube-system -l app=csi-azuredisk-node -c azuredisk --tail=50 --prefix
  ```
  
  The CSI controllers that create and attach disks and shares run in the AKS control plane,
  so their errors reach you only as events on the PVC (provisioning) or the pod (attach).
  The `csi-azuredisk-node` and `csi-azurefile-node` pods on each node do the mounting; their
  `azuredisk` and `azurefile` containers log mount errors.
  
  A PV's `volumeHandle` is the disk's Azure resource ID; read it with:
  
  ```bash
  az disk show --ids my-disk-id --query "{size:diskSizeGb, sku:sku.name, zones:zones, state:diskState, managedBy:managedBy}" -o json
  ```
  
  ### Read the storage results
  
  - **`ProvisioningFailed ... failed to provision volume with StorageClass "managed-csi": rpc
    error: code = Internal desc = PUT .../providers/Microsoft.Compute/disks/pvc-... RESPONSE
    400: 400 Bad Request ERROR CODE: InvalidParameter`** followed by Azure's message: Azure
    refused the disk. **`The value 40960 of parameter 'disk.diskSizeGb' is out of range. The
    value must be between '1' and '32767', inclusive.`**: an Azure Disk holds at most 32 TiB
    (32767 GiB); the request in the PVC is too large. Other messages name the SKU, zone or
    quota that Azure rejected. The PVC stays `Pending` and the pod `Pending` with it, and no
    disk is created.
  - **`0/N nodes are available: N node(s) had volume node affinity conflict`**: the pod's
    existing disk is in a zone where it cannot run now (an LRS disk is zonal). Compare the
    PV's zone with the nodes' `topology.kubernetes.io/zone`.
  - **`AttachVolume.Attach failed ... Disk cannot be attached to the VM because it is not in
    the same zone as the VM`**: the same mismatch, seen at attach time.
  - **`N node(s) exceed max volume count`**: the nodes' VM size allows only so many data
    disks (`kubectl get csinode` shows `allocatable.count`).
  - **`Multi-Attach error for volume "pvc-..." Volume is already used by pod(s) ...`**: an
    Azure Disk is still attached to another node, usually during a rollout or after a node
    failure; it clears once the old pod is gone (a few minutes after a node failure). Use
    Azure Files for storage that several pods mount at once.
  - **Azure Files `Mount error(13): Permission denied`**: the storage account's firewall does
    not allow the node subnet, a private endpoint resolves to a public IP (no private DNS
    link), or the account key the driver uses changed. Do not read the driver's Secret to
    compare keys; ask the user to check the storage account in Azure.
  - **`Mount error(2): No such file or directory`**: the share does not exist.
  
  ### Fix storage
  
  The size, class or access mode of a PVC is fixed in its manifest or chart; a `Pending` PVC
  that was never bound can be deleted and created again with the right values, which the
  user does (a bound PVC's data goes with it):
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  kubectl delete pvc my-pvc -n my-namespace
  # Undo: kubectl apply -f my-pvc.yaml with the PVC's original manifest
  ```
  
  Then apply the corrected manifest from its source (for example `storage: 1Ti`). A zone
  mismatch is fixed with a ZRS class, `WaitForFirstConsumer`, or a node pool in the disk's
  zone.
  
  ## Workload identity
  
  A pod uses an Azure identity without secrets when:
  
  - its ServiceAccount has the annotation `azure.workload.identity/client-id` with the
    **client ID** of a user-assigned managed identity (or an app registration);
  - the pod **template** has the label `azure.workload.identity/use: "true"`; and
  - that identity has a **federated credential** whose issuer is the cluster's OIDC issuer
    URL and whose subject is `system:serviceaccount:my-namespace:my-app`, with the audience
    `api://AzureADTokenExchange`.
  
  The webhook `azure-wi-webhook` (Deployment `azure-wi-webhook-controller-manager` in
  `kube-system`) then injects `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_AUTHORITY_HOST` and
  `AZURE_FEDERATED_TOKEN_FILE` (a projected token) into the pod. The Azure SDK exchanges that
  token at Microsoft Entra ID for an access token.
  
  ### Investigate workload identity
  
  ```bash
  kubectl get serviceaccount my-app -n my-namespace -o jsonpath='{.metadata.annotations}{"\n"}'
  kubectl get pod my-pod -n my-namespace -o jsonpath='{.metadata.labels}{"\n"}{.spec.serviceAccountName}{"\n"}{.spec.containers[*].env[*].name}{"\n"}'
  kubectl logs my-pod -n my-namespace --tail=50
  kubectl get pods -n kube-system -l azure-workload-identity.io/system=true
  az aks show -g my-resource-group -n my-cluster --query "{issuer:oidcIssuerProfile.issuerUrl, enabled:securityProfile.workloadIdentity.enabled}" -o json
  az identity list --query "[].{name:name, resourceGroup:resourceGroup, clientId:clientId}" -o table
  az identity federated-credential list -g my-resource-group --identity-name my-identity --query "[].{name:name, issuer:issuer, subject:subject, audiences:audiences}" -o json
  ```
  
  Print only the names of the pod's environment variables, as above, and never the token
  file.
  
  ### Read the identity results
  
  - **`AADSTS700016: Application with identifier '<client ID>' was not found in the directory
    '<tenant>'`**: no identity or app in that tenant has this client ID. Compare the
    ServiceAccount's annotation with `az identity list`: a typo, an identity that was deleted
    and created again (new client ID), or one from another tenant.
  - **`AADSTS70021: No matching federated identity record found for presented assertion`**:
    the identity exists, but none of its federated credentials matches. Compare `issuer`
    with the cluster's issuer URL exactly (trailing slash included), and `subject` with
    `system:serviceaccount:my-namespace:my-app` (namespace and ServiceAccount name). A new
    credential can take a minute or two to work.
  - **`AADSTS700211`** (issuer), **`AADSTS700213`** (subject), **`AADSTS700212`** (audience)
    and **`AADSTS70025`** (no federated credentials at all) name the part that does not
    match.
  - **No `AZURE_` variables in the pod**: the label is missing on the pod template (on the
    ServiceAccount it does nothing), the pod started before the annotation was added, or the
    webhook is not running (`securityProfile.workloadIdentity.enabled` false).
  - The token works but Azure says **`AuthorizationFailed`** or **`403`**: the identity has
    no role on the resource it calls; that is an Azure role assignment, not a Kubernetes
    problem.
  
  ### Fix workload identity
  
  Set the right client ID in the ServiceAccount's manifest (the identity's `clientId` from
  `az identity list`), then restart the pods so the webhook injects it again. On the live
  objects:
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  kubectl annotate serviceaccount my-app -n my-namespace azure.workload.identity/client-id=my-client-id --overwrite
  # Undo: kubectl annotate serviceaccount my-app -n my-namespace azure.workload.identity/client-id=my-old-client-id --overwrite
  ```
  
  > Ask first: the agent runs this only after the user approves it.
  
  ```bash
  kubectl rollout restart deployment/my-app -n my-namespace
  # Undo: none needed; the restart only recreates the pods with the same template
  ```
  
  A missing or wrong federated credential is created on the identity by someone with rights
  on it (`my-issuer-url` from `az aks show`):
  
  > Proposed fix: review before running. The agent does not run this.
  
  ```bash
  az identity federated-credential create -g my-resource-group --identity-name my-identity -n my-app --issuer my-issuer-url --subject system:serviceaccount:my-namespace:my-app --audiences api://AzureADTokenExchange
  # Undo: az identity federated-credential delete -g my-resource-group --identity-name my-identity -n my-app
  ```
  

• skills/k8s-aks-troubleshooting/.claude-plugin/plugin.json1.2 KBab32f3c2ffceupstream ↗

  {
    "name": "k8s-aks-troubleshooting",
    "version": "0.9.0",
    "description": "Use when a problem is specific to Azure Kubernetes Service (AKS): a cluster or node pool in \"Failed\" state, an upgrade or scale that fails (\"UpgradeFailed\", \"violate the pod's disruption budget\", \"QuotaExceeded\", \"AllocationFailed\"), a LoadBalancer Service stuck at <pending> (\"SyncLoadBalancerFailed\", \"AuthorizationFailed\"), an Azure Disk or Files PVC that fails (\"ProvisioningFailed\", \"volume node affinity conflict\", \"Multi-Attach error\", \"Mount error(13)\"), workload identity errors (\"AADSTS700016\", \"AADSTS70021\"), CoreDNS edits that AKS reverts, \"error dialing backend\", Azure Policy denials, or API server access (authorized IP ranges, private clusters, kubelogin, \"User does not have access to the resource in Azure\"). Investigates with kubectl and read-only az commands, and proposes fixes. Reads first and asks before changing anything.",
    "author": {
      "name": "k8s-servant",
      "url": "https://github.com/k8s-servant"
    },
    "homepage": "https://github.com/k8s-servant/k8s-skills",
    "repository": "https://github.com/k8s-servant/k8s-skills",
    "license": "Apache-2.0",
    "skills": [
      "./"
    ]
  }
  

• README.mdupstream ↗

  Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/k8s-servant-k8s-skills-skills-k8s-aks-troubleshooting.md