# k8s-aks-troubleshooting Adopt this skill: https://proskills.md/skills/other/k8s-servant-k8s-skills-skills-k8s-aks-troubleshooting Page: https://proskills.md/skills/other/k8s-servant-k8s-skills-skills-k8s-aks-troubleshooting This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required. ## File: skills/k8s-aks-troubleshooting/SKILL.md size: 14537 sha: b5cbb9d2b8c91ac01b87290192fd0e226a21cf0c upstream: https://github.com/k8s-servant/k8s-skills/blob/main/skills/k8s-aks-troubleshooting/SKILL.md ````text --- name: k8s-aks-troubleshooting description: >- Use when a problem is specific to Azure Kubernetes Service (AKS): a cluster or node pool in "Failed" state, an upgrade or scale that fails ("UpgradeFailed", "violate the pod's disruption budget", "QuotaExceeded", "AllocationFailed"), a LoadBalancer Service stuck at ("SyncLoadBalancerFailed", "AuthorizationFailed"), an Azure Disk or Files PVC that fails ("ProvisioningFailed", "volume node affinity conflict", "Multi-Attach error", "Mount error(13)"), workload identity errors ("AADSTS700016", "AADSTS70021"), CoreDNS edits that AKS reverts, "error dialing backend", Azure Policy denials, or API server access (authorized IP ranges, private clusters, kubelogin, "User does not have access to the resource in Azure"). Investigates with kubectl and read-only az commands, and proposes fixes. Reads first and asks before changing anything. license: Apache-2.0 metadata: verified-with: "aks, kubernetes 1.33" verified-on: "2026-10-25" --- # AKS troubleshooting Find out why something that depends on Azure fails in an AKS cluster, and propose a fix. **Read first, ask before changing.** Run the investigation commands freely: kubectl reads and the read-only `az` commands below (`show`, `list`, `get`). Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user. Every `az` change (update, upgrade, role assignments, anything that writes) is a proposed fix. An AKS cluster lives in two places. Kubernetes objects are in the cluster; the cluster itself, its **node pools** (virtual machine scale sets), load balancers, public IPs, disks and identities are Azure resources. The cluster resource is in the user's resource group; the resources AKS manages for it are in a separate **node resource group** (by default `MC___`). Many AKS failures show in Kubernetes only as a short event, while the real error is in Azure: in the cluster's `provisioningState`, in the latest operation, or in the activity log. The commands use placeholders: `my-resource-group` and `my-cluster` (the cluster), `my-node-resource-group`, `my-nodepool`, `my-namespace`, `my-app`, `my-pvc`, `my-pod`, `my-node` and `my-identity`. Replace them with the real names. The `az` commands need an Azure CLI login (`az account show` tells which subscription it uses). Never read Secrets, and do not print tokens, keys or client secrets from any source. ## When to use - The cluster or a node pool shows `Failed`, or an upgrade, scale or update failed. - A LoadBalancer Service has no `EXTERNAL-IP`, or traffic through it fails. - A PVC on an Azure Disk or Azure Files class stays `Pending`, or a pod cannot mount it. - A pod using workload identity cannot get an Azure token. - A change to the `coredns` ConfigMap disappears, `kubectl logs` or `exec` fail with `error dialing backend`, or Azure Policy denies a pod. - kubectl cannot reach or log in to the cluster. Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help: - A pod that crashes for its own reasons (its logs, probes, OOMKilled): `k8s-workload-troubleshooting`. A node that is NotReady or full, or pods that do not fit: `k8s-node-troubleshooting`. - A PVC problem that is not about Azure (no StorageClass, access modes, a full volume): `k8s-storage-troubleshooting`. - A Service with no endpoints, or a NetworkPolicy: `k8s-network-troubleshooting`. The Corefile itself: `k8s-coredns-troubleshooting`. - A `Forbidden` that comes from Kubernetes RBAC (no Azure suffix): `k8s-rbac-troubleshooting`. Webhook or admission errors that are not Azure Policy: `k8s-api-troubleshooting`; Gatekeeper constraints themselves: `k8s-gatekeeper-troubleshooting`. - Before an upgrade, a check of everything in the cluster that would block or break it (PDBs, unmanaged pods, webhooks, deprecated APIs, add-ons): `k8s-upgrade-readiness`. ## Investigate **1. Find the cluster and its state.** If the user did not name it, list the clusters the login can see, then read the one in question: ```bash az aks list -o table az aks show -g my-resource-group -n my-cluster --query "{state:provisioningState, power:powerState.code, version:kubernetesVersion, nodeResourceGroup:nodeResourceGroup, network:networkProfile.networkPlugin, networkMode:networkProfile.networkPluginMode, outbound:networkProfile.outboundType, private:apiServerAccessProfile.enablePrivateCluster, authorizedIpRanges:apiServerAccessProfile.authorizedIpRanges, azureRbac:aadProfile.enableAzureRbac, localAccountsDisabled:disableLocalAccounts, oidcIssuer:oidcIssuerProfile.issuerUrl, workloadIdentity:securityProfile.workloadIdentity.enabled}" -o json ``` **2. Node pools and recent operations.** A `Failed` state is about the last operation, not the running workloads: ```bash az aks nodepool list -g my-resource-group --cluster-name my-cluster --query "[].{name:name, mode:mode, state:provisioningState, power:powerState.code, count:count, min:minCount, max:maxCount, autoscale:enableAutoScaling, vmSize:vmSize, zones:availabilityZones, version:currentOrchestratorVersion, maxSurge:upgradeSettings.maxSurge}" -o table az monitor activity-log list -g my-resource-group --offset 3d --status Failed --query "[].{time:eventTimestamp, operation:operationName.value, status:status.value, message:properties.statusMessage}" -o json az aks operation show-latest -g my-resource-group -n my-cluster ``` `az aks operation show-latest` needs the `aks-preview` extension; if it is missing, the activity log has the same error. **3. The Kubernetes side.** ```bash kubectl get nodes -L kubernetes.azure.com/agentpool,topology.kubernetes.io/zone,node.kubernetes.io/instance-type kubectl events -A --types=Warning kubectl get pods -n kube-system -o wide ``` **4. Follow the symptom** to its reference, which has the commands for that area: | Symptom | Read | |---|---| | Cluster or node pool `Failed`, upgrade or scale errors, quota and VM capacity, autoscaler, node auto-repair, versions | [references/nodepools-upgrades.md](references/nodepools-upgrades.md) | | LoadBalancer Service `` or unreachable, `SyncLoadBalancerFailed`, static IPs, internal load balancers, health probes, SNAT | [references/load-balancers.md](references/load-balancers.md) | | Azure Disk or Azure Files PVC `Pending` or not mounting, zones, attach limits; workload identity and `AADSTS` errors | [references/storage-identity.md](references/storage-identity.md) | | CoreDNS customization, `error dialing backend`, Azure Policy, other managed add-ons; API server reachability, Entra ID login, Azure RBAC | [references/addons-access.md](references/addons-access.md) | ## Read the results - **`provisioningState` is `Failed` while nodes and pods run normally**: the last Azure operation on the cluster failed; the cluster itself usually keeps working. The activity log's `statusMessage` names the cause, often nested: an outer code such as `ResourceOperationFailure`, `UpgradeVMSSAgentPoolFailed` or `CreateOrUpdateExtensionFailed` wraps the real one in `details`. **`Update failed for this resource, as there is a conflicting operation in progress`** means two operations overlapped (for example an add-on update during a scale); nothing else needs fixing. Fix the real cause first, then reconcile the cluster (see Proposed fix). A reconcile does not fix an unresolved quota, policy, network or capacity error. - **`(UpgradeFailed) Drain node ... failed when evicting pod ... Cannot evict pod as it would violate the pod's disruption budget`**: a PodDisruptionBudget allows 0 disruptions; see [references/nodepools-upgrades.md](references/nodepools-upgrades.md). - **`QuotaExceeded`**, **`AllocationFailed`**, **`ZonalAllocationFailed`**, **`SkuNotAvailable`**: Azure could not create the virtual machines, the subscription's quota or the region's capacity, not Kubernetes. - **A LoadBalancer Service stays `` with `SyncLoadBalancerFailed`**: the event's message is the Azure error from the cloud provider. **`AuthorizationFailed ... does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read' over scope '.../subnets/my-subnet' or the scope is invalid`** on an internal load balancer usually means that **subnet does not exist** (a typo in `azure-load-balancer-internal-subnet`): the cluster identity can read only the subnets it was granted, so a missing one looks like missing permission. Check the subnet exists before granting any role. - **A PVC `Pending` with `ProvisioningFailed ... RESPONSE 400 ... InvalidParameter`**: Azure refused the disk itself, for example **`The value 40960 of parameter 'disk.diskSizeGb' is out of range. The value must be between '1' and '32767', inclusive.`** (an Azure Disk is at most 32 TiB). The message after `ERROR CODE` is Azure's; the rest is the CSI driver. - **A workload identity pod logs `AADSTS700016: Application with identifier '' was not found in the directory`**: the client ID in the ServiceAccount annotation `azure.workload.identity/client-id` belongs to no identity in that tenant. **`AADSTS70021: No matching federated identity record found for presented assertion`**: the identity exists, but no federated credential matches the token's issuer and subject (`system:serviceaccount:my-namespace:my-app`). See [references/storage-identity.md](references/storage-identity.md). - **`Error from server: error dialing backend: dial tcp :10250: i/o timeout`** on `kubectl logs` or `exec`, while `get` works: the API server cannot reach the kubelet through the konnectivity tunnel; see [references/addons-access.md](references/addons-access.md). - **`... User does not have access to the resource in Azure. Update role assignment to allow access.`**: the cluster uses Azure RBAC for Kubernetes, and the caller's Azure role assignments, not Kubernetes RoleBindings, decide. ## Proposed fix Show each fix to the user before running it: what it changes and how to undo it. Every change to an Azure resource is a proposed fix the user runs; make it in the cluster's source of truth (Bicep, Terraform, ARM templates, a pipeline) when there is one, since a later deployment would undo a change made by hand. Kubernetes objects such as Services, PVCs and ServiceAccounts belong in the workload's manifests or chart. After the cause of a `Failed` state is fixed, a reconcile re-applies the cluster's current configuration without changing it: > Proposed fix: review before running. The agent does not run this. ```bash az aks update -g my-resource-group -n my-cluster --yes # Undo: none needed; it re-applies the existing configuration ``` For a node pool in `Failed`: > Proposed fix: review before running. The agent does not run this. ```bash az aks nodepool update -g my-resource-group --cluster-name my-cluster -n my-nodepool # Undo: none needed; it re-applies the existing node pool configuration ``` Never start another operation while one is still running (`az aks operation show-latest` shows `InProgress`), and never edit the scale sets, load balancers or disks in the node resource group by hand: AKS owns them, reverts such changes, and they are not supported. The fixes for each area, with their undo, are in the references. ## Sources Checked against AKS with Kubernetes 1.33 (a private cluster with Azure CNI, Entra ID and Azure RBAC, workload identity and the Azure Policy add-on) and the Microsoft Learn pages below, as of October 2026: - [Troubleshoot AKS upgrade, scaling and failed-state errors](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/availability-performance/cluster-node-virtual-machine-failed-state), [PodDrainFailure](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/create-upgrade-delete/error-code-poddrainfailure), [QuotaExceeded](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/error-codes/quota-exceeded-error) and [ZonalAllocationFailed and AllocationFailed](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/error-codes/zonalallocation-allocationfailed-error). - [Supported Kubernetes versions](https://learn.microsoft.com/azure/aks/supported-kubernetes-versions), [upgrade options](https://learn.microsoft.com/azure/aks/upgrade-options), [upgrade rules and version skew](https://learn.microsoft.com/azure/aks/upgrade-aks-control-plane#kubernetes-version-upgrade-rules), [node auto-repair](https://learn.microsoft.com/azure/aks/node-auto-repair) and [cluster autoscaler](https://learn.microsoft.com/azure/aks/cluster-autoscaler). - [Standard load balancer](https://learn.microsoft.com/azure/aks/configure-load-balancer-standard), [internal load balancer](https://learn.microsoft.com/azure/aks/internal-lb), [static public IP](https://learn.microsoft.com/azure/aks/static-ip) and [SNAT](https://learn.microsoft.com/azure/aks/troubleshoot-source-network-address-translation). - [Storage concepts](https://learn.microsoft.com/azure/aks/concepts-storage), [Azure Disk mount failures](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/storage/fail-to-mount-azure-disk-volume) and [Azure Files mount failures](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/storage/fail-to-mount-azure-file-share). - [Workload identity](https://learn.microsoft.com/azure/aks/workload-identity-overview). - [CoreDNS customization](https://learn.microsoft.com/azure/aks/coredns-custom), [tunnel connectivity](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/connectivity/tunnel-connectivity-issues), [API server connection issues](https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/connectivity/troubleshoot-cluster-connection-issues-api-server), [kubelogin](https://learn.microsoft.com/azure/aks/kubelogin-authentication) and [Azure RBAC for Kubernetes](https://learn.microsoft.com/azure/aks/manage-azure-rbac). - Messages quoted from the test cluster: the internal load balancer `AuthorizationFailed` for a missing subnet, the Azure Disk `InvalidParameter` size error, `AADSTS700016` from a workload identity pod, and a `Failed` cluster after `CreateOrUpdateExtensionFailed` with `conflicting operation in progress`. ```` ## File: skills/k8s-aks-troubleshooting/references/addons-access.md size: 6105 sha: e7906bf1d093b8220e2b2029aa155d80f3ab1090 upstream: https://github.com/k8s-servant/k8s-skills/blob/main/skills/k8s-aks-troubleshooting/references/addons-access.md ````text # Managed add-ons and API server access AKS runs and updates a set of components itself: CoreDNS, konnectivity-agent, metrics-server, the CSI drivers, the workload identity webhook, Azure Policy (Gatekeeper) and the monitoring agents (`ama-logs`, `ama-metrics`), mostly in `kube-system`. AKS puts them back to its own configuration, so changes go through the documented hooks below, not through edits of their objects. ## CoreDNS AKS owns the `coredns` ConfigMap and reverts edits to it. Customization goes in the ConfigMap `coredns-custom` in `kube-system`: keys ending in `.server` add server blocks (for example a stub domain), keys ending in `.override` add plugins to the default block. ```bash kubectl get configmap coredns-custom -n kube-system -o yaml kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50 --prefix ``` - **An edit to the `coredns` ConfigMap is gone after a while**: expected; move it to `coredns-custom`. - **A `coredns-custom` change has no effect**: the key does not end in `.server` or `.override`, or CoreDNS was not restarted (it reads `coredns-custom` at start). A syntax error there crashes the CoreDNS pods like a Corefile error: `k8s-coredns-troubleshooting`. ## Logs, exec and the konnectivity tunnel The API server reaches the kubelets (for `logs`, `exec`, `port-forward` and webhooks that run in the cluster) through `konnectivity-agent` pods in `kube-system`, unless the cluster uses API Server VNet Integration. ```bash kubectl get pods -n kube-system -l app=konnectivity-agent -o wide kubectl logs -n kube-system -l app=konnectivity-agent --tail=50 --prefix ``` - **`Error from server: error dialing backend: dial tcp :10250: i/o timeout`**, or `logs` and `exec` failing while `get` works: the tunnel is down or the API server cannot reach the node. Usual causes: the konnectivity-agent pods are not Running, a network security group or firewall blocks the agent's outbound connection to the API server (TCP 443, or 9000 on older clusters), an egress firewall that strips the TLS ALPN extension, or SNAT exhaustion. Often only some nodes are affected. ## Azure Policy The Azure Policy add-on runs Gatekeeper; its constraints come from policy assignments in Azure and are named `azurepolicy-...`. ```bash kubectl events -n my-namespace --types=Warning kubectl get constraints az aks show -g my-resource-group -n my-cluster --query addonProfiles.azurepolicy -o json ``` - **`admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-...] ...`**: an Azure Policy assignment in `deny` mode. Constraints in `dryrun` (audit) only report. Editing the constraint in the cluster does not last: the add-on rewrites it from Azure within about 15 minutes. The fix is in the workload (meet the policy) or in Azure (an exemption or a changed assignment), by whoever owns the policy. ## API server access ```bash az aks show -g my-resource-group -n my-cluster --query "{fqdn:fqdn, privateFqdn:privateFqdn, private:apiServerAccessProfile.enablePrivateCluster, authorizedIpRanges:apiServerAccessProfile.authorizedIpRanges, aad:aadProfile.managed, azureRbac:aadProfile.enableAzureRbac, localAccountsDisabled:disableLocalAccounts}" -o json kubectl auth whoami kubectl auth can-i list pods -n my-namespace ``` - **`Unable to connect to the server: dial tcp :443: i/o timeout`**: - With `authorizedIpRanges` set, the client's public IP must be in the list. - A private cluster (`enablePrivateCluster: true`) answers only inside its network or peered networks, and its `privateFqdn` must resolve to a private IP there. - Otherwise a firewall or proxy on the client's side. - **`kubelogin is not installed`**, or a login prompt in a non-interactive tool: the kubeconfig uses Entra ID through the `kubelogin` exec plugin; it needs `kubelogin` and, for scripts, a non-interactive login mode. The user fixes their kubeconfig; never fetch credentials yourself. - **`You must be logged in to the server (Unauthorized)`**: the token expired or belongs to another tenant; the user logs in again. - **`Error from server (Forbidden): ... User does not have access to the resource in Azure. Update role assignment to allow access.`**: the cluster uses Azure RBAC for Kubernetes (`azureRbac: true`). The caller needs an Azure role such as **Azure Kubernetes Service RBAC Reader**, **Writer** or **Admin** on the cluster or a namespace scope; `az role assignment list` shows what they have. Without that suffix it is Kubernetes RBAC: `k8s-rbac-troubleshooting`. - **`Getting static credential isn't allowed because this cluster is set to disable local accounts`**: admin credentials are off (`disableLocalAccounts: true`); use Entra ID credentials. ## Proposed fix Add a client's IP to the authorized ranges. The flag replaces the whole list, so give the current ranges plus the new one: > Proposed fix: review before running. The agent does not run this. ```bash az aks update -g my-resource-group -n my-cluster --api-server-authorized-ip-ranges 203.0.113.0/24,198.51.100.10/32 # Undo: az aks update -g my-resource-group -n my-cluster --api-server-authorized-ip-ranges 203.0.113.0/24 ``` Give a user read access to one namespace through Azure RBAC (someone allowed to assign roles runs it; `my-cluster-id` is the cluster's resource ID from `az aks show --query id`): > Proposed fix: review before running. The agent does not run this. ```bash az role assignment create --assignee my-user@example.com --role "Azure Kubernetes Service RBAC Reader" --scope my-cluster-id/namespaces/my-namespace # Undo: az role assignment delete --assignee my-user@example.com --role "Azure Kubernetes Service RBAC Reader" --scope my-cluster-id/namespaces/my-namespace ``` Restart CoreDNS after changing `coredns-custom`: > Ask first: the agent runs this only after the user approves it. ```bash kubectl rollout restart deployment/coredns -n kube-system # Undo: none needed; the restart only recreates the pods with the same template ``` ```` ## File: skills/k8s-aks-troubleshooting/references/load-balancers.md size: 6158 sha: a466740e03bb7e5e69749fbd0ba84ee8285e96bd upstream: https://github.com/k8s-servant/k8s-skills/blob/main/skills/k8s-aks-troubleshooting/references/load-balancers.md ````text # LoadBalancer Services For a Service of `type: LoadBalancer`, the Azure cloud provider (part of the AKS control plane) adds a frontend, rules and a health probe to the load balancer `kubernetes` (public) or `kubernetes-internal` (internal) in the node resource group, and puts the frontend's IP in the Service's `EXTERNAL-IP`. Annotations under `service.beta.kubernetes.io/azure-...` change what it builds. Errors appear as `SyncLoadBalancerFailed` events on the Service, with the Azure error inside. ## Investigate ```bash kubectl get service my-app -n my-namespace -o wide kubectl describe service my-app -n my-namespace kubectl get service my-app -n my-namespace -o jsonpath='{.metadata.annotations}{"\n"}{.spec.externalTrafficPolicy}{"\n"}{.spec.loadBalancerSourceRanges}{"\n"}' kubectl get endpointslices -n my-namespace -l kubernetes.io/service-name=my-app ``` The cluster's network and the Azure side (`my-node-resource-group` from `az aks show --query nodeResourceGroup`): ```bash az aks show -g my-resource-group -n my-cluster --query "{nodeResourceGroup:nodeResourceGroup, outbound:networkProfile.outboundType, loadBalancerProfile:networkProfile.loadBalancerProfile, identity:identity.type, principal:identity.principalId}" -o json az aks nodepool list -g my-resource-group --cluster-name my-cluster --query "[].{name:name, subnet:vnetSubnetId}" -o table az network lb list -g my-node-resource-group -o table az network lb frontend-ip list -g my-node-resource-group --lb-name kubernetes-internal -o table az network lb probe list -g my-node-resource-group --lb-name kubernetes -o table az network public-ip list -g my-node-resource-group -o table ``` For an internal load balancer, list the subnets of the cluster's virtual network (its resource group and name are in the node pool's `vnetSubnetId`), and the roles of the cluster identity: ```bash az network vnet subnet list -g my-vnet-resource-group --vnet-name my-vnet --query "[].{name:name, prefix:addressPrefix}" -o table az role assignment list --assignee my-principal-id --all --query "[].{role:roleDefinitionName, scope:scope}" -o table ``` ## Read the results - **`EXTERNAL-IP ` with `EnsuringLoadBalancer` and `SyncLoadBalancerFailed` repeating**: read the Azure error inside the event (`ERROR CODE: ...` and its `message`). - **`AuthorizationFailed ... does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read' over scope '.../subnets/my-subnet' or the scope is invalid`** with `azure-load-balancer-internal-subnet` set: compare the subnet name with the virtual network's subnets. **If the subnet does not exist**, the annotation has a typo or names a subnet of another network; this is not a permission problem, and granting a role would not fix it. If the subnet exists, the cluster identity really lacks `Microsoft.Network/virtualNetworks/subnets/join/action` and `subnets/read` on it (usually through **Network Contributor**); `az role assignment list` shows what it has. - **`AuthorizationFailed ... 'Microsoft.Network/publicIPAddresses/read'`** or **`user supplied IP address was not found`** with `azure-pip-name` or `loadBalancerIP`: the public IP is not in the node resource group, so the Service must name its resource group with `service.beta.kubernetes.io/azure-load-balancer-resource-group` (the group's **name**, not its ID), and the cluster identity needs Network Contributor there. - **A frontend IP exists but traffic does not arrive**: - No endpoints: `k8s-network-troubleshooting`. - `externalTrafficPolicy: Local`: the health probe is HTTP `/healthz` on the Service's `healthCheckNodePort`, and only nodes with a ready pod pass. - With `Cluster`, an HTTP app's probe uses `/` unless `service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path` names a path that returns 200; a probe that fails on every node drops all traffic. - `loadBalancerSourceRanges` or `azure-allowed-ip-ranges` limit the source addresses in the network security group; the client must be in them. - **Outbound connections fail or time out under load**, with `outboundType: loadBalancer`: SNAT ports may be exhausted (each outbound IP gives 64,000 ports, shared by the nodes). `InvalidLoadBalancerProfileAllocatedOutboundPorts`: the requested ports per node do not fit the number of nodes and IPs. - Do not edit the `kubernetes` load balancer, its rules or the network security group in the node resource group by hand: the cloud provider rewrites them on the next sync. ## Proposed fix Fix the Service's annotation in the workload's manifest or chart, for example the right subnet name, and apply it there. The cloud provider retries on its own; the Service gets its IP within a minute or two. Changing the annotation on the live Service shows at once whether that fixes it; the next deployment from the source overwrites it either way: > Ask first: the agent runs this only after the user approves it. ```bash kubectl annotate service my-app -n my-namespace service.beta.kubernetes.io/azure-load-balancer-internal-subnet=my-subnet --overwrite # Undo: kubectl annotate service my-app -n my-namespace service.beta.kubernetes.io/azure-load-balancer-internal-subnet=my-missing-subnet --overwrite ``` Only when the subnet or public IP exists and the identity really lacks access, grant the cluster identity Network Contributor on that scope (someone with rights to assign roles runs it): > Proposed fix: review before running. The agent does not run this. ```bash az role assignment create --assignee my-principal-id --role "Network Contributor" --scope my-subnet-id # Undo: az role assignment delete --assignee my-principal-id --role "Network Contributor" --scope my-subnet-id ``` SNAT exhaustion: more outbound IPs, or a NAT gateway (a larger change, planned with the network owner): > Proposed fix: review before running. The agent does not run this. ```bash az aks update -g my-resource-group -n my-cluster --load-balancer-managed-outbound-ip-count 2 # Undo: az aks update -g my-resource-group -n my-cluster --load-balancer-managed-outbound-ip-count 1 ``` ```` ## File: skills/k8s-aks-troubleshooting/references/nodepools-upgrades.md size: 8290 sha: a553c63cdb4bca8b797ff4ac878e3d561368ffb1 upstream: https://github.com/k8s-servant/k8s-skills/blob/main/skills/k8s-aks-troubleshooting/references/nodepools-upgrades.md ````text # Node pools, upgrades, capacity and versions Each node pool is a virtual machine scale set in the node resource group. AKS changes it through operations (create, scale, upgrade, update); one that fails leaves the pool, and often the cluster, in `provisioningState: Failed` until the next successful operation. ## Investigate ```bash az aks nodepool show -g my-resource-group --cluster-name my-cluster -n my-nodepool --query "{state:provisioningState, power:powerState.code, count:count, min:minCount, max:maxCount, autoscale:enableAutoScaling, vmSize:vmSize, zones:availabilityZones, version:currentOrchestratorVersion, nodeImage:nodeImageVersion, maxSurge:upgradeSettings.maxSurge, maxUnavailable:upgradeSettings.maxUnavailable}" -o json az aks operation show-latest -g my-resource-group -n my-cluster --nodepool-name my-nodepool az monitor activity-log list -g my-resource-group --offset 3d --status Failed --query "[].{time:eventTimestamp, operation:operationName.value, message:properties.statusMessage}" -o json az aks get-upgrades -g my-resource-group -n my-cluster -o table az aks nodepool get-upgrades -g my-resource-group --cluster-name my-cluster --nodepool-name my-nodepool -o table az aks show -g my-resource-group -n my-cluster --query "{autoUpgrade:autoUpgradeProfile, autoscalerProfile:autoScalerProfile}" -o json ``` Kubernetes side of an upgrade, a drain or a scale: ```bash kubectl get pdb -A kubectl get nodes -L kubernetes.azure.com/agentpool,kubernetes.azure.com/node-image-version kubectl events -A --types=Warning kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml ``` Capacity and quota, when VMs cannot be created (`my-region` is the cluster's location from `az aks show`): ```bash az vm list-usage -l my-region -o table az vm list-skus -l my-region --size Standard_D4s_v5 --zone -o table ``` ## Read the results Failed operations: - The activity log's `statusMessage` holds the error as JSON. Read the innermost `details[].code` and `message`: `UpgradeVMSSAgentPoolFailed`, `ScaleVMSSAgentPoolFailed` and `ResourceOperationFailure` only say which operation failed. - **`Operation is not allowed because there's an in progress ... operation`** (`OperationNotAllowed`), or **`conflicting operation in progress`**: another operation ran at the same time. Wait until `az aks operation show-latest` shows it finished, then reconcile. - **`AKSOperationPreempted`**: a newer operation replaced this one. - **`RequestDisallowedByPolicy`**: an Azure Policy assignment on the subscription or resource group blocked the change (for example required tags). **`ResourceLocked`**: a delete or read-only lock on the node resource group. Upgrades: - **`(UpgradeFailed) Drain node aks-my-nodepool-... failed when evicting pod my-pod failed with Too Many Requests error ... Cannot evict pod as it would violate the pod's disruption budget.. PDB debug info: my-namespace/my-pod blocked by pdb my-pdb with 0 unready pods.`** and the event `Eviction blocked by Too Many Requests (usually a pdb)`: `kubectl get pdb -A` shows `ALLOWED DISRUPTIONS 0` for that budget, usually a `minAvailable` equal to the replica count, or a single replica with `minAvailable: 1`. **`Evict blocked by conflicting disruption budgets`**: two PDBs select the same pod. - **`UpgradeBlockedOnDeprecatedAPIUsage`**: workloads still call APIs that the target version removes; the upgrade is held until they stop or the user overrides it. - **No upgrade target in `get-upgrades`**, or **`OperationNotAllowed: Upgrading Kubernetes version ... is not allowed`**: a cluster on community support takes minor versions one step at a time (for example 1.33 to 1.34, then 1.35); only long-term-support clusters may skip to a higher LTS version. `az aks get-versions -l my-region -o table` lists what AKS supports in the region now; a version that has left standard support runs only with paid long-term support, and AKS will upgrade unsupported clusters itself (see the supported versions page in Sources for the current dates). - **`NodePoolMcVersionIncompatible ... Minor version of node pool cannot be more than 3 versions less than control plane's version`**: since Kubernetes 1.28 a node pool may be up to three minor versions behind the control plane, and never ahead of it. Upgrade the old pool first, then the control plane. Capacity: - **`QuotaExceeded`**: **`Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota ... Current Limit: ..., Current Usage: ..., Additional Required: ...`**: compare with `az vm list-usage`. An upgrade needs extra room for the surge nodes (`maxSurge`, by default 10% on newer pools). - **`ZonalAllocationFailed`** (`We do not have sufficient capacity for the requested VM size in this zone`), **`AllocationFailed`**, **`OverconstrainedAllocationRequest`**: the region or zone has no capacity for that size right now. **`SkuNotAvailable`**: `az vm list-skus` shows `NotAvailableForSubscription` under restrictions for that size or zone. - **`SubnetIsFull`** or `InsufficientSubnetSize`: the node subnet (Azure CNI without overlay also gives every pod a subnet IP) has no free addresses for new nodes. Autoscaler and nodes: - The cluster autoscaler runs in the AKS control plane, so it has no pod. Its state is in the ConfigMap `cluster-autoscaler-status`, and its decisions are events from `cluster-autoscaler` (`NotTriggerScaleUp` on a pending pod, with the reason per node pool: a max count reached, a taint or selector no pool matches). A pool at `max` does not grow. - A manual scale of a pool with the autoscaler on fails with **`Cannot scale cluster autoscaler enabled node pool`**. - Node auto-repair: an unhealthy (NotReady) node is rebooted, then reimaged, then redeployed by AKS, with events from `aks-auto-repair` (`NodeRebootStart`, `NodeReimageStart`, `NodeRedeployStart`, and `...Error` when one fails). A node that keeps coming back is a node problem: `k8s-node-troubleshooting`. ## Proposed fix A PodDisruptionBudget that blocks the drain: give the workload more replicas than the budget's `minAvailable` (a budget with `maxUnavailable: 0` blocks every eviction whatever the count), or relax the budget in its manifest or chart, then retry the upgrade. Note the current replica count first (`kubectl get deployment my-app -n my-namespace`); raising it is reversible: > Ask first: the agent runs this only after the user approves it. ```bash kubectl scale deployment/my-app --replicas=3 -n my-namespace # Undo: kubectl scale deployment/my-app --replicas=2 -n my-namespace (the count you noted before) ``` Then retry the upgrade to the same target version; AKS continues from where it stopped: > Proposed fix: review before running. The agent does not run this. ```bash az aks nodepool upgrade -g my-resource-group --cluster-name my-cluster -n my-nodepool --kubernetes-version my-target-version # Undo: none; a node pool cannot be downgraded. my-target-version is the version the failed upgrade aimed at (az aks nodepool get-upgrades) ``` Not enough room for surge nodes (quota or capacity): upgrade with fewer extra nodes (as below), or with `--max-surge 0` and `--max-unavailable 1`, which takes a node out at a time instead. Note the current `maxSurge` from `az aks nodepool show` first: > Proposed fix: review before running. The agent does not run this. ```bash az aks nodepool update -g my-resource-group --cluster-name my-cluster -n my-nodepool --max-surge 1 # Undo: the same command with the previous --max-surge from az aks nodepool show ``` A quota increase is requested in the Azure portal (Quotas) by someone with rights on the subscription. A different VM size means a new node pool: AKS cannot change the size of an existing pool's VMs in place. After the cause is fixed, reconcile the pool or cluster as in SKILL.md. A cluster autoscaler maximum that is too low: > Proposed fix: review before running. The agent does not run this. ```bash az aks nodepool update -g my-resource-group --cluster-name my-cluster -n my-nodepool --update-cluster-autoscaler --min-count 3 --max-count 6 # Undo: the same command with the previous --min-count and --max-count from az aks nodepool show ``` ```` ## File: skills/k8s-aks-troubleshooting/references/storage-identity.md size: 8880 sha: 385c4c054871bcdd7d26d7506e643ed6c9cc1922 upstream: https://github.com/k8s-servant/k8s-skills/blob/main/skills/k8s-aks-troubleshooting/references/storage-identity.md ````text # Azure storage and workload identity ## Azure Disk and Azure Files AKS ships CSI drivers and StorageClasses for both. Azure Disk classes (`managed-csi`, `managed-csi-premium`, `managed-csi-premium-v2` from 1.35, and the older `default` and `managed-premium`) give `ReadWriteOnce` disks that attach to one node at a time, and use `WaitForFirstConsumer`, so a disk is created only once a pod is scheduled. Azure Files classes (`azurefile-csi`, `azurefile-csi-premium`) give SMB or NFS shares that many pods can mount (`ReadWriteMany`). On multi-zone clusters the disk classes use zone-redundant storage (ZRS) since 1.29; older or custom classes with LRS disks are zonal. ### Investigate storage ```bash kubectl get pvc my-pvc -n my-namespace kubectl describe pvc my-pvc -n my-namespace kubectl describe pod my-pod -n my-namespace kubectl get storageclass kubectl get storageclass managed-csi -o yaml kubectl get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,HANDLE:.spec.csi.volumeHandle,ZONE:.spec.nodeAffinity.required.nodeSelectorTerms kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type kubectl get csinode my-node -o jsonpath='{.spec.drivers}' kubectl logs -n kube-system -l app=csi-azuredisk-node -c azuredisk --tail=50 --prefix ``` The CSI controllers that create and attach disks and shares run in the AKS control plane, so their errors reach you only as events on the PVC (provisioning) or the pod (attach). The `csi-azuredisk-node` and `csi-azurefile-node` pods on each node do the mounting; their `azuredisk` and `azurefile` containers log mount errors. A PV's `volumeHandle` is the disk's Azure resource ID; read it with: ```bash az disk show --ids my-disk-id --query "{size:diskSizeGb, sku:sku.name, zones:zones, state:diskState, managedBy:managedBy}" -o json ``` ### Read the storage results - **`ProvisioningFailed ... failed to provision volume with StorageClass "managed-csi": rpc error: code = Internal desc = PUT .../providers/Microsoft.Compute/disks/pvc-... RESPONSE 400: 400 Bad Request ERROR CODE: InvalidParameter`** followed by Azure's message: Azure refused the disk. **`The value 40960 of parameter 'disk.diskSizeGb' is out of range. The value must be between '1' and '32767', inclusive.`**: an Azure Disk holds at most 32 TiB (32767 GiB); the request in the PVC is too large. Other messages name the SKU, zone or quota that Azure rejected. The PVC stays `Pending` and the pod `Pending` with it, and no disk is created. - **`0/N nodes are available: N node(s) had volume node affinity conflict`**: the pod's existing disk is in a zone where it cannot run now (an LRS disk is zonal). Compare the PV's zone with the nodes' `topology.kubernetes.io/zone`. - **`AttachVolume.Attach failed ... Disk cannot be attached to the VM because it is not in the same zone as the VM`**: the same mismatch, seen at attach time. - **`N node(s) exceed max volume count`**: the nodes' VM size allows only so many data disks (`kubectl get csinode` shows `allocatable.count`). - **`Multi-Attach error for volume "pvc-..." Volume is already used by pod(s) ...`**: an Azure Disk is still attached to another node, usually during a rollout or after a node failure; it clears once the old pod is gone (a few minutes after a node failure). Use Azure Files for storage that several pods mount at once. - **Azure Files `Mount error(13): Permission denied`**: the storage account's firewall does not allow the node subnet, a private endpoint resolves to a public IP (no private DNS link), or the account key the driver uses changed. Do not read the driver's Secret to compare keys; ask the user to check the storage account in Azure. - **`Mount error(2): No such file or directory`**: the share does not exist. ### Fix storage The size, class or access mode of a PVC is fixed in its manifest or chart; a `Pending` PVC that was never bound can be deleted and created again with the right values, which the user does (a bound PVC's data goes with it): > Proposed fix: review before running. The agent does not run this. ```bash kubectl delete pvc my-pvc -n my-namespace # Undo: kubectl apply -f my-pvc.yaml with the PVC's original manifest ``` Then apply the corrected manifest from its source (for example `storage: 1Ti`). A zone mismatch is fixed with a ZRS class, `WaitForFirstConsumer`, or a node pool in the disk's zone. ## Workload identity A pod uses an Azure identity without secrets when: - its ServiceAccount has the annotation `azure.workload.identity/client-id` with the **client ID** of a user-assigned managed identity (or an app registration); - the pod **template** has the label `azure.workload.identity/use: "true"`; and - that identity has a **federated credential** whose issuer is the cluster's OIDC issuer URL and whose subject is `system:serviceaccount:my-namespace:my-app`, with the audience `api://AzureADTokenExchange`. The webhook `azure-wi-webhook` (Deployment `azure-wi-webhook-controller-manager` in `kube-system`) then injects `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_AUTHORITY_HOST` and `AZURE_FEDERATED_TOKEN_FILE` (a projected token) into the pod. The Azure SDK exchanges that token at Microsoft Entra ID for an access token. ### Investigate workload identity ```bash kubectl get serviceaccount my-app -n my-namespace -o jsonpath='{.metadata.annotations}{"\n"}' kubectl get pod my-pod -n my-namespace -o jsonpath='{.metadata.labels}{"\n"}{.spec.serviceAccountName}{"\n"}{.spec.containers[*].env[*].name}{"\n"}' kubectl logs my-pod -n my-namespace --tail=50 kubectl get pods -n kube-system -l azure-workload-identity.io/system=true az aks show -g my-resource-group -n my-cluster --query "{issuer:oidcIssuerProfile.issuerUrl, enabled:securityProfile.workloadIdentity.enabled}" -o json az identity list --query "[].{name:name, resourceGroup:resourceGroup, clientId:clientId}" -o table az identity federated-credential list -g my-resource-group --identity-name my-identity --query "[].{name:name, issuer:issuer, subject:subject, audiences:audiences}" -o json ``` Print only the names of the pod's environment variables, as above, and never the token file. ### Read the identity results - **`AADSTS700016: Application with identifier '' was not found in the directory ''`**: no identity or app in that tenant has this client ID. Compare the ServiceAccount's annotation with `az identity list`: a typo, an identity that was deleted and created again (new client ID), or one from another tenant. - **`AADSTS70021: No matching federated identity record found for presented assertion`**: the identity exists, but none of its federated credentials matches. Compare `issuer` with the cluster's issuer URL exactly (trailing slash included), and `subject` with `system:serviceaccount:my-namespace:my-app` (namespace and ServiceAccount name). A new credential can take a minute or two to work. - **`AADSTS700211`** (issuer), **`AADSTS700213`** (subject), **`AADSTS700212`** (audience) and **`AADSTS70025`** (no federated credentials at all) name the part that does not match. - **No `AZURE_` variables in the pod**: the label is missing on the pod template (on the ServiceAccount it does nothing), the pod started before the annotation was added, or the webhook is not running (`securityProfile.workloadIdentity.enabled` false). - The token works but Azure says **`AuthorizationFailed`** or **`403`**: the identity has no role on the resource it calls; that is an Azure role assignment, not a Kubernetes problem. ### Fix workload identity Set the right client ID in the ServiceAccount's manifest (the identity's `clientId` from `az identity list`), then restart the pods so the webhook injects it again. On the live objects: > Proposed fix: review before running. The agent does not run this. ```bash kubectl annotate serviceaccount my-app -n my-namespace azure.workload.identity/client-id=my-client-id --overwrite # Undo: kubectl annotate serviceaccount my-app -n my-namespace azure.workload.identity/client-id=my-old-client-id --overwrite ``` > Ask first: the agent runs this only after the user approves it. ```bash kubectl rollout restart deployment/my-app -n my-namespace # Undo: none needed; the restart only recreates the pods with the same template ``` A missing or wrong federated credential is created on the identity by someone with rights on it (`my-issuer-url` from `az aks show`): > Proposed fix: review before running. The agent does not run this. ```bash az identity federated-credential create -g my-resource-group --identity-name my-identity -n my-app --issuer my-issuer-url --subject system:serviceaccount:my-namespace:my-app --audiences api://AzureADTokenExchange # Undo: az identity federated-credential delete -g my-resource-group --identity-name my-identity -n my-app ``` ```` ## File: skills/k8s-aks-troubleshooting/.claude-plugin/plugin.json size: 1225 sha: ab32f3c2ffced260fbb402e64b9e685cc0e65a91 upstream: https://github.com/k8s-servant/k8s-skills/blob/main/skills/k8s-aks-troubleshooting/.claude-plugin/plugin.json ```text { "name": "k8s-aks-troubleshooting", "version": "0.9.0", "description": "Use when a problem is specific to Azure Kubernetes Service (AKS): a cluster or node pool in \"Failed\" state, an upgrade or scale that fails (\"UpgradeFailed\", \"violate the pod's disruption budget\", \"QuotaExceeded\", \"AllocationFailed\"), a LoadBalancer Service stuck at (\"SyncLoadBalancerFailed\", \"AuthorizationFailed\"), an Azure Disk or Files PVC that fails (\"ProvisioningFailed\", \"volume node affinity conflict\", \"Multi-Attach error\", \"Mount error(13)\"), workload identity errors (\"AADSTS700016\", \"AADSTS70021\"), CoreDNS edits that AKS reverts, \"error dialing backend\", Azure Policy denials, or API server access (authorized IP ranges, private clusters, kubelogin, \"User does not have access to the resource in Azure\"). Investigates with kubectl and read-only az commands, and proposes fixes. Reads first and asks before changing anything.", "author": { "name": "k8s-servant", "url": "https://github.com/k8s-servant" }, "homepage": "https://github.com/k8s-servant/k8s-skills", "repository": "https://github.com/k8s-servant/k8s-skills", "license": "Apache-2.0", "skills": [ "./" ] } ``` ## File: README.md upstream: https://github.com/k8s-servant/k8s-skills/blob/main/README.md ````text # k8s-skills Kubernetes troubleshooting skills for any AI agent: read first, ask before changing anything. [![CI](https://github.com/k8s-servant/k8s-skills/actions/workflows/ci.yml/badge.svg)](https://github.com/k8s-servant/k8s-skills/actions/workflows/ci.yml) [![Release](https://img.shields.io/github/v/release/k8s-servant/k8s-skills)](https://github.com/k8s-servant/k8s-skills/releases) [![License](https://img.shields.io/github/license/k8s-servant/k8s-skills)](LICENSE) [![Skills](https://img.shields.io/badge/skills-30-326ce5)](#skills) [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/k8s-servant/k8s-skills/badge)](https://scorecard.dev/viewer/?uri=github.com/k8s-servant/k8s-skills)

An agent with the k8s-workload-troubleshooting skill finds that my-app is OOMKilled with a 32Mi limit, and proposes kubectl set resources with an undo line, asking before it runs it

A real run of the oom-killed scenario on a kind cluster, with a view-only kubeconfig. The commands and output are shortened, and the answer is the agent's own words. k8s-skills is a set of [Agent Skills](https://agentskills.io) that teach an AI agent to investigate Kubernetes problems the way an experienced SRE does: what to look at, in which order, what each finding means, and what to propose. The skills work in Claude Code, Codex, Gemini CLI, Antigravity, GitHub Copilot and any other agent that reads `SKILL.md` files and can run `kubectl`. How it works, how each skill is tested, and what breaking a kind cluster 76 ways showed about the docs: [k8s-skills: Teaching AI Agents to Troubleshoot Kubernetes Without Breaking It](https://sysadminas.eu/K8s-Skills-Read-First-Ask-Before-Changing/). ## Quick start ```bash npx skills add k8s-servant/k8s-skills -a claude-code # or -a codex, ... gemini extensions install https://github.com/k8s-servant/k8s-skills # Gemini CLI ``` Then describe the problem to your agent, for example *"my-app in my-namespace keeps restarting. Find out why."* For an agent that can only look, give it a read-only kubeconfig ([Your agent, your decision](#your-agent-your-decision)). More ways to install are under [Install](#install). ### Try it on a throwaway cluster first Not ready to point an agent at a real cluster? With Docker, [kind](https://kind.sigs.k8s.io) and kubectl, one command creates a local test cluster, breaks something on purpose and prints what to give your agent: ```bash git clone https://github.com/k8s-servant/k8s-skills.git && cd k8s-skills tests/try.sh # default: a pod that is OOMKilled; tests/try.sh --list for more ``` It prints a prompt and a view-only `KUBECONFIG`. Start your agent in an empty folder with that `KUBECONFIG`, ask the prompt, then compare the answer with the scenario's `expected.md`. The identity can only read, so the agent cannot change anything. When you are done, `tests/try.sh --down` deletes the cluster. The cluster is called `k8s-skills-test`, and the script never touches your other clusters or your current kubectl context. ## Why you can trust it - **30 skills, 92 scenarios.** Each scenario breaks something on purpose in a kind cluster. A fresh agent with only the skill and a view-only kubeconfig has to name the cause, point at the evidence and propose an acceptable fix, without changing anything. - **Quoted from a real cluster.** Every error message and log line a skill quotes was reproduced on kind, at a pinned version. Where the cluster said something different from the docs, the skill quotes the cluster. - **Checked on every change.** CI checks each command against its tier, requires an `# Undo:` line for every change, and rejects any command that reads Secrets. ## Read first, ask before changing Every command in a skill falls into one of three tiers. CI checks each one against the allow-lists in `tools/skillcheck/allowlist.yaml`. | Tier | What | What the agent does | |---|---|---| | Run freely | Read-only commands: `get`, `describe`, `logs`, `events`, `top` and a short list of others, plus read-only `az ... show`/`list` for the AKS skill | Runs them while it investigates | | Ask first | Reversible changes (`set image`, `set resources`, `rollout undo`, `scale`, `patch`, `cordon`, `uncordon` and a few more) and one read-only program run in a container with `exec` (`nslookup`, `curl`, `cat`, ...) | Shows the command, what it changes and how to undo it, under `> Ask first: the agent runs this only after the user approves it.`, and runs it only after you say yes | | Never run | Anything destructive or hard to undo (`delete`, `drain`, `replace`, `apply`, ...) | Proposes it under `> Proposed fix: review before running. The agent does not run this.`, with an `# Undo:` line, for you to review and run | Secrets are never read, in any tier. Every change, in either tier, comes with an `# Undo:` line. ## Skills | Skill | Use it when | |---|---| | `k8s-triage` | Something is wrong in the cluster and it is not clear where: a ranked first look at nodes, pods, events and resource use, with a first fix and the skill that goes deeper | | `k8s-workload-troubleshooting` | A pod or workload is unhealthy: Pending, CrashLoopBackOff, image pull errors, OOMKilled, failing probes, stuck rollouts, failing Jobs | | `k8s-loki-troubleshooting` | Loki rejects, drops or cannot query logs: pushes refused with 400, 401 or 429 (old timestamps, missing tenant, rate and stream limits), crashes on schema errors, storage flushes, rings, query limits | | `k8s-network-troubleshooting` | Traffic does not reach running pods: Services without endpoints, wrong selectors or ports, DNS, NetworkPolicies, Ingress and Gateway routes | | `k8s-coredns-troubleshooting` | Cluster DNS itself fails: CoreDNS crashing on Corefile errors or loops, reloads that fail, upstream timeouts and SERVFAIL, stub domains, rewrites and hosts, NodeLocal DNSCache, the AKS, EKS and GKE DNS ConfigMaps | | `k8s-storage-troubleshooting` | Storage blocks a workload: PVCs stuck Pending, missing StorageClasses or unsupported access modes, attach and mount errors, full volumes | | `k8s-autoscaling-troubleshooting` | Autoscaling does not scale: HPA targets `` (missing requests, no metrics), stuck at `maxReplicas`, metrics-server and `kubectl top`, KEDA ScaledObjects not Ready or without an HPA, the VPA | | `k8s-cert-manager-troubleshooting` | cert-manager does not issue or renew: Certificates not Ready, issuers not Ready, a wrong `issuerRef`, ACME challenges stuck, the cert-manager webhook unreachable | | `k8s-flux-troubleshooting` | Flux does not deploy: sources not Ready (missing branch, credentials, chart version), Kustomizations with a wrong path or failed build, HelmReleases stalled, suspended objects | | `k8s-argocd-troubleshooting` | Argo CD does not deploy: comparison errors (wrong path or revision), destinations not allowed by the project, no automated sync, failed syncs, drift, health | | `k8s-helm-troubleshooting` | Helm installs or upgrades fail or a release is stuck: `--wait` timeouts, ownership conflicts, `another operation is in progress`, hooks, immutable fields; helm commands are left to the user | | `k8s-kyverno-troubleshooting` | Kyverno denies or fails to police resources: Enforce denials, Audit-only policies and PolicyReports, ClusterPolicy and CEL ValidatingPolicy, mutate and generate rules, webhooks down | | `k8s-fluent-bit-troubleshooting` | Fluent Bit does not deliver logs: config errors that crash the DaemonSet, outputs that cannot connect or drop chunks, records without Kubernetes metadata, tail paths and CRI parsers, backpressure, the Fluent Operator | | `k8s-gatekeeper-troubleshooting` | OPA Gatekeeper denies or fails to police resources: constraint denials and actions, templates that do not compile, a webhook that fails open, exemptions, sync, mutation | | `k8s-external-secrets-troubleshooting` | External Secr ````