When the Kubernetes API Server won’t Answer: Debugging AKS Nodes with kubectl-aks

It’s 2 AM. PagerDuty fires. Your AKS cluster’s API server is unresponsive. kubectl get nodes hangs. You know something is wrong on the nodes, but your primary debugging tool — kubectl — is useless without a functioning control plane.

Sound familiar? This is the exact scenario that led us to build kubectl-aks.

The Problem

If you’ve operated AKS clusters long enough, you’ve hit that wall: the Kubernetes API server is misbehaving — maybe a certificate issue, maybe etcd is overwhelmed, maybe a misconfigured admission webhook is deadlocking requests. Whatever the cause, you’re locked out of your own cluster.

The Azure CLI (az) can tell you about your Azure resources, but it won’t tell you what’s happening inside your nodes. What’s the route table? Is DNS resolving? Is kubelet even running? Is the disk full?

You’re stuck SSH-ing into nodes one by one — if you even have SSH access configured.

Enter kubectl-aks

kubectl-aks is a kubectl plugin that lets you run commands directly on AKS nodes, even when the control plane is completely down. It works by leveraging the Azure VMSS RunCommand API — a side-channel that’s entirely independent of Kubernetes.

kubectl krew install aks

That’s it. You now have a new kubectl aks subcommand available.

How It Works

The plugin supports two runtimes:

• azure-api (default) — Control plane is down or unreachable
• kube-api — Control plane is healthy, you want a debug pod

The azure-api runtime talks directly to the Azure VMSS RunCommand API. No kube-apiserver involved. No kubeconfig needed (beyond Azure auth). It’s your escape hatch.

The kube-api runtime creates an ephemeral privileged pod with `nsenter` on the target node — useful for quick debugging when the cluster is healthy.

Real-World Workflow

Let’s walk through a realistic debugging session.

1. Import your cluster nodes

kubectl aks config import \
 --subscription $SUB_ID \
 --resource-group $RG \
 --cluster-name my-prod-cluster

Tip: Use --runtime kube-api to import the cluster from current kubeconfig context without the need to specify all parameters

This discovers all VMSS instances backing your cluster and stores them locally. Nodes are grouped by cluster, so you can manage multiple clusters easily.

2. Run a command

kubectl aks run-command "journalctl -u kubelet --lines 10 --no-pager"

No API server. No SSH. Just direct node access through the Azure fabric. The command runs across all imported nodes, giving you cluster-wide visibility instantly.

Tip: Use--node <node-name>to focus on a specific node if you already know which one is misbehaving.

What It’s Not

kubectl-aks doesn’t replace the Azure CLI. You still use azto create, scale, and delete clusters. Think of kubetl-aksas the tool that fills the gap between “my cluster exists” and “I can actually see what’s happening on the nodes.”

Getting Started

# Install via krew (recommended)
kubectl krew install aks
# Import your cluster
kubectl aks config import --subscription $SUB --resource-group $RG --cluster-name $CLUSTER
# Start debugging
kubectl aks run-command "dmesg | tail -20"

The project is open source: github.com/Azure/kubectl-aks

— -

In the next post, we’ll look at the built-in health checks that ship with kubectl-aks — ready-made diagnostics for DNS, connectivity, disk pressure, and more.