The cluster is private. Pods still cannot resolve privatelink.blob.core.windows.net. Or they can, but egress bypasses the firewall because the route table never made it onto the node subnet. Someone swears Kubernetes broke DNS. The landing zone seam broke DNS. Kubernetes only made the failure louder.

AKS networking is Series 1 networking with tighter failure modes. Same hub. Same private DNS. Same UDR to Azure Firewall. The difference is how fast a missing link shows up as CrashLoop, CreateContainerError, or a timeout that looks like a deny rule.

This is article three of Series 4. Article two locked the baseline. Here the traffic has to match the spoke contract: consume the hub, do not invent a second edge inside the spoke.

CNI choice is secondary to hub integration

Microsoft documents Azure CNI Overlay, Azure CNI with pod or node subnets, and Cilium-powered data planes. Overlay is usually the right default in a landing zone with a real IP budget: nodes take VNet addresses, pods take an overlay CIDR, and you stop burning large spoke prefixes on pods. Classic flat CNI is familiar and still valid when you need pods to be directly addressable on the VNet and you sized for it.

Neither choice forgives missing UDRs or DNS links. Overlay does not mean egress skips the firewall. Pods still leave through the node path you configured. If outboundType is userDefinedRouting, Microsoft expects a 0.0.0.0/0 to your firewall or NVA on the cluster subnet. AKS does not invent SNAT or a sneaky public IP for that path. Microsoft’s UDR guidance is blunt: you configure egress; AKS validates that the default route does not point straight at the Internet when you claimed UDR.

Pick Overlay when:

• Spoke address space is tight or shared across many teams
• You do not need every pod IP routable on the VNet
• Your network policy engine and Windows needs match what Overlay supports in your version band

Pick flat Azure CNI when:

• You have IP space and a reason pods must be VNet-native
• An existing pattern already depends on it and migration cost exceeds the savings

Do not migrate casually. Microsoft treats updating IPAM mode to Overlay as irreversible, with Kubernetes version and feature prerequisites (including uninstalling certain network policy engines first). Plan it like a platform change, not a parameter flip. Read the version gate before you schedule the change window. Surprises on Overlay migrations are expensive because you cannot casually undo them.

Cilium as a data plane is a separate decision from Overlay as IPAM. Microsoft documents them as independent operations. Adopt Cilium when you staff the operational model and want the policy and observability features you will actually use. Do not flip dataplanes because a blog said it was faster.

Subnets and sizing

Dedicated subnets for nodes. Size for growth plus upgrade surge. Running out of IPs mid-scale is a migration, not a tfvars tweak.

With Overlay, node subnet sizing is about nodes and surge, not pods-per-node times node count on the VNet. With flat CNI, do the pod math honestly or you will learn it during an upgrade when surge needs extra addresses. Microsoft’s Day-2 upgrade guidance calls out address pressure during surge as a common failure mode. If autoscaler max plus max surge exceeds subnet capacity, the upgrade fails while the control plane thinks it is progressing.

Place private endpoints for spoke PaaS in a PE subnet in the AKS spoke when that matches your Series 2 pattern. Do not drop PE NICs into the node subnet “to keep it simple.” Simple becomes an NSG argument later. PE subnets also keep lifecycle separate when you rebuild a node pool.

Service CIDR and DNS service IP must not overlap hub, spoke, on-prem, or the pod CIDR. Overlaps fail late and look like random connectivity. Put the chosen ranges in the module README next to the diagram. I still see teams reuse 10.0.0.0/16 for service CIDR because a tutorial used it, then wonder why on-prem routes behave oddly.

Egress through the hub

Node subnet UDR to Azure Firewall. Firewall network and application rules for required Microsoft endpoints, your package registries, and the FQDNs AKS needs to operate. Microsoft publishes required egress endpoints for AKS. Treat that list as platform firewall content, versioned next to the hub policy. When the list changes, update the rule collection in the same change process you use for any hub firewall edit. Do not keep a sticky note titled “AKS needs * temporarily.”

If you open * to make cluster create succeed and never tighten it, you built a very expensive NAT gateway that happens to schedule pods. Cost will notice. Security will notice later, which is worse.

Outbound types matter:

• loadBalancer: AKS-managed egress, fine for some nonprod, usually wrong when the landing zone promises inspection
• userAssignedNATGateway / managed NAT: better SNAT behavior, still not the same as hub firewall policy
• userDefinedRouting: landing zone default when Azure Firewall is the inspection point
• none / block: advanced cases with explicit design, not a weekend experiment

Match the spoke contract. Workloads do not get a second Internet path because a Helm chart needed Docker Hub on Friday. Mirror approved images into ACR and pull privately. If a vendor chart still needs a public pull, that is a catalog exception with an Owner and an expiry, not a permanent hole in the firewall.

SNAT pressure shows up as intermittent timeouts that look like app bugs. NAT gateway paths help when you chose that outbound type. UDR paths depend on the firewall’s SNAT capacity and rule design. Measure before you blame Kubernetes.

Ingress that matches Series 2

Prefer Application Gateway Ingress Controller or Front Door to private origins, aligned with Series 2 ingress patterns. Public LoadBalancer Services on every app in prod should hit Deny or at least a change review loud enough that people remember it.

Internal ingress for service-to-service where appropriate. Do not invent a second public edge inside the spoke because the shared gateway ticket had a two-day SLA. That second edge becomes the path nobody monitors and the certificate nobody renews.

TLS termination belongs where your platform product puts it: shared Application Gateway, Front Door, or a platform-owned ingress controller. App teams should not each mint a public IP and a certificate from a personal Key Vault. Certificate lifecycle is a platform incident domain when it sits on the shared edge. Keep it there on purpose.

When using AGIC or similar, remember Policy constraints must allow the managed pods. Baseline article already warned about Gatekeeper silently blocking addon pods. Network and policy are one product. If ingress pods cannot schedule, check Policy before you rebuild the gateway.

Private endpoints from pods

Same platform private DNS zones linked to the AKS VNet. Workload identity to PaaS beats keys in pods (article four). If DNS is wrong, you will see timeouts that look like firewall denies. Use the Kusto habits from the logging companion: prove whether the name resolved, whether the packet left, and whether the firewall saw it.

Link the zones the workloads actually use: blob, file, vault, azurecr, sql, redis, service bus, whatever your catalog supports. A missing zone link is the most common “AKS networking is broken” ticket I still see. The fix is a platform PR, not a CoreDNS ConfigMap edit that drifts on the next upgrade.

Private endpoint approval and DNS record creation stay on the platform pipeline or a tightly scoped role on the zone. Contributor on the hub DNS resource group for app teams already failed identity.

When pods resolve a public PaaS name and hairpin through the Internet instead of the PE, you usually have a missing zone link or a conditional forwarder gap. Fix the platform DNS story. Do not teach apps to hardcode privatelink FQDNs unless you deliberately own that pattern.

Network policy and NSGs

Pick Azure NPM, Calico, or Cilium-based policy based on what your CNI and staffing support. Enforce default-deny in namespaces that hold sensitive workloads once the starter charts include the allow rules they need. Audit first if you are not ready to break deploys.

NSGs still apply at the subnet. NetworkPolicy is not a replacement for hub inspection when traffic hairpins through the firewall. Use both with clear jobs: NSG for subnet edges, NetworkPolicy for east-west inside the cluster, firewall for egress and hub-bound paths.

Do not disable NetworkPolicy engines casually before an Overlay migration and forget to put a replacement back. That gap becomes permanent. Document the engine in the baseline one-pager so the next hire does not install a second one “temporarily.”

Private cluster DNS and jump paths

Private clusters put the API server on a private network. Operators need a path: VPN, bastion with network access, or a secure admin jump that is still audited. Do not solve operator access by flipping the cluster public “just for the sprint.”

Bring-your-own private DNS for the API is a platform decision. Wire it like every other privatelink zone: platform owns the zone, spokes get links, pipelines create records. App teams do not get Contributor on the hub DNS resource group to “fix kubectl.”

If you use API Server VNet Integration patterns, follow Microsoft’s naming and identity requirements for the private DNS zone. Changing or deleting those zones after create can break upgrades. Treat them as immovable platform assets. Put them in the same change control as ExpressRoute circuits.

az aks command invoke can help when you lack direct network path, but it is not a substitute for designing operator access. Use it as break-glass tooling, not as the day-to-day kubectl story for every engineer.

Prove the path before you blame the chart

When a pod cannot reach Storage or SQL, run the same order every time:

1. Resolve the name from inside the pod or a debug pod in the same namespace
2. Confirm the answer is a private IP in the PE range you expect
3. Confirm the node subnet UDR still points at the firewall
4. Confirm the firewall saw the flow (allow or deny)
5. Confirm Azure RBAC or firewall application rules, not only network rules

Skipping to “open * on the firewall” teaches the wrong lesson and burns the investigation trail. Series 3 IR still wants evidence. Networking tickets are where that discipline dies first if you let it.

Diagram debt

Publish one diagram and keep it next to the module: hub, firewall, AKS subnets, PE path, ingress path. The image at the top of this article is the minimum. When the module changes outbound type or ingress pattern, update the PNG in the same PR. Folklore diagrams are how the next hire routes around the firewall.

Next: workload identity on AKS. OIDC issuer, federation, per-app identities, and retiring Storage keys living in Kubernetes Secrets.

Randy Bordeaux

Azure Trainer | Veteran | Engineer

Helping IT professionals grow through cloud, leadership, and shared knowledge.