The Junior Climb hardcover mockup
Cluster Baseline You Will Defend

Cluster Baseline You Will Defend

The marketplace of AKS addons looks like progress. Enable all of them and you inherit five upgrade trains, three ways to ship logs, and an on-call graph nobody can draw. The cluster still boots. Day 2 is when you learn which checkboxes had owners.

A baseline is the set of choices you will still defend at 2 a.m. Everything else is opt-in with a named owner, a runbook, and a path to remove it when that owner leaves.

This is article two of Series 4. Article one put AKS in the landing zone. Here you decide what the platform module ships by default so Deployment Model does not become a debate over every preview flag in tfvars.

System pool for addons, user pools for apps, canary pool for soak before migrate.

Node pools you can explain

Microsoft documents system node pools for critical cluster components. App pods belong on user pools. That split is not fashion. It is how you keep CoreDNS, metrics, and policy agents alive when an app chart OOMs the neighborhood.

  • System pool: critical addons only, taints and labels so app Deployments stay off, sized for the control-plane adjacent load you actually run
  • User pools: per workload class (general compute, memory, GPU if you staff it, spot for nonprod when the app can tolerate eviction)
  • Autoscale with floors and ceilings you can pay for and that leave surge room for upgrades
  • Node OS and image upgrade strategy you actually run (article eight), not a wiki promise

Do not run app pods on the system pool because the cluster “was quiet.” Quiet ends the first time a noisy neighbor and a CoreDNS incident share a node. If your starter chart lacks a toleration that keeps apps off the system taint, fix the chart in the catalog. Do not remove the taint to make a Friday deploy green.

Pin VM sizes to what SKU policy already allows in the landing zone. AKS is not a back door for D-series that policy denied on VMs. Cost and policy both catch that if you let them. When Advisor recommends a size outside the allow-list, that is either noise or a signal to update the initiative with Series 2 module owners. It is not a reason to open a one-off exemption so one pool can chase a recommendation.

Availability zones should match the spoke promise. If the landing zone sells zonal resilience, node pools should not silently land in one zone because someone copied a three-node tutorial. Write the zonal layout into the module defaults for prod. Document which zones the region actually offers for your SKUs. An empty zone list in Terraform is how you discover imbalance during the first AZ outage drill.

Canary pools belong in the diagram even when you only stand them up for upgrades. Article eight covers the drain path. The baseline rule here is simpler: you must be able to add a pool without redesigning networking. If every new pool requires a firewall change, your hub story is too fragile for Day 2.

Outbound and API as baseline, not options

Outbound type has to match the hub story. In a landing zone with Azure Firewall inspection, that is usually bring-your-own VNet plus userDefinedRouting. Microsoft is explicit: with UDR, AKS does not configure egress for you. The route table on the node subnet must send 0.0.0.0/0 to the firewall or NVA. Create that before you pretend the cluster is “secure.”

Required Microsoft endpoints and your package registries belong in the hub firewall policy as versioned rules, not as a temporary * that survived the first successful az aks create. Microsoft publishes outbound FQDN and network rules for AKS. Treat that list as platform content next to the hub module. Article three covers the traffic paths. The baseline rule is simpler: the module fails the apply if firewall dependency outputs or private DNS outputs are missing, the same way a spoke module fails without peering.

Private API server DNS must be wired when you run a private cluster. Prefer disabling public FQDN so the control plane address is not disclosed on the public internet. Operator access then requires VPN, ExpressRoute, or a bastion path you already operate for the landing zone. Do not solve “I cannot kubectl from my café” by flipping the cluster public for a sprint that never ends.

Disable local accounts when Entra integration and Azure RBAC are solid. Keep a break-glass path that is PIM and audited, not a local admin password in a password manager titled aks-prod-temp. Built-in Policy can deny clusters that leave local authentication on. Assign it at the management group so portal snowflakes cannot skip it.

API authorized IP ranges on a public cluster are an exception you write down. They are not a substitute for private cluster when the spoke contract already promised private control planes. If you allow the exception, canary it, put an expiry on the ticket, and keep the authorized ranges in Terraform next to the cluster resource.

Addons: default on vs default off

Addons are product surface. Each one is an upgrade dependency and an incident domain. Microsoft’s baseline AKS architecture warns that Azure Policy constraints must allow managed addon pods or you will silently break ingress, secrets sync, and observability with no obvious Deploy error. That is another reason the baseline stays thin and the Policy exclusions for system namespaces stay deliberate.

Default on when you operate them:

  • Azure Policy add-on for AKS (Gatekeeper-backed constraints you will keep green)
  • Monitoring you sized: Container Insights / Azure Monitor agent path pointed at a workspace with a budget story
  • Workload identity and OIDC issuer (article four)
  • Secrets Store CSI driver if Key Vault CSI is your secrets pattern
  • Microsoft Entra integration settings that match identity

Default off until requested with an owner:

  • Every preview addon
  • Service mesh you cannot staff through upgrades
  • Multiple ingress controllers fighting for the same edge
  • GitOps controllers if you already ship with OIDC pipelines another way and nobody owns Flux or Argo day-2
  • Virtual nodes, event-driven addons, and anything whose only justification is a conference talk

If the addon has no runbook, it is not part of the baseline. “We might need it” is how the on-call graph grows without headcount. When a ticket asks for a mesh “like the reference architecture,” the answer is item seven on the one-pager below, not a Friday install into kube-system.

Track who owns each default-on addon the same way you track Owners on subscriptions. Container Insights without an Owner is how Series 5 finds a workspace that doubled overnight and nobody to call.

Version pins are the baseline

Kubernetes version sits at N-1 or N-2 of what you tested in nonprod. Track the AKS release notes, release tracker, and support calendar. Floating to the portal’s suggestion is not a baseline. Azure Advisor will nag as you approach end of support. Put those dates in the product README next to the soak rule so the upgrade PR is not a surprise six weeks before EOL.

Pin node image versions in Terraform the same way you pin module versions when you manage upgrades manually. Node image upgrades are more frequent than Kubernetes minors. Microsoft’s Day-2 guidance treats node OS channels such as NodeImage and SecurityPatch as first-class settings, often with planned maintenance windows. Article eight covers cadence. Here you refuse latest as a strategy and you refuse silent drift between twin prod clusters.

Addon versions move with the cluster. Thin baseline means fewer surprise breaks when Microsoft ships a control plane upgrade. Fat baseline means every upgrade is a negotiation with five product owners who thought their addon was free.

Auto-upgrade channels belong in the baseline only when you have planned maintenance windows and a soak story. Enabling stable everywhere without nonprod parity is how prod becomes the test ring. Prefer explicit pins plus a calendar until automation is real. If you do adopt channels, enforce them with Policy so one cluster does not quietly opt out.

SKU, SLA, and support story

Production clusters run on the tier that matches how you call Microsoft and how you page yourselves. Free tier for prod is a budget decision that becomes an availability incident with worse options. Match the support story to the spoke contract. App teams need to know whether platform is selling a regional cluster, a zonal layout, or a multi-cluster pattern you have not built yet. Uptime language belongs next to the module, not in a sales slide.

Disk encryption, host encryption options you standardize, and key management for secrets patterns should be choices in the module, not afterthoughts discovered during an audit. If Series 3 evidence packs ask for encryption posture, the answer should come from Terraform state and Policy compliance, not from a screenshot tour.

Capacity and quota are part of the baseline product too. An upgrade that needs surge nodes in a SKU your subscription cannot place fails before the first drain. Check regional SKU availability and quota headroom in the planning PR. Do not discover missing quota during the change window.

Module interface: expose decisions, hide landmines

Expose parameters operators actually need:

  • Node pool VM sizes, min/max counts, and labels/taints for known classes
  • Which optional addon flags are allowed in this environment
  • Tags (Environment, CostCenter, Owner, Application) and budget amount
  • Kubernetes version within the allowed band
  • Whether nonprod may use spot pools
  • Maintenance window inputs when auto-upgrade is part of the product

Do not expose a casual ability to flip private cluster off in prod tfvars, re-enable public FQDN, open the API to the Internet, turn local accounts back on, or disable Azure Policy without a change that policy itself will notice. Terraform is not a substitute for Deny.

Keep the module boring. Boring clusters upgrade. Exciting clusters collect screenshots for postmortems. When someone asks for “just one more tfvars knobs for egress,” ask which on-call owns the new failure mode. If the answer is silence, the knob stays closed.

What you write down next to the module

A one-page baseline doc beats a CAF appendix nobody reads. Include:

  1. Default-on addon list and the owner for each
  2. Node pool classes and who may request a new class
  3. Allowed Kubernetes versions and the soak rule (nonprod first)
  4. Outbound type and firewall rule set name
  5. Private DNS / public FQDN posture
  6. Break-glass group and PIM window
  7. Explicit out-of-scope list until the next product increment

When a ticket asks for Istio “like the reference architecture,” you point at item seven instead of inventing a mesh on Friday. Publish the one-pager in the same repo as the module. Wiki copies drift. PR reviews catch drift.

How you know the baseline is real

Count the exceptions. If every cluster has a different addon set, you do not have a baseline. You have eight one-offs with a shared logo. If upgrades stall because an abandoned addon has no owner, that addon was never baseline. Remove it.

Compare Terraform plans across prod clusters. Drift in pool counts is normal. Drift in private cluster flags, local accounts, or Policy add-on state is a product failure. Resource Graph queries that list clusters missing workload identity or still on free tier belong in the weekly platform pack, not in a hero spreadsheet after an audit finding.

Next: AKS networking through the hub. CNI Overlay, private API, egress through the firewall, ingress that matches Series 2, and private endpoints pods can actually resolve.

Randy Bordeaux

Azure Trainer | Veteran | Engineer

Helping IT professionals grow through cloud, leadership, and shared knowledge.


Discover more from Randy Bordeaux

Subscribe to get the latest posts sent to your email.