The Junior Climb hardcover mockup
Guardrails on the Cluster

Guardrails on the Cluster

Azure Policy on the subscription was green. The cluster still allowed privileged pods from latest tags pulling public images. Governance stopped at the resource ID. Inside the API server it was open season with nicer YAML.

Guardrails for AKS are Azure Policy for Kubernetes plus admission controls you are willing to keep green. Policy already taught canary discipline. The cluster is another assignment scope with sharper failure modes when Deny hits a Deploy.

This is article six of Series 4. Article five split who deploys what. Here you decide what those deploys are allowed to ship.

Azure Policy for AKS is not optional in a landing zone

Deploy the Azure Policy add-on for AKS. Microsoft builds it on Gatekeeper so constraints evaluate at admission. Built-in definitions cover patterns you already care about at the Azure resource layer and inside the cluster: private clusters enabled, workload identity enabled, Policy add-on installed, local authentication disabled, allowed registries, privileged container controls, host networking, and more.

Use DeployIfNotExists where Microsoft provides it to install the add-on at scale, then Audit/Deny for the behaviors you care about. Do not rely on engineers remembering a portal checkbox.

Assign initiatives that match your baseline:

  • No privileged containers in prod namespaces (with a documented exemption path for the rare node agent that truly needs it)
  • Allowed registry hosts pointing at your ACR, not the public internet by default
  • Required labels such as Owner and Application so cost and on-call still work
  • Block host networking and host PID where you do not need them
  • Cluster-level audits for private cluster, workload identity, local accounts disabled, and auto-upgrade settings you standardized

Microsoft also ships pod security baseline and restricted initiatives for Linux workloads. Start with baseline in Audit. Move sensitive namespaces toward restricted when starter charts can comply.

Start in Audit. Export compliance. Fix starter charts and namespace vending. Flip to Deny when the happy path complies. Same canary discipline as landing zone policy: one cluster first, then the management group.

Do not assign fifty constraints on day one. Assign the ones that stop the incidents you have already had. Add more when the first set stays green.

System namespaces and addon carve-outs

Microsoft’s baseline AKS architecture warns that Gatekeeper policies must allow managed addon pods. If you Deny broadly without excluding system namespaces correctly, ingress, secrets sync, and monitoring fail quietly. That looks like “AKS is broken” when it is Policy biting the platform.

Maintain an explicit exclusion list for namespaces Policy should skip or loosen: kube-system, gatekeeper-system, and other platform-owned namespaces you document. App namespaces do not get those exclusions by renaming themselves cleverly. Watch for teams creating kube-system-app folklore.

Pod Security Standards

Use Pod Security Admission at namespace level. Restricted for sensitive prod namespaces. Baseline for build or tooling namespaces that cannot meet restricted yet. Privileged should be rare, labeled, and owned.

Vendor charts that demand privileged are not a reason to leave the whole cluster open. Fork the chart, isolate the workload into a namespace with a tighter blast radius, or refuse the product until the chart grows up. Series 2 blast radius applies to bad charts too.

PSA labels on the namespace should be set by the vending pipeline, not by hope. Drift here is how a “temporary” privileged namespace survives three fiscal years.

Image provenance and supply chain minimums

Prefer digests over floating tags. Builds promote @sha256:... into prod values. latest in prod is how you cannot answer “what shipped” during an IR.

At minimum:

  • Only allow pulls from your ACR (Policy constraint)
  • Quarantine or scan patterns you actually operate, do not claim a scanner you ignore
  • Signed images when your pipeline and admission stack support verification you will keep green
  • Mutating webhooks that inject sidecars are platform-owned, versioned, and monitored

Public image pulls “just for a sidecar” become permanent. Put approved sidecars in your ACR and reference them from the starter chart.

Deployment safeguards from Microsoft’s Day-2 guidance can enforce probes, limits, PDBs, and related practices at deploy time. Use them where they fit your policy stack. Guarding deploys is cheaper than explaining why surge upgrades stalled.

Starter chart as the compliance engine

Policy Deny without a compliant starter chart is how you earn a reputation for blocking the business. Fix the golden path first. The starter should already set resource requests and limits, probes, PDBs, required labels, non-root where possible, and registry references that match the allow-list.

When Audit shows non-compliance, publish a single remediation PR against the starter, not thirty one-off chart edits in panic. App teams rebasing onto the fixed starter is how Deny becomes boring. App teams inventing their own privileged overrides is how Deny becomes political.

I still walk into estates where Policy is Deny and the starter still ships latest from Docker Hub. That is not governance. That is a fight you scheduled on purpose.

Resource hygiene as a guardrail

Resource requests and limits belong in the starter chart. Namespaces without quotas invite one team to eat the node pool. Quotas are a guardrail with teeth that show up before the OOMKill pager.

Limit ranges catch containers that ship with no requests. Combine quotas, limit ranges, and Policy so the first bad chart fails in nonprod, not during a Black Friday scale event.

What you do not invent on Friday

Random OPA Gatekeeper constraints copied from a blog without owners. Three competing tools that all deny differently: Azure Policy for AKS, a hand-rolled Gatekeeper install, and a service mesh authorization story nobody mapped. Pick a stack. Document it next to the baseline. Exemptions get an expiry and an Owner, same as Series 3.

Kyverno, Gatekeeper standalone, and Azure Policy can coexist in theory. In practice, two sources of Deny without a clear precedence will burn a week of “why won’t my pod schedule” tickets. Platform picks one primary admission story for app teams.

Exemptions are product features

Every Deny needs an escape hatch that is not “disable the policy.” Use Azure Policy exemptions with expiration, or a dedicated namespace with documented risk acceptance for vendor exceptions. Publish how to request one. Unpublished exemptions become Slack approvals. Slack approvals become permanent.

Review exemptions monthly. If the same chart needs privileged every quarter, fix the chart or isolate the workload. Do not grow a permanent privileged ghetto without saying so in the spoke contract.

Evidence

Policy compliance for AKS gatekeeping belongs in the same continuous evidence pack as management group assignments. Resource Graph and policy compliance views should show the cluster add-on healthy and the initiative non-compliant counts trending down after canaries.

Kubectl-only screenshots do not satisfy an auditor who asked whether privileged pods are blocked. Show the assignment, the effect (Audit then Deny), and a failed admission from a canary namespace when you flipped the mode.

Logging should receive policy denial events you care about. If Deny fires and nobody can query it, you will turn Deny back to Audit the first time a VIP deploy fails.

Canary the Deny flip

Pick one nonprod cluster. Assign the initiative in Audit for two weeks. Publish the non-compliant resources to the app teams that own those namespaces. Fix the starter chart once, not chart-by-chart in panic. Flip that cluster to Deny. Watch a deliberate bad Deploy fail admission. Then roll the assignment outward.

Skipping canary is how you earn a reputation for “Policy broke prod” and lose the mandate to use Deny at all.

Record the canary results in the platform repo: date, cluster, initiative version, known false positives, exemptions created. The next cluster flip should copy a playbook, not invent courage.

Registry allow-lists in practice

An allow-list that only says *.azurecr.io is better than the public internet and still weak if every team has a public ACR with anonymous pull. Prefer exact registry hosts you operate. Pair with ACR network rules and private endpoints from article three so the Policy win is not undone by a public registry that anyone can push to.

When a vendor insists on pulling from Docker Hub, mirror the digest into your ACR and reference the mirror. The exception should be time-boxed if mirroring is not ready, not eternal.

HostPath, privileged, and node theft

Privileged pods and HostPath mounts are how app workloads steal node identity and host network. Policy should deny those in app namespaces by default. Platform agents that need them live in system namespaces with documented exceptions. If a Helm chart needs HostPath to “read a config file,” fix the chart.

Combine Azure Policy constraints with Pod Security restricted where you can. Defense in depth is boring. Boring is the point.

Measuring guardrail health

Track monthly: Policy add-on healthy clusters, initiative non-compliant count in prod namespaces, exemption count with and without expiry, admission denials that were true positives versus starter bugs, and privileged namespace count.

If exemptions rise while non-compliance stays flat, you are negotiating governance away one Slack thread at a time. If denials are mostly starter bugs, invest in the golden path. If denials are mostly intentional bad deploys, Deny is doing its job and your education loop needs the error message links from the denial events.

What success looks like

A new namespace from the vending pipeline already has PSA labels, quotas, and Policy coverage. A developer who deploys nginx:latest from Docker Hub gets a clear admission error pointing at the allow-list. Exemptions have owners and end dates. Compliance dashboards show AKS initiatives next to the management group initiatives from Series 1.

Next: observability without melting the Log Analytics workspace. Container Insights, platform vs app signal, and alerts that deserve a page.

Randy Bordeaux

Azure Trainer | Veteran | Engineer

Helping IT professionals grow through cloud, leadership, and shared knowledge.


Discover more from Randy Bordeaux

Subscribe to get the latest posts sent to your email.