Container Insights at default verbosity on a busy cluster will teach FinOps faster than any spreadsheet. The workspace fills. The bill spikes. Someone disables logging to “save money” and the next outage is blind. You already lived a version of that story when a cleanup deleted the sandbox workspace in the cost article.

AKS observability must respect the logging seams: what is platform signal vs app signal, who pays, and which tables on-call is allowed to read.

This is article seven of Series 4. Guardrails stopped the privileged pod. Observability is how you prove what the cluster did after admission.

Platform signal

Ship these to the platform workspace with retention you can defend:

• Cluster and node health, including NotReady and disk pressure patterns
• Control plane audit: who changed RBAC, what deployed at cluster scope
• Ingress and shared gateway metrics for the edge platform owns
• Policy denial and admission events you rely on in IR
• Upgrade-related Kubernetes events you will alert on (Microsoft points at KubeEvents from the upgrader path in Day-2 guidance)

Platform signal answers “is the product up” and “who changed the control plane.” It is not every stdout line from every container.

Diagnostic settings on the AKS resource itself belong in the platform module the same way firewall diagnostics did in Series 1. Do not wait for a remediation task that never runs.

App signal

Application logs, metrics, and traces belong to the workload cost center when volume is high. Prefer an app telemetry workspace or clearly attributed tables, following Series 1 cost rules. Do not force every pod’s console noise into the same workspace that holds hub firewall logs.

App teams own verbosity. Platform owns the collection plumbing and the deny-list for namespaces that should never ingest at debug forever.

OpenTelemetry or Application Insights from the app is still workload-owned. Container Insights does not replace application telemetry. It also does not mean you need both at maximum verbosity.

Collection choices that do not bankrupt you

Microsoft documents enabling monitoring for AKS with Log Analytics for container logs and related inventory tables, plus Azure Monitor managed service for Prometheus for metrics when you adopt that path. Use managed identity authentication for the agents (useAADAuth). Point at a workspace you meant to create, not DefaultWorkspace-... that appeared because someone clicked Enable in the portal without reading the subscription.

If the Log Analytics workspace lives in another subscription, register the required resource providers and use CLI or ARM as Microsoft documents. Portal shortcuts will fight you.

Size collection:

• Prefer ContainerLogV2 and resource-specific patterns where Microsoft steers you
• Drop or exclude noisy namespaces (dev scratch, load-test sandboxes)
• Sample or filter where you must; “all containers, all levels” is not a requirement
• Syslog from nodes is not the same decision as every container log
• Prometheus metrics need scrape budgets too. High cardinality labels will hurt as badly as verbose logs
• Keep agent and extension versions in the baseline so upgrades are planned

Run the cost query from Series 1 against the workspace that receives AKS data:

Usage
| where TimeGenerated > ago(24h)
| where IsBillable == true
| summarize IngestionGB = round(sum(Quantity) / 1000, 2) by DataType
| order by IngestionGB desc

If ContainerLog or ContainerLogV2 dominates and nobody can name an incident it solved last month, change collection before finance changes it for you.

Retention still matters. Thirty days hot because Terraform defaulted it is a billing accident. Ninety days hot plus archive is an IR conversation. Three hundred sixty-five days hot on a cluster that already archives is how the logging bill outruns the node pool bill.

Exclusion and sampling recipes that stay owned

Publish a short collection matrix next to the AKS product README:

• Platform namespaces: health, audit, and upgrade events always on
• Prod app namespaces: info and above, with Owner approval for debug windows
• Nonprod scratch namespaces: exclude or short retention by default
• Load-test namespaces: exclude from the platform workspace entirely unless a named test Owner funds a temporary sink

Debug windows need an end time in the ticket. Permanent debug on a chatty service is how Usage doubles and nobody remembers why. I still walk into workspaces where a single namespace accounts for half of ContainerLogV2 because someone left verbose logging on after a Friday incident and never filed the follow-up to turn it off.

Sampling is not failure. Sampling without documenting what you cannot prove afterward is failure. If IR requires full request logs for payment services, say so in the matrix and fund that table. Do not pretend every namespace has the same evidence needs.

Access boundaries

Table-level or workspace RBAC so app teams do not read hub firewall tables while debugging their pod. On-call needs cluster metrics and their namespace logs without subscription Owner.

Separate platform and app workspaces when volume or confidentiality demands it. One workspace per team is how you got deleted sandboxes. Prod vs nonprod splits are usually enough if tags and budgets are honest.

Publish which tables are platform-only. Make that part of the AKS product README so VIP escalations do not become “just give them Contributor on the workspace.”

Alerts that page

Page on:

• CrashLoopBackOff on production namespaces for customer-facing apps (after a short burn-in, not every restart)
• Node NotReady beyond a threshold
• Persistent volume failures that block prod
• Ingress 5xx on shared controllers
• Cluster autoscaler or upgrade failures that leave the control plane mid-flight
• Agent or monitoring extension unhealthy when you depend on it for IR

Do not page on every FailedScheduling in nonprod, every image pull backoff during a registry blip you already know about, or informational kube events that belong on a dashboard.

Microsoft’s upgrade guidance suggests custom log search alerts on KubeEvents for upgrader events. Put those in the platform action group, not buried in an app team’s unused rule.

Action groups should page humans who can fix the cluster product, not a distribution list that forwards to nowhere.

When the cluster goes blind

Treat monitoring agent unhealthy as a platform incident, not a curiosity. If Container Insights or the Prometheus collector stops, you lose the evidence path Series 3 assumes you have. Page the platform on-call. Do not wait for an app outage to discover the agent died last Tuesday.

Keep a minimal break-glass query path documented: Azure Activity for control plane changes, Kubernetes audit if still flowing through diagnostic settings, and node metrics from Azure Monitor even when container logs are delayed. Blind should be partial, not total, when you design seams deliberately.

Dashboards vs evidence

Dashboards are for humans watching. Evidence is for IR and audit. Keep saved queries for:

• Who bound cluster-admin this week
• Which namespaces ingested the most GB yesterday
• Node image and Kubernetes version inventory across clusters
• Policy non-compliance counts for the AKS initiative
• Workspace Usage by DataType for the AKS-related tables

The Kusto companion mindset still applies. Pretty tiles that nobody can re-run from a query are demos.

Azure Advisor recommendations on upgrades and retirements belong in the weekly platform review next to ingestion cost. Observability without an ops cadence is just storage.

Network and identity still show up in the logs

Missing private DNS links present as timeouts. Firewall denies present as egress failures. Workload identity failures present as 401s from Azure APIs. Your observability design should make those distinguishable. If every failure looks like “pod unhealthy,” you will restart pods forever.

Container Insights and Prometheus are not a substitute for firewall logs and DNS investigation habits from Series 1. Keep the seams: platform network evidence stays queryable when AKS is the patient.

Prometheus cardinality as a first-class budget

If you adopt Azure Monitor managed Prometheus, treat high cardinality labels like verbose container logs. pod_name and unbounded user IDs on every metric will melt scrape and storage the same way debug stdout melts Log Analytics.

Publish allowed label guidelines in the starter chart. Reject custom ServiceMonitor objects that explode series count without a named Owner and a cost estimate. Series 5 will dig into observability cost. Here the rule is simple: metrics need a budget the same way logs need a retention matrix.

Cost preview for Series 5

Series 5 goes deeper on observability cost. Here the platform product requirements are enough:

• Budget alert on the workspace resource ID tied to AKS ingestion
• Weekly review of Usage by DataType
• Named Owner for Container Insights configuration
• Documented split of platform vs app signal in the spoke contract / AKS product README
• Nonprod verbosity defaults that are quieter than prod debug culture

Turning monitoring off to save money is the same class of mistake as deleting the sandbox workspace. Fix categories, filters, and retention first.

Put the AKS workspace resource ID on the same weekly FinOps pack that already reviews orphans and Advisor. Cluster teams that never look at Usage will discover the bill when finance does. Platform product owners who glance at the top three DataTypes every week catch regressions while they are still a ticket, not a war room.

What success looks like for cluster signal

On-call can answer three questions in five minutes: is the node pool healthy, who changed cluster RBAC last, and which namespace is driving ingestion. App owners can query their own tables without subscription Owner. Policy denials and upgrade events are findable without opening a support case. The workspace budget has not been “temporarily” disabled since the last invoice surprise.

If those answers still require a hero who knows three undocumented workspaces, observability is storage with hope attached.

Next: AKS Day 2, control plane upgrades, node images, canary pools, capacity for surge, and break-glass that returns to git.

Randy Bordeaux

Azure Trainer | Veteran | Engineer

Helping IT professionals grow through cloud, leadership, and shared knowledge.