The Junior Climb hardcover mockup
Secrets and Certificates at Scale

Secrets and Certificates at Scale

The pipeline still has a client secret that expires in nine days. Nobody owns the rotation calendar. Last time it expired, someone generated a new secret in the portal and pasted it into three variable groups. That is not a secrets strategy. That is how breaches become paperwork.

At landing-zone scale, the default is managed identity and federation. Shared keys are exceptions with owners and expiry.

This is article four of Series 3. Continuous evidence needs something true to say about encryption keys and application secrets. Identity already pushed humans onto PIM. This article covers machines.

Identity-first secrets: managed identity and OIDC to Key Vault, with shared keys as dated exceptions.

Managed identity first

Azure resources talking to Azure resources should use system-assigned or user-assigned managed identities and data-plane RBAC. App Service to Key Vault. Functions to Storage. Azure Kubernetes Service workload identity to ACR when Series 4 lands. SQL Azure AD authentication instead of SQL passwords where the engine supports it.

If your design needs a shared secret between two Microsoft Entra-aware services, question the design before you question the Key Vault SKU. Secrets between cloud services are often a leftover from a lift-and-shift mental model.

User-assigned identities help when several resources should share one principal, or when identity lifecycle must outlive a single App Service slot swap. System-assigned is fine when the resource and the identity should die together. Pick intentionally in the module. Do not leave both configured “just in case.”

OIDC for pipelines

Platform and workload pipelines authenticate to Azure with federated credentials on an app registration or user-assigned managed identity. Long-lived service principal passwords in variable groups are debt with an expiry date you will miss.

Rules that keep pipelines from becoming a second break-glass:

  • Separate identities for platform vs workload (Deployment Model)
  • Federated subject scoped to the repo, branch, and environment you intend
  • No client secret “backup” that someone created for local testing and left active
  • Inventory federated credentials quarterly; alert on create and update

Humans still use PIM. Machines use federation. If a human is copying a secret from Key Vault into a pipeline variable to “just run it,” your SDLC has a hole article six will call by name.

Key Vault patterns that survive the second team

Series 2 already warned against shared subscriptions pretending to be shared services. Secrets follow the same blast-radius rules.

One vault per app per environment for application secrets and app-managed certificates. The workload identity gets Secrets User (or narrower). Platform operators do not get standing Secrets Officer on every app vault.

One platform vault (or a small set) for platform secrets only: state storage keys if you still have any, shared connection material the hub modules need, certificates for platform ingress. Workload teams do not get Contributor on that vault’s resource group.

RBAC permission model for new estates. Access policies are legacy for greenfield. Mixing both models on the same vault is how nobody can explain who can read a secret.

Soft delete and purge protection on production vaults. Enforce with Azure Policy Deny in prod management groups. Soft delete without purge protection still lets a privileged principal finish the job.

Private endpoint into the spoke or shared services VNet, with public network access disabled when you can defend the operational path. Hub and spoke DNS from networking has to resolve privatelink.vaultcore.azure.net or the private endpoint is a decoration.

Diagnostic settings to the platform workspace. Key Vault logs are how you detect secret set/delete from unexpected principals. Logging already said not to wait on a remediation task for critical platform resources. Same rule for the platform vault.

Rotation without a spreadsheet hero

Prefer platform-managed rotation where the service supports it: Key Vault keys with rotation policies, certificates with lifetime actions, storage account customer-managed keys wired so the vault key version can advance without a Friday scramble.

For secrets that cannot auto-rotate:

  1. Name an Owner on the secret metadata or in the vault’s tagging standard
  2. Set an expiry on the secret object
  3. Alert before expiry (Event Grid on Key Vault events, or a scheduled query on secret attributes)
  4. Document the consumer list so rotation is a checklist, not an archaeology dig

A rotation calendar in OneNote is not a control. An expired secret that pages the owner is a control.

Certificates

Store certificates in Key Vault, or integrate a real PKI that issues into Key Vault. Storing a .pfx in git “temporarily” is forever. Git history does not forget.

Patterns that work in landing zones:

  • App Gateway or Application Gateway (and Front Door where applicable) with Key Vault references for TLS
  • Platform-managed certs for shared ingress; workload-managed certs for app-specific hostnames
  • Lifetime actions that send events before expiry
  • Deny policies against weak key sizes if your compliance baseline requires them

Portal uploads on Friday afternoon bypass every PR gate you bragged about in the architecture review. Put certificate changes through the same pipeline that owns the gateway.

Customer-managed keys without theater

CMK for storage, SQL, or disk encryption is a real control when:

  • Keys live in a vault with purge protection
  • Key permissions are least privilege for the platform-managed identity that wraps/unwraps
  • Rotation is tested in nonprod
  • You accept the operational blast radius (lose the key, lose the data)

CMK is theater when the same standing Owner who can delete the storage account can also purge the key, and nobody monitors either. MCSB and CIS both care about key management. Your evidence pack should show RBAC on the key, not only a checkbox that CMK is “enabled.”

What Policy should Deny (and when)

Good Deny candidates after the module is fixed:

  • Key Vaults in prod without purge protection
  • Key Vaults using access-policy model when your standard is RBAC-only (phase carefully)
  • Public network access enabled on vaults in prod MG when private endpoint is mandatory
  • Storage accounts with shared key access enabled when your standard requires Entra-only (often Audit first, then Deny)
  • Diagnostic settings missing on vaults in platform scope (DINE may fit better than Deny)

Do not Deny what you cannot remediate in a week. Audit, fix the module, remediate stragglers, then Deny. Policy already taught that order. Secrets are where teams skip it because “security wants Deny now” and break every pipeline that still uses a legacy pattern.

Detection that belongs in the same workspace

Alert on:

  • SecretSet / SecretDelete / VaultPatch from principals outside the expected pipeline and break-glass set
  • App registration password or certificate credentials added (Entra audit)
  • Key Vault access policy changes if any vault still uses that model
  • Purge operations and failed authentication spikes

Defender for Key Vault adds threat detection when you enabled that plan on purpose in article two. It does not replace RBAC hygiene or private endpoints. It tells you something weird happened after the posture work failed or an identity was abused.

Wire alerts into the same action groups that page platform on-call for hub issues. A Key Vault purge at 2 a.m. is a platform incident, not a “workload owner optional email.”

Anti-patterns that keep returning

  • One “enterprise” vault with every team’s secrets and standing Officer for a shared ops group
  • Connection strings in App Settings when Key Vault references are available
  • SAS tokens with far-future expiry pasted into wikis
  • Terraform azurerm_key_vault_secret values committed in tfvars
  • Personal access tokens for Azure DevOps or GitHub used as the cloud identity “until OIDC is ready”

Each of these shows up in audit season as a story. Stories are weaker than RBAC assignments and expiry dates.

Definition of done for a spoke

Before you call a reference spoke secure on secrets:

  1. No client secrets required for Azure-to-Azure calls in the happy path
  2. Pipeline auth is federated
  3. App vault exists with RBAC, purge protection, private endpoint (per standard), diagnostics on
  4. Policy Deny (or tracked Audit with due date) matches the standard
  5. Rotation owners exist for any remaining static secrets
  6. ARG query for public vaults / missing purge protection is in the evidence pack

Series 2’s shared-services article drew the subscription lines. This article fills those lines with identities that do not share passwords.

OIDC subject design that auditors can follow

Write the federation rules down:

  • Platform repo subjects deploy platform identities only
  • Workload repo subjects deploy workload subscription identities only
  • Environment protection rules for prod (GitHub Environments or Azure DevOps environments with required reviewers)
  • No wildcard subject that matches every branch on a fork

Inventory federated credentials with ARG or Microsoft Graph exports in the evidence pack. Auditors ask how pipelines authenticate. “We use OIDC” without subject examples is a slogan. Subject examples plus role assignments are evidence.

When a vendor still demands a client secret for a non-Azure system, store it in Key Vault with expiry and Owner. Do not put it in the same app registration that federates to Azure for Terraform. Mixing federation and long-lived passwords on one registration confuses rotation and confuses incident response.

Managed HSM and when you actually need it

Most landing zones start with Key Vault Premium or standard vaults with managed HSM only where regulatory language forces FIPS 140-3 level 3 style separation. Do not buy Managed HSM because a slide said “enterprise keys.” Buy it when the control map names it and operations can staff rotation, private connectivity, and break-glass for keys you cannot afford to lose.

CMK with a software-protected key in a purge-protected vault already beats unchecked Microsoft-managed keys with standing Owner everywhere. Escalate to Managed HSM when the framework and Risk say so in writing.

App settings, Key Vault references, and the Friday paste

App Service and Functions can reference Key Vault secrets without copying values into configuration at deploy time. Use that. Slot settings and sticky configs still need a design so swap does not point at the wrong vault.

When a vendor demands a static connection string, wrap it:

  • Store in Key Vault
  • Reference from the app
  • Restrict who can read the secret
  • Put expiry on the secret and a rotation owner
  • Prefer vendor support for Entra auth when it exists; reopen the ticket with the vendor instead of accepting forever-SAS as architecture

The Friday paste into three variable groups happens when the module does not offer a Key Vault reference pattern and the pipeline docs still show plain text examples. Fix the example. Engineers copy examples under pressure.

Inventory before you rotate everything

You cannot rotate what you cannot name. Run a one-time inventory:

  • App registrations with password credentials and their end dates
  • Federated credentials already in place (good) vs missing (debt)
  • Key Vaults lacking purge protection in prod
  • Storage accounts with key access enabled in prod
  • Pipeline variable groups that still hold clientSecret style values

Put the inventory output in the evidence repo. Burn it down with owners and dates. Defender recommendations will echo parts of this list. The inventory is how you assign humans. Recommendations alone do not assign humans.

I still find estates that “moved to managed identity” for the demo app and left thirty production SP passwords untouched. The demo is not the control.

Next: Network Posture Beyond the Firewall Slide. NSG/ASG, WAF, Private Link, and what Deny actually proves.

Randy Bordeaux

Azure Trainer | Veteran | Engineer

Helping IT professionals grow through cloud, leadership, and shared knowledge.


Discover more from Randy Bordeaux

Subscribe to get the latest posts sent to your email.

Discover more from Randy Bordeaux

Subscribe now to keep reading and get access to the full archive.

Continue reading