Book a call

Security

Make a provider breach a drill: rotate keys, swap certificates, move hosts

Amol Patil, Founder, CEO · 22 October 2026 · 7 min read

Explore with AI
ChatGPTPerplexityClaudeGoogleGrok
  • In one week of October 2026, eMudhra's intermediate keys turned out to be guessable and Railway detected anomalous activity in the systems that hold customer secrets. Customers did the cleanup in both.
  • Rotation stalls when a secret can only change through a deploy, and Railway paused deploys while it investigated.
  • Keep secrets in one manager that services read at runtime, run old and new keys side by side, and keep a second certificate authority ready.
  • Rehearse a rotation, a certificate swap and a host move on a schedule, and time each one.

We think a breach at a provider you depend on should cost you a drill, not a crisis. Two incidents a few days apart in October 2026 showed the crisis version, and in both the work landed on the customers.

What happened at eMudhra

On 17 October 2026 a researcher reported that the private keys behind eight eMudhra intermediate certificates could be recovered from public data. Anyone holding those keys could sign a certificate that browsers would trust for any website. Mozilla and Google both describe the report.

eMudhra says the keys came from an April 2025 key ceremony, where a faulty wrap and unwrap step created a second, weak set of keys, and the weak set is the one that was published. Help Net Security reports the same cause. The keys were close to guessable. The Register reported that one of them was the number 4.

The browsers moved within hours. Chrome blocked the certificates the same day, Firefox did the same that evening, and eMudhra revoked them on 18 October. Chrome is also removing the eMudhra root that signed them. Google told every site with a certificate under that root to install one from another certificate authority before the removal reaches Chrome on 27 October. eMudhra says the certificates affected by the root removal sit within one customer ecosystem in the US.

eMudhra says it is not aware of any evidence that the weak keys were used before the disclosure. Google says no certificate from these intermediates appeared in Certificate Transparency logs before the block. Browsers update themselves, though. Cloudflare warns that containers, API clients and devices keep trusting whatever certificate store they shipped with until someone updates it.

What happened at Railway

On 21 October 2026 Railway disclosed anomalous activity in the systems that store customer environment variables. An unauthorized party may have read encrypted variables and, for some of them, the decrypted values. Those values can include API keys, database passwords and registry credentials. Railway says the impact may be limited to a subset of users and asked every customer to roll every variable and review their logs.

Railway paused all deployments while it investigated. It links the attack to a series aimed at platform companies with the goal of compromising AI inference providers, and says it asked those providers to revoke keys that may have been exposed. GitGuardian advises treating every secret stored there as exposed.

A rotated key helps only once the running service uses it. With deployments paused, customers could rotate a key at the upstream service but could not roll out a build to pick it up, as OfficeChai noted.

Rotation should not need a deploy

Neither incident was the customers' doing, and both handed customers a job with a deadline they did not set. Our view is that each of those jobs should be routine, built in before anyone needs it.

Start with an inventory of every secret: what it unlocks, who owns it, where it is stored and how to replace it. Rotation can only cover the secrets on that list.

Then hold secrets in one secrets manager and have services read them at runtime or on a refresh. A new value then reaches production without a build, so a platform's deploy freeze does not hold up the rotation.

Prefer credentials that expire on their own. With OpenID Connect, a CI job trades its identity for a short-lived cloud token, so there is no long-lived key to steal. Where a long-lived key is unavoidable, rotate with two keys live. The alternating users strategy in AWS Secrets Manager keeps a valid credential available at every point of a rotation, and the same pattern works for any API that accepts two keys at once.

  1. Inventory: every secret, owner and scope. Leads to Issue a new key.
  2. Issue a new key: in the secrets manager. Leads to Old and new both live.
  3. Old and new both live: the service accepts either. Leads to Services switch.
  4. Services switch: read at runtime, no build. Leads to Revoke the old key.
  5. Revoke the old key: then check the access logs. Leads to Inventory (rehearse regularly).
A rotation like this keeps working while the hosting platform's deploys are paused.

Give every service and environment its own narrowly scoped key, so one leaked key exposes one service. Put spend limits and usage alerts on AI provider keys, since Railway says the attackers were after AI inference providers. Stop secrets before they reach a repository, too. GitHub's push protection blocks pushes that contain secrets.

Certificates you can swap

Certificates need the same preparation. Automate issuance with ACME, the standard protocol for automated certificate issuance, so that changing authority is a configuration change.

Publish CAA records that already allow a second authority, so a switch does not wait on DNS. Watch the Certificate Transparency logs for certificates issued for your domains.

Update the certificate stores in your containers and devices along with the browsers. Never answer a certificate error by turning verification off, a step Cloudflare warns against.

Production you can move

The third job is moving production to another host. Define infrastructure as code, ship containers any host can run, and write down a replacement for every provider feature on the critical path. Keep DNS under your own control.

Restore a recent backup onto a second host on a schedule, since only a restore shows that a backup works. Run production and staging in separate accounts so a leaked staging key opens nothing in production, and keep audit logs with alerts on unusual access.

The same logic applies to model vendors, which is why we argue you should keep your own test set and be able to switch.

Run it as a drill

The drill

  1. Rotate one secret

    Pick one at random. Issue a new value, run both, switch the services, revoke the old one, and record the time.

  2. Swap one certificate

    Issue a certificate for a test domain from your second authority and serve it.

  3. Move staging

    Rebuild staging on a second host from code and restore a recent backup into it.

  4. Fix the slowest step

    Write down each step that needed a person, and fix the slowest one before the next drill.

Put the drill in the same incident runbook that covers your other failures. Then start with the inventory: list every secret and certificate, who owns it and how long it takes to replace. The longest entry on that list is the first thing to fix.

Sources

The eMudhra incident:

The Railway incident:

Practices:

Amol Patil
Founder, CEONearly twenty years turning new technology into products, including eight at Globant as Technical Director for its mobile, wearables and IoT studios. Has taken vision, voice and agentic systems into production across many industries.

Working on something in this territory?

Tell us what you are trying to win. We answer within one business day, from the people who build.

Book a callgrowth@extendfuture.com