One customer's entire staff cannot sign in
Your task
It is 07:50. Your HR product lets business customers sign in through their own identity provider using SAML. In SAML, the provider signs a sign-in message, and your app checks that signature with a certificate it stored for that customer. About 240 customers use it. This alert has been firing since 07:36:
SsoFailureRate: 100% of SAML sign-ins failed for tenant northwind in the last 15 minutes (threshold 50%)
What you see:
- Northwind has 6,000 staff, and its payroll closes at noon today. The other 239 customers sign in normally.
- Each failed attempt logs
saml tenant=northwind issuer=ok audience=ok signature=invalid. - The last change to Northwind's sign-in settings was 14 months ago, and your last deploy was on Monday. The certificate you store for Northwind is valid until next March.
- Northwind's IT team opened a ticket at 07:44: "Our identity provider had planned maintenance from 07:00 to 07:30. Nothing should look different to users."
- When it joined, Northwind asked you to turn off password sign-in for its staff.
How do you get Northwind's staff signed in before noon, and what do you change so a customer does not get locked out like this again?
It covers
- Saml
- Single sign on
- Enterprise customers
About this question
One customer's entire staff cannot sign in
Model answer
Free on every accountFirst move: call Northwind's IT contact and get the certificate their identity provider signs with now. Their ticket reads like a notice, so tell them first that their staff are locked out. Ask whether the signing certificate changed during maintenance, and ask for the new one or their metadata URL, the address where an identity provider publishes its current settings.
Whatever certificate you store for Northwind is what your app trusts, and whoever holds its key can sign in as anyone at Northwind. So take it through a channel you can trust: their provider's metadata over HTTPS at a known address, or a fingerprint confirmed by phone with a named admin you already have on file. Do not load one from an email attachment, or copy it from the failing message, until that fingerprint matches. Any certificate in that message could have been put there by anyone.
Then add it beside the old one, if your app can hold 2. If it holds only 1, replace it, since the provider has stopped signing with the old one, and keep the old one to hand in case Northwind switches back. If your app caches tenant settings, clear the cache. Test one sign-in with their IT, which shows whether it took, then watch the failure rate fall. Northwind can also switch its provider back to the old certificate if the provider still has it, since yours is good until March.
What this does not fix: the next change of this kind, until you change the setup below. Set yourself a time, such as 10:00, to escalate through the account manager, so noon stays safe.
Keep password sign-in off. Northwind asked for it, and staff may have no passwords. Only someone at Northwind with authority can agree to an exception. Never turn off the signature check to get people in.
Read the signals. One customer of 240 fails, so your shared code and your deploy are fine. issuer=ok audience=ok means the message names the expected sender and is addressed to your app. Neither proves who sent it. Only the signature can, and that check fails. The certificate you store is in date, so expiry is not the reason. An in-date certificate still cannot check a signature made with a different key. The alert fired at 07:36. At steady traffic, a 50% rate over 15 minutes means failures began by about 07:28, inside Northwind's maintenance. Nothing changed on your side.
A forged message would also fail the signature check. But every Northwind user has been failing since their maintenance, and you confirm with Northwind before you trust anything.
A clock problem does not fit: it would fail the message's time conditions, not its signature. A changed signing algorithm, or a change in which element is signed, could also show signature=invalid. If the fingerprints match, the key did not change. Ask Northwind what else the maintenance changed, and ask them to undo it. This is one customer, so skip the status page. Northwind's IT and the account manager need the facts and a time.
Confirm. Take one failed SAML response from your logs or a browser's developer tools. A browser shows it in base64, so decode it first. Read its X509Certificate value, if it has one. That value is a base64 certificate with no PEM header, so decode it: echo "<value>" | base64 -d | openssl x509 -inform DER -noout -fingerprint -sha256. Run openssl x509 -noout -fingerprint -sha256 -in stored.pem on the one you hold and compare. If they differ, the key probably changed. If the message carries a certificate, the one Northwind gives you matches it. If it carries none, the test sign-in is your check. After you add it, the log line shows signature=ok.
Fix.
- Read each customer's metadata URL on a schedule, where they have one, and trust every signing certificate it lists. A provider may publish the new one before it switches, so you have it in time. With no overlap it still fails until your next read.
- Let each customer hold 2 certificates at once, so old and new can overlap.
- Log the fingerprint you saw beside the one you hold, and alert when a tenant's fingerprint changes, so the next diagnosis takes a minute.
- Tell customers at setup to warn you before changing this, and keep a named admin on file for each.
Follow-ups.
- Check how many of the other 239 customers have a metadata URL and a named contact.
- Agree an emergency route with large customers ahead of time, signed off by their security team.