Instant for whoever is on call, loud for everyone else, reviewed in the morning

At three in the morning the settlement batch has stopped, merchants in Europe are waiting for their money, and the engineer on call needs to change something in production. The resource owner is asleep. Security is asleep. Any access process that needs one of them awake has just become the reason the outage lasts longer. This is why so many organisations keep a “break-glass” administrator account with a password in a sealed envelope or a vault, and why that account so often becomes the most powerful and least watched credential in the estate.

This post describes how break-glass works in InsightKeys, using the demo organisation, Northwind Labs: instant access for whoever is on call, after a second factor, with security told straight away and a review in the morning. It also explains how on-call status and open incidents lower the risk score for ordinary requests, so that many night-time requests never need break-glass at all.

The business problem: speed against standing risk

Every minute of a payments outage has a cost the business can count: delayed settlements, support calls, contractual penalties, reputation. The pressure that creates is reasonable, and the traditional answer is reasonable too: keep an administrator account that anyone on call can use without asking. The problem is everything that account does on the other 364 nights.

Standing break-glass accounts tend to share four properties. They have the widest possible permissions, because nobody knows in advance what an emergency will need. They are shared, so use cannot be attributed to a person without extra work. They are rarely used, so nobody notices when they are. And they are rarely reviewed, because reviewing them is awkward. Northwind’s discovery found one: breakglass-admin with AdministratorAccess on prod-core, last used 200 days ago, scored Critical 90 on the standing-access list.

First line: on call and incidents lower the score

Most night-time requests do not need break-glass at all. The risk score knows two things about the context of a request. If the requester is on call for the team that owns the resource, it takes 10 points off. If the request references an open incident, it takes another 10 off. Both come from the On-call & incidents page.

Those credits are what made Priya’s incident requests at Northwind fast. The request to read the invoices table during INC-4412 (AR-0008) scored 30 and was approved automatically; so did the request to read the payments-api logs (AR-0001). The read-only request on the eks-prod/payments Kubernetes namespace (AR-0007) scored 14. In each case the score fell because Priya was the right person, at the right time, with a reason the system could check, and the permission was narrow. None needed anyone to wake up.

The credits are deliberately limited. They do not override the rule that admin in production is never approved automatically unless a flow allows it, and they do not lift a write request on a critical database below the step-up line on their own. That is where break-glass comes in.

Flow #5: break-glass for whoever is on call

Northwind’s flow #5 reads, in the console: “Instant, after a step-up; security is told and the grant is reviewed after.” Who: whoever is on call. What: production. Longest: one hour. Step-up MFA and a reason are required. Because flows are evaluated in priority order, lowest number first, #5 is checked before the ordinary read, write and admin flows.

The important word is “whoever”. The flow does not name people. It names a condition: being on call now. When Priya’s shift ends on Sunday evening, Priya stops matching it, and the next person on the rota starts. Nobody has to add or remove anyone from a break-glass group.

How a night-time request finds its route at Northwind. Most requests from the person on call stop at the second box.

The 3 a.m. walk-through

Take an illustrative case, not one of the screenshots. Priya is on call for payments. INC-4412 has escalated overnight and Priya needs to change something on prod-payments to restart the settlement workers. Priya asks in Slack, with one sentence and the incident number. An illustrative score for a narrow write on that account at that hour might look like this:

Forty-eight is too high for an ordinary automatic approval and, through flow #20, would wait for the owner. Because Priya is on call, flow #5 matches first. Priya confirms with an authenticator app or a Duo push; the assertion lasts five minutes and can be used once. The grant is made immediately and the broker inside Northwind’s network creates it, still within its own local policy: the longest grant it will create, actions it will never grant, and whether admin may be granted at all. Break-glass changes who has to approve. It does not change what the broker is willing to do.

Break-glass at Northwind. Nobody is woken to approve, but security is told at once.

Session watch keeps working during the hour. If an action falls outside the grant or the reason, it is flagged exactly as on any other grant, and at Northwind’s default L2 setting the agent asks security to approve a revoke. Needing longer means asking again; there is no extension, only a new request, scored with what is true at that moment.

Two design choices keep break-glass from becoming the new standing admin. First, it is tied to a condition that changes on its own: the rota. Nobody is “a break-glass user”; people are on call or they are not. Second, the grant is short and singular. One hour, one reason, one person, one record. If the incident needs three people, three people ask, and the morning review has three lines rather than one shared account with no names on it.

The morning review

Break-glass grants are reviewed afterwards. The reviewer, typically someone in security, opens the grant and sees the full story in one place: the sentence Priya typed, the incident, the step-up, the permission, the session events with anything flagged first, and the broker instructions that created and removed it. The audit trail records the same events in words.

The questions for the review are short. Was the person on call? Was there an incident? Did what they did match what they said? Was the access removed? If all four answers are yes, the review takes a minute. If not, it is a conversation, and the evidence is already gathered. Over a quarter, the request records show how often break-glass was used. A rising count is a sign that ordinary flows are too tight, or that something in production is unstable; either is worth knowing.

Keep reading