Rolling back a deploy without anyone keeping AdministratorAccess

Every organisation that moves to just-in-time access meets the same objection within a week: “some jobs really do need admin.” It is true. Rolling back a broken deploy in production, during an incident, may need permissions nobody should hold every day. The usual compromise is to leave a handful of senior engineers with AdministratorAccess “for emergencies”. Those accounts become the most valuable credentials in the company, and the ones least often reviewed.

This post walks through a high-risk request in the demo organisation, Northwind Labs: an engineer asking for admin on a production account to roll back an authentication service deploy. It shows how the request is narrowed, why it needs a second factor and two approvers, and what the broker actually creates in AWS.

The business case against standing admin

An account that holds AdministratorAccess all the time is a liability whether or not its owner is trustworthy. Laptops are stolen, sessions are hijacked, tokens leak into logs. When that happens to a standing administrator, the attacker inherits the whole account. When it happens to someone whose admin rights exist for thirty minutes a quarter, the attacker most likely inherits nothing.

There is also an audit and accountability cost. Auditors testing privileged access ask who holds it, who approved it and whether it was removed. A standing admin group answers the first question and fails the other two. And there is an operating cost: when the only route to admin is a person who already has it, incidents queue behind whoever is awake.

Step 1: Sam asks, and the scoper narrows

Sam is a platform engineer. During incident INC-4412 the team decides to roll back a recent deploy of the authentication service on the prod-core account, rated high and holding personal data. Sam types: “admin on prod-core to roll back the auth deploy for INC-4412, 30 min”. This is AR-0005.

The intake agent matches prod-core and the Administrator entitlement, whose ceiling is AdministratorAccess: *. The least-privilege scoper reads “roll back the auth deploy” and narrows inside that ceiling to what a rollback on Amazon ECS (Elastic Container Service) needs:

That is the difference between being able to do anything in the account and being able to look at and update the auth services. No IAM, no data stores, no other services. The scoper runs at L3: it applies the narrowing and records it on the timeline, where any approver can see the before and after. If the narrowed set turns out not to be enough, the answer is a new, specific request, not a wider one.

Step 2: the score and the flow

The risk agent scores the narrowed request 57, High:

Two things now decide the route. The score is above 55, so step-up is required. And the first flow that matches is #30, “Admin in production: two people”: approvers payments.lead@ and security@, a ticket required, step-up required, one hour maximum. The flow would have demanded two people even if the score had been lower, because admin in production is never approved automatically unless a flow explicitly allows it, and #30 does not. Sam is not on call for this team this week, so there is no on-call credit.

The route AR-0005 takes. The order of the two approvals does not matter; both are required.

Step 3: the second factor comes first

Before anyone is asked to approve, Sam must prove the request is really theirs. Sam’s page says: “Waiting for your second factor — approvers see this request once you confirm it is you.” Sam confirms with the authenticator app on their phone (a time-based one-time code, with replay protection) or a Duo push. The assertion lasts five minutes and can be used once.

The order is deliberate. If Sam’s browser session had been hijacked, the attacker could type the sentence but could not produce the second factor, and the approvers would never be woken for a request Sam did not make. Sign-in MFA from Northwind’s identity provider protects the login; step-up protects this specific, risky action.

Step 4: two approvers, neither of them Sam

Once Sam confirms, the request reaches both approvers, in the console and as approve/deny buttons in Slack. Each sees the same page: the sentence, the ticket, the score with its factors, the narrowed permission and the thirty-minute window. The payments lead is asked whether the rollback is the right operational call. Security is asked whether the access is proportionate. The approval count goes from 0/2 to 2/2.

The requester can never be one of the two, even if Sam were listed as an approver on the flow for some other reason. An approver can also deny with a note, which is recorded and shown to the requester. In the same incident, Sam’s request AR-0006 for operator access on prod-payments to change an export bucket was denied by security: “Exports are frozen during INC-4412 — ask again after it closes.” A denial with a reason is part of the record too.

Step 5: what the broker creates, and removes

With both approvals in, the control plane signs a grant instruction with Northwind’s organisation key. The broker, running in Northwind’s network, collects it over an outbound HTTPS connection and checks it: signature, nonce, freshness, addressee. Then it checks its own policy file, broker.yaml, which Northwind controls. Is admin allowed to be granted at all? Is thirty minutes within the longest grant it will create? Does the permission contain an action it will never grant? By default those are iam:*, organizations:* and sts:*; none appear here.

AR-0005 end to end. The granting credential, an IAM role the broker assumes, never leaves Northwind’s network.

For AWS IAM Identity Center the broker creates a permission set for this grant alone, named with an IK- prefix, containing the four ECS actions as an inline policy and a session duration no longer than the grant. It assigns that permission set to Sam’s Identity Center user on the prod-core account. Sam signs in through the AWS access portal as usual and sees one extra, short-lived option. At expiry the broker removes the assignment and deletes the permission set, from its own local ledger, whether or not the InsightKeys service is reachable at that moment.

What each reader takes from it

Keep reading