Break-glass · The privileged-access tier
Break-glass is what we built so the day a vendor has a bad day is not the day our customers cannot reach their mailbox. A WireGuard-gated tier with hardware-key SSH for the most-privileged operations, sitting alongside — never on top of — the routine identity layer.
Two tiers of reachability. No third-party SSO in the privileged path. A perimeter you can read in one sitting.
Surface signal
Status
LIVE
Tiers
Two
Recovery surface
recover.eleven11.pro
The question every modern stack ducks
Almost every team runs the operations that matter most — credential rotation, mailbox recovery, incident response, destructive admin — through the same identity vendor that runs their login page. When that vendor degrades, you don't just lose a tab. You lose the path to fix things. The blast radius of a third-party outage is the radius of your most-privileged actions.
Break-glass is the answer we ship for our own platform — and for clients who refuse to accept that the worst day of the year is somebody else's responsibility. A separate trust zone, with a separate transport, that doesn't share fate with the everyday-login surface.
Self-sustained by design
This is the most sovereign part of the fleet. No vendor whose outage takes you offline. No SSO provider whose pricing page you cannot afford to lose. The privileged path is ours, in our network, on our terms.
01
Separate trust zone
The privileged-access tier is its own network, its own listener, its own auth path. A phished routine session cannot reach a destructive route — the origin returns 403 unless the request arrived on the privileged tier. Stronger than step-up re-auth: the path simply does not exist on the routine tier.
02
No third-party SSO in the privileged path
Routine surfaces sit behind Cloudflare Access — a good fence for the everyday work. The privileged tier does not. WireGuard credentials and a hardware-resident SSH key are the only way through. A phished Gmail does not produce either.
03
Hardware key, not memory
The most-privileged operations require a physical touch on a hardware token (YubiKey or Nitrokey, ed25519-sk). A compromised laptop without the key in the slot cannot run the action. We do not type passwords for the privileged path.
04
Caddy is the load-bearing artifact
Header trust comes from the ingress shape, not from re-verifying tokens at every origin. One caddy config, audited line by line, decides which path can carry which headers. Smaller perimeter than a JWT verification stack — and legible in one sitting.
05
Hetzner Cloud Firewall in front
Phase 2.5 of the rollout, live since 2026-04-18. Inbound surface narrowed at the cloud edge before any container ever sees a packet. Firewalld inside as a third-tier gate. Three independent layers, every one auditable.
The primitive
Break-glass is built on a small, opinionated trio — a routine tier for the everyday, a privileged tier for the destructive, and a hardware-key escape hatch for the day everything else is on fire. They are deliberately not collapsible into one. The separation is the product.
ρ · Routine tier
<surface>.eleven11.pro
Cloudflare Tunnel + Access in front of every human-facing surface. Google OAuth or email OTP at the edge; origin issues an HMAC-signed session cookie after the first ingress. Short TTL on client surfaces (2h on the recovery path), longer on operator surfaces (11h). The everyday door.
π · Privileged tier
<surface>.<recovery-tier>
Hub-and-spoke WireGuard terminating on the eleven11 server. Internal-only TLD resolved by a dnsmasq helper bound to the WG interface. A separate caddy listener answers only on this interface and strips inbound CF headers — there is no path for a routine token to impersonate this tier.
ε · Escape hatch
ssh + ed25519-sk
The break-glass under break-glass. WireGuard plus a hardware-resident SSH key gets a senior operator to a shell when every HTTP path is broken. Audited; rate-limited; physical-touch on every authentication.
How it fits the fleet
Break-glass is not a product customers click on every day. It is the substrate the operator-facing surfaces sit on, and the discipline that decides which actions can run from where.
recover
Client mailbox recovery at recover.eleven11.pro. Tunnel-only — Caddy's public listener never serves this hostname. First surface to consume the vendored cf_access middleware; the proof point that the auth contract works end to end.
mail-admin
Operator surface for the mail control plane. Routine reads on the public tier; destructive ops (DELETE /domains, bulk mailbox actions, DKIM resync) gated behind the privileged tier and refuse to run from anywhere else.
operator
Admin plane for every Eleven11 tool. Settings mutation, credential rotation, bulk ops route through the privileged tier. The routine session can browse; only the privileged session can change.
tira
Harvester operator UI. Worker provisioning and destructive cleanup are privileged-only. Recipe browsing and run history live on the routine tier — read-mostly, safe even on a phished session.
alerts
When a privileged action runs, an alert fires. When a recovery tripwire trips, recovery contacts get an email out-of-band. The auth tier is not a silent layer — every state change carries an audit row and a signal.
dhara
Lab API stays WireGuard-only regardless of CF Access. The privileged tier is dhara's default; Cloudflare Access narrows the browser surface only. Defense in depth on the most sensitive scan paths.
Surfaces & contracts
Two hostnames a person opens; two endpoints the middleware exposes; one decorator the routes carry; one shell that comes out when nothing else does. The smallest contract that holds.
recover.eleven11.pro
Client mailbox recovery
Live since Phase 4b. Tunnel-only. CF Access TTL 2h. Verified Gmail → mailbox list → reset password → tripwire email to recovery contacts. No privileged ops on this surface — clients never need the recovery tier.
*.<recovery-tier>
Privileged-tier hostnames
Internal-only TLD on the recovery tier. WireGuard-resolved, dnsmasq-served, never on public DNS. Caddy answers each privileged vhost only on the WG-facing listener. The Host header is the tier resolver — it cannot be spoofed because the path does not exist on the public internet.
POST /auth/cf-ingress
Routine-tier bootstrap
Vendored cf_access middleware. Reads Cf-Access-Authenticated-User-Email, upserts the principal, issues a session row, returns the HMAC-signed cookie. ~80 lines per language. Identical contract across every consumer.
POST /auth/handshake
Tier-promotion exchange
Single-use, 30s-TTL ticket bridges a routine session to a privileged session. Issued on the routine tier, consumed on the privileged tier. The bearer token never travels with the cookie; the cookie never travels with the bearer.
@requires_privileged_network
Origin-side gate
Decorator on every destructive route. Returns 403 with X-Privileged-Network-Required: true if the request did not arrive over the privileged tier. The frontend catches the header and tells the operator exactly what to do — connect WG, retry on the recovery hostname.
ssh + ed25519-sk
Hardware-key escape
When the HTTP layer is unavailable, the path is WireGuard plus a hardware-resident SSH key. No password fallback. No remembered credential. Physical touch is the gate — and it cannot be replicated by a phisher who has only the laptop.
Senior engineering, visible
Five decisions visible in the caddy config, the firewall rules, the vendored middleware file, and the audit table — not adjectives, design choices.
Ingress is the perimeter, not the token
Caddy decides which path can carry which header. Cf-Access-* headers exist only on the CF Tunnel ingress; they are stripped on every other listener. Origins trust the header because the path is the proof — and the path is one config file, auditable in a sitting.
Vendored, not packaged
The cf_access middleware is one file per language, copied into each consumer via a sync script. No npm/PyPI pull-through in the auth path. Zero transitive deps. Recent supply-chain disclosures (Axios, OpenBSD crypto) are why — the auth perimeter stays one file deep.
Sessions are pointers, not claims
Tokens are opaque HMAC-signed pointers; all state (email, tier, TTL, revocation) lives in a sessions row. Revoking a compromised session is one UPDATE. No JWT parsing in the hot path; no JWKs cache; no header/payload separation to get wrong.
Defense in depth, in three layers
Hetzner Cloud Firewall narrows the inbound surface at the cloud edge. Firewalld on the host is a second gate. Caddy and the per-tier listeners are the third. Every layer is independent — and Phase 2.5's rollout caught a real firewalld-vs-Hetzner asymmetry that would have silently dropped WireGuard packets.
Audit row on every privileged action
Every destructive op writes a log row with came_via, principal, session_id, and CF identity nonce. One compromised session is one nonce — one-command revocation in CF and at the origin. The log is the rollback plan.
Who this is for
Break-glass earns its keep when the cost of a third-party outage cascading into your customers exceeds the cost of running a sovereign privileged tier of your own.
FAQ
Both. The substrate (WireGuard hub, internal TLD, two-tier caddy listeners, vendored cf_access middleware, hardware-key SSH) is what we operate for our own platform. We deploy the same shape on a per-tenant box for clients who want a sovereign privileged-access tier of their own. There is no SaaS multi-tenant version — that would defeat the point.
Live: the routine tier (Cloudflare Tunnel + Access on every operator surface), the privileged-tier WireGuard hub with its internal TLD and per-surface caddy listeners, the vendored cf_access middleware in production on the recovery surface, Hetzner Cloud Firewall on the eleven11 box, and the hardware-key SSH escape hatch. Roadmap: full mail-admin cutover to the new auth shape, and porting the operator plane and tira onto the same middleware contract.
Cloudflare Access is the routine tier. We use it. The point is not to replace it — the point is to refuse to share fate with it on the actions that matter most. A vendor outage on the routine tier is a paged incident. A vendor outage on the privileged tier is a customer-facing one. Two tiers, two providers' worth of failure independence.
The vendored cf_access middleware is around 80 lines per language. It does four things: resolve the tier from the Host header, look up the session row, return a Principal, and gate routes via a decorator. No JWT parsing, no JWKs cache, no network calls in the hot path beyond the local Postgres lookup. You can read it in one sitting and audit it in two.
Discuss break-glass
Break-glass is what we run for our own platform and what we deploy on a per-tenant box for clients who need the same posture. Talk to us about the threat model, the rollout shape, or how to retrofit it onto an existing stack.
Direct line
Consultation requests stay owned. We reply from e11 after reviewing fit and timing.
