Sentinel — Benchmark Results¶
Fresh benchmark run: 2026-08-18, three full harness runs on main @ e8d8916. This
document replaces the Phase 14 baseline entry (2026-08-15) as the current reference record.
Numbers are median-of-runs across three full harness executions; the run-to-run spread is
disclosed in §7 so the noise floor of this measurement environment is known.
Reported as-is per vision §12: single-machine Docker-Compose loopback topology, disclosed, not implied scale. No thresholds are asserted anywhere in this document — these numbers exist to be regression-compared against later runs.
1 · What was measured (cell legend)¶
The harness (benchmarks/benchmark.py, stdlib + in-repo deps only) measures nine cells at two
concurrencies (1 and 8 concurrent clients), three repetitions per cell, 100 ops per batch:
| Cell | What it measures | Redis |
|---|---|---|
| B1 | Unguarded FastAPI handler — the pure HTTP baseline, no Sentinel | live |
| B2 | Full HTTP journey with Sentinel, token bucket policy | live |
| B3 | Full HTTP journey with Sentinel, sliding window policy | live |
| B4 | RateLimiter.evaluate only, token bucket (no HTTP/ASGI layer) |
live |
| B5 | RateLimiter.evaluate only, sliding window (no HTTP/ASGI layer) |
live |
| B6 | Bare EVALSHA of the token-bucket Lua script — the Redis round-trip floor |
live |
| B7 | Breaker-OPEN short-circuit — the fail path without Redis (pre-tripped) | none |
| B8 | Dead-port fail-open — the full failure journey (breaker + emergency limiter) | dead |
| B9 | Dead-port fail-closed — the full failure journey (503 denials) | dead |
Policy parameters follow invariant #6 (time made irrelevant): token bucket
capacity_micro = 2**30, refill_rate_micro_per_sec = 0, fresh uuid4 bucket key per run —
so every op is ALLOWED and over_limit == 0 is asserted for every cell. Sliding window
limit = 1000, window_size_micro = 60_000_000; fallback rate 1_000_000 µtokens/s; fresh
JWT sub per batch. Live cells hit the real SentinelGuard/RateLimiter/breaker/emergency
against real Redis (ASGITransport in-process); the dead cells use a dedicated client at
localhost:6399 with the production 20 ms fail-fast socket budget (B8/B9 have no warm-up — a
warm-up would trip the breaker and erase the measured failure journey).
Op counts per cell (×3 reps ×3 runs): HTTP cells 2,000, limiter cells 5,000, Redis floor 10,000, failure cells 500.
2 · Environment (recorded by the harness)¶
| Key | Value |
|---|---|
| git commit | e8d8916cbf77a69450b3f17ca83aa8be0101814d (main, post-v1.0.1) |
| Platform | Windows-11-10.0.26200-SP0 |
| Python | 3.13.7 |
| CPU | Intel64 Family 6 Model 186 Stepping 2, GenuineIntel (16 logical cores) |
| Redis | 7.4.9 (Docker Compose, localhost:6380/0, noeviction, bounded maxmemory) |
| Timestamps | 2026-08-18T10:31:46 / 10:34:19 / 10:36:10 +0530 |
| Method | benchmarks/benchmark.py --redis-url redis://localhost:6380/0, 3 reps/cell/run, 3 runs; median-of-runs below |
3 · Results — throughput and latency (median of 3 runs)¶
| Cell | c=1 ops/s | c=1 p50/p95/p99 (µs) | c=8 ops/s | c=8 p50/p95/p99 (µs) |
|---|---|---|---|---|
| B1 unguarded HTTP | 5746.5 | 155 / 273 / 426 | 2167.6 | 440 / 669 / 1148 |
| B2 guarded, token bucket | 950.2 | 1036 / 1456 / 1708 | 818.3 | 9556 / 12204 / 14716 |
| B3 guarded, sliding window | 723.0 | 1278 / 2223 / 2936 | 822.8 | 9722 / 12240 / 13611 |
| B4 limiter, token bucket | 1520.2 | 654 / 1030 / 2691 | 3370.1 | 2186 / 3502 / 4311 |
| B5 limiter, sliding window | 1337.9 | 615 / 1312 / 2854 | 3385.8 | 2112 / 3554 / 4073 |
| B6 EVALSHA floor | 1721.3 | 518 / 930 / 1400 | 3688.4 | 1898 / 3228 / 3775 |
| B7 breaker OPEN | 40717.9 | 21 / 24 / 43 | 76481.8 | 11 / 14 / 20 |
| B8 dead-port fail-open | 2969.7 | 21 / 37 / 29570 | 8146.4 | 20 / 40 / 29383 |
| B9 dead-port fail-closed | 3092.5 | 9 / 20 / 29440 | 7536.5 | 16 / 29 / 28518 |
4 · Results — CPU utilization (percent, median of runs)¶
| Cell | c=1 API / Redis | c=8 API / Redis |
|---|---|---|
| B1 | 99.9 / 0.4 | 96.3 / 0.6 |
| B2 | 64.8 / 8.1 | 92.5 / 8.3 |
| B3 | 62.9 / 7.6 | 95.5 / 7.9 |
| B4 | 49.6 / 10.0 | 93.7 / 17.2 |
| B5 | 45.6 / 11.0 | 96.8 / 16.7 |
| B6 | 47.9 / 13.2 | 94.4 / 17.7 |
| B7 | 116.3* / 0.0 | 0.0* / 0.0 |
| B8 | 9.3 / 0.0 | 76.4 / 0.0 |
| B9 | 9.1 / 0.0 | 94.2 / 0.0 |
* time.process_time() on Windows is quantized to ~15.6 ms scheduler ticks; in short
measurement windows per-cell API CPU is unreliable (B7 c=1 shows a >100% tick artifact, B7 c=8
a 0.0%). Treat all API-CPU cells as order-of-magnitude only.
5 · Results — decision-reason counts (summed over 3 runs; 4,500 ops per failure cell)¶
| Cell | counts |
|---|---|
| B7 | emergency_local_limit 4500 / 4500 |
| B8 | emergency_local_limit 4491 / 4491, redis_timeout 9 / 9 |
| B9 | circuit_open 4455 / 4392, fail_closed 45 / 108 |
over_limit (rate-limit denials) = 0 for all 18 cells, all runs — every decision in the live
cells was ALLOWED and every dead-port denial was a failure decision, not a quota denial.
Reading the B8 counts: the 9 redis_timeout entries across 4,500 ops = exactly 3 per
run = one per rep — the emergency limiter's initial burst (1 s of fallback rate) on each rep,
then the breaker trips and every subsequent op short-circuits (emergency_local_limit). This
is the post-fix semantics (see §9): pre-fix, denied calls banked refills and produced phantom
allows (5+3 per run).
6 · What the numbers mean (plain-English walkthrough)¶
- The with-Sentinel overhead at c=1 is ~6× throughput (5,746 → 950 ops/s), ~+880 µs p50 (155 → 1,036 µs). Where does the time go? B6 shows a bare Redis round trip costs ~518 µs p50; B2 − B6 ≈ 520 µs is the JWT verify + decide + HTTP plumbing; B6 − B1 ≈ 360 µs is the Lua round trip itself vs a bare handler. A remote Redis adds its own network RTT to every live row — these numbers are loopback.
- The limiter without the HTTP layer (B4/B5) is ~1.5–2× cheaper than the full journey (B2/B3) — for non-FastAPI consumers, that is the relevant number.
- The breaker short-circuit is nearly free: B7 p50 is 4–21 µs across runs (~32k–135k ops/s), median 21 µs. Fail-open protection does not tax healthy traffic; the failure path pays for the emergency decision, not for the breaker.
- Failure latency ≈ the socket timeout, not the limiter: B8/B9 p99 ≈ 27–33 ms on the dead port across all runs. The 20 ms production socket budget plus Windows connect overhead dominates; fail-closed (B9) pays the same and returns 503, fail-open (B8) absorbs it into the emergency decision. This is the designed failure journey, and it has not moved since Phase 14.
- c=8 throughput improves over c=1 for the limiter/floor/failure cells (B4–B9), but p50 rises (e.g. B4 654 → 2,186 µs). This is the documented in-process serialization artifact of 8 concurrent in-flight ops over one asyncio loop + loopback Redis; a real multi-worker deployment spreads this across processes.
- The guarded HTTP cells at c=8 (B2/B3) are the noisiest numbers in the table (p50 spread 5–13 ms across runs, see §7) — they are still ~10× faster than the failure path, which is the comparison that matters operationally.
- Redis CPU tracks live-cell load only (0.4–17.7%); the failure cells never touch Redis (0.0%) — the breaker really does short-circuit before any network call.
7 · Run-to-run variance (the noise floor of this machine)¶
This document reports medians of three consecutive harness runs because single runs on this Windows/Docker loopback setup swing materially under background load (the repo lives under OneDrive sync; desktop apps churn CPU intermittently). The measured spread:
| Cell | ops/s (min→max) | p50 µs (min→max) | p99 µs (min→max) |
|---|---|---|---|
| B1 c=1 | 5723 → 5845 | 151 → 156 | 425 → 428 |
| B1 c=8 | 1125 → 3916 | 224 → 590 | 716 → 4168 |
| B2 c=1 | 322 → 1019 | 915 → 2674 | 1586 → 8710 |
| B2 c=8 | 691 → 1159 | 6816 → 10662 | 10638 → 22477 |
| B6 c=1 | 1052 → 2375 | 373 → 834 | 1276 → 2681 |
| B7 c=1 | 32428 → 135223 | 4 → 21 | 27 → 301 |
| B8 c=1 | 2863 → 3257 | 4 → 22 | 26770 → 29636 |
| B9 c=1 | 2935 → 3229 | 3 → 15 | 27025 → 30317 |
Reading the spread:
- B1 c=1 is rock-stable (155 µs ± 3) — the single-flight path is deterministic; the instability appears exactly where the machine's background load lands (concurrency, longer windows).
- B7's p50 swings 4→21 µs — both are "tens of microseconds"; the honest statement is "the breaker short-circuit is a single-digit-to-low-tens-of-µs decision."
- The failure-path p99 (B8/B9) is the most stable number in the whole table (27–33 ms across 6 cells × 3 runs) — it is dominated by a socket timeout, which is immune to CPU load.
- Raw per-run JSONs are archived in
benchmarks/results/(20260818-*.json, gitignored) for anyone who wants the per-rep detail behind the medians.
8 · Comparison against the Phase 14 baseline (2026-08-15)¶
| Cell | Phase 14 | Now (median) | Verdict |
|---|---|---|---|
| B1 c=1 | 5946 ops/s, 150 µs p50 | 5746, 155 µs | unchanged (≈ −3%) |
| B2 c=1 | 1141 ops/s, 827 µs p50 | 950, 1036 µs | within this machine's noise band |
| B6 c=1 | 2188 ops/s, 440 µs p50 | 1721, 518 µs | within noise band (373–834 µs) |
| B7 c=1 | 96204 ops/s, 7 µs p50 | 40718, 21 µs | same magnitude; noisy cell |
| B8 c=1 p99 | 27243 µs | 29570 µs | unchanged (socket-timeout bound) |
| B9 c=1 p99 | 26520 µs | 29440 µs | unchanged (socket-timeout bound) |
| B8 counts | 5 circuit_open + 3 redis_timeout + 1492 emergency | 3 redis_timeout/run, rest emergency | post-fix semantics confirmed |
No regression is visible anywhere: every live cell sits inside the noise band this machine demonstrably produces (compare the B6/B7 spreads in §7 with the Phase 14 deltas above — the deltas are smaller than the spread). The failure path and the decision counts are unchanged or better.
9 · Historical record: the emergency-limiter defect found by Phase 14¶
The original Phase 14 run (2026-08-15) surfaced a production defect in the fail-open
emergency path: TokenBucketEmergencyLimiter persisted bucket state on every call while the
Lua contract is "denied requests never write", so each denied call banked its partial refill
and the next evaluation refilled the same window again — admitting up to ~2.3× the
configured fallback_rate_per_process_micro under sustained Redis failure (measured: 7 allows
in 3 s at 1 token/s vs the Lua's 3).
Fixed and verified: sentinel/emergency.py now persists state only on ALLOW (mirroring the
Lua's no-write-on-deny); regression tests cover sustained-rate exactness and the full-journey
fail-open path; a post-fix benchmark re-run showed no throughput regression and exactly
capacity+elapsed allows. The B8 counts in §5 are the live confirmation: exactly 1 initial
burst per rep (3 redis_timeout allows per run), no phantom allows. Full defect write-up:
the original finding, root cause, and fix status are recorded in the project record §09.
10 · Reproduce¶
docker compose up -d # Redis with noeviction + maxmemory
python benchmarks/benchmark.py --redis-url redis://localhost:6380/0
python benchmarks/benchmark.py --smoke # CI sanity subset
The harness asserts over_limit == 0 and the noeviction policy at startup, records the full
environment block, and writes per-run JSON to benchmarks/results/<timestamp>-<sha>.json.
Raw data: benchmarks/results/20260818-103146-*.json, 20260818-103419-*.json,
20260818-103610-*.json.