# Store Integration — production-readiness and performance plan

**Date:** 2026-10-03. **Builds on:** [`integration-architecture-review.md`](integration-architecture-review.md)
(direction: harden the existing Logistic Store API; no new platform layer).
**Status:** proposal. Nothing here is implemented. **Nothing has been load-tested** — every target in
§3 is a number to *prove*, not a measurement.

Evidence labels: **[V]** verified in this repo / this environment today, **[R]** read from code or a
prior report but not exercised, **[T]** target to be proven by test.

---

## 0. Summary

1. **One defect would cause a production outage on day 31: store API tokens expire after 30 days.**
   **[V]** (probe in §1, V1). It affects every store token — including the self-serve "Generate token"
   button shipped this week. Fix before any store goes live.
2. **Real volume is small.** kapitano_web produced **1,298 orders since 1 Sep ≈ 39/day ≈ 200 webhooks/day**
   **[V]**. "High performance" here means **low, predictable latency, no head-of-line blocking, and
   large headroom** — not horizontal scale. We design for **~50 orders/min sustained and a 10× burst**
   on one app node + one worker pool + Redis + MySQL, and explicitly refuse heavier machinery (§12).
3. **The remaining risks are operational, not architectural:** shared queue, Redis as a hard
   dependency, a few missing indexes, no request-id/metrics/error tracking, no load or resilience tests,
   no restore drill.
4. This document turns each into a **gated work item (§13)** and a **go-live checklist (§11)**.

## 1. Verified findings (what must change, and why)

| ID | Sev | Finding | Evidence | Fix |
|---|---|---|---|---|
| **V1** | **P0** | **Machine tokens die after 30 days.** `sanctum.expiration = 43200` min is applied to *every* token and overrides `expires_at`. | Probe through the real `store` guard: a token created 1 d and 29 d ago → accepted; **31 d and 200 d → REJECTED**. An expired token also looks like any bad token (generic 401, no `token_expired` key). A first probe through the HTTP route was misleading because the signature check answers first. | Exempt store-client tokens from the global age rule with `Sanctum::$accessTokenAuthenticationCallback` (store tokens: valid unless their own `expires_at` has passed; everything else unchanged). Give machine tokens an explicit `expires_at` (default 365 d, `INTEGRATION_TOKEN_TTL_DAYS`), show it in the portal, notify the owner at 30 and 7 days, add a distinct `token_expired` error key. Regression test with a 200-day-old token. |
| V2 | P1 | **Queue, cache and session default to `database`** (`.env.example`), and the API rate limiter uses the default cache store, so every request does cache I/O against the DB or a file. | `config/queue.php` default `database`; `.env.example` `QUEUE/CACHE/SESSION = database`; `RateLimiterProvider` uses `Limit::perMinute` on the default store. | Production: **Redis for queue, cache, rate limiter and sessions** (Redis is already mandatory for the signature nonce). Keep `after_commit` true on the Redis queue connection as defence in depth. |
| V3 | P1 | **Webhooks share the `default` queue** with captain push and order notifications; the runbook runs a single `queue:work`; each send may hold a worker up to the 15 s timeout. One dead store endpoint can delay every other store's updates. | `SendStoreWebhook` has no `onQueue`; runbook "Production checklist"; `webhook_timeout_s = 15`. | Dedicated `webhooks` queue, ≥ 2 workers supervised (Supervisor/systemd), per-store concurrency cap (e.g. 2 in flight) and a **circuit breaker**: after N consecutive failures a store's deliveries are parked until the next probe succeeds, instead of burning 15 s each. |
| V4 | P1 | **Root cause of the "first retry after 60 s" defect found on 21 Sep.** The job reads `$backoff[$delivery->attempts]` *after* the attempt count was incremented, so `backoff[0] = 10` is never used. | `SendStoreWebhook::handle` + `StoreWebhookService::backoff`. | Index by `attempts - 1`; add **jitter ±20 %** (an outage ending releases every store's retries at once); test the exact schedule. |
| V5 | P2 | **Sanctum writes `last_used_at` on every authenticated request** (one UPDATE per call). | `vendor/laravel/sanctum/src/Guard.php:167,171`; `config('sanctum.last_used_at', true)`. | Measure first. If it shows up, throttle the write to once per minute per token (keeps the portal's "last used" accurate). |
| V6 | P1 | **Missing indexes for planned and existing queries.** `orders` has `(store_client_id, created_at)` but not `(store_client_id, updated_at, id)` (needed by `updated_since`); `webhook_deliveries` has no `(store_client_id, created_at)` / `(store_client_id, status, created_at)` for the portal log. | Migrations `…090100`, `…090400`. | Add the three composite indexes in additive migrations (online `ALGORITHM=INPLACE`). `EXPLAIN` the list/lookup/portal queries on a copy with ≥ 1 M synthetic rows. |
| V7 | P2 | **Intake serialises on one row** — `order_number_sequences` is locked per `(prefix, period)` to stop duplicate numbers. | Migration `…090600` header; `OrderRepository` notes the lock. | Fine at today's volume (tens of ms held). Measure at 50 orders/min; if lock wait appears, allocate numbers in blocks. Do not change before measuring. |
| V8 | P1 | **No request-id, file logs only, no metrics / APM / error tracker.** | No `X-Request-Id` anywhere; `LOG_STACK=single`; no Sentry/Pulse/Telescope in `composer.json`. | Request-id middleware; JSON logs with `request_id`, `store_client_id`, `order_uuid`; error tracking (Sentry or equivalent); the metrics in §6. |
| V9 | P1 | **Redis is a hard dependency that fails closed.** Without it every signed call returns 503 `replay_check_unavailable`. Redis is **not running on this machine right now**. | `SignatureNonceStore` header; nothing on :6379. | Keep fail-closed (it is the correct security choice). Make Redis highly available with persistence (managed Redis or Sentinel), alert on latency/availability, and add Redis to the readiness check. |
| V10 | P2 | **Webhook URL guard cannot see DNS changes.** The guard checks the URL as typed; a hostname can later resolve to a private address (DNS rebinding). | `WebhookUrlGuard` doc comment: "cannot know where a *name* resolves". | At send time, resolve the host, reject private/loopback results, and connect to the resolved IP (pinned) with the original `Host`/SNI. |
| V11 | P2 | **No edge rate limiting** in `deploy/nginx.conf` (only app-level limits that depend on the cache). | No `limit_req` directive. | `limit_req` on the integration path and a stricter one on `/api/store-portal/auth/*`. |
| V12 | — | **After-commit dispatch is already correct.** `OrderCreated`, `OrderChanged`, `OrderStatusChanged` implement `ShouldDispatchAfterCommit`. | `app/Events/Order/*`. | Keep. Add a test that a webhook job is never enqueued for a rolled-back transaction. |
| V13 | P1 | **Same-second signed reads collide** (`signature_replayed`). | 21 Sep run, finding 1. | `v2` signature over method + path + body, accepted alongside `v1`. |
| V14 | P2 | **PHP-FPM sized for ~20 concurrent requests, 256 MB each**; OPcache/JIT settings are not in the repo. | `deploy/php-fpm-kapitano.conf`. | Confirm OPcache (`validate_timestamps=0` in prod, `preload` optional); keep `max_children` ≥ peak concurrency × safety (see §4). |

## 2. Targets proposed for the SLOs

All **[T]** — adopt, then prove with the §9 tests.

| Area | Target |
|---|---|
| Availability (store API) | 99.9 % monthly, excluding announced maintenance |
| `POST /orders` latency (server side) | p50 < 150 ms, **p95 < 400 ms**, p99 < 800 ms |
| Reads / lookups | p95 < 150 ms |
| Webhook first attempt after commit | p95 < 5 s |
| Webhook delivery to a healthy endpoint | 99 % within 60 s, 99.9 % within 5 min |
| Reconciliation lag (store side) | < 5 min |
| Duplicate orders from retries | **0** |
| Recovery objective | RPO ≤ 5 min (binlog), RTO ≤ 1 h |
| Error budget | ≤ 0.1 % 5xx of intake over 30 days |

## 3. Capacity model (kept honest)

- **Observed:** 1,298 orders in 33 days ≈ **39/day**, ≈ 5 webhook events each → **≈ 200/day**.
- **Design point:** **50 orders/min sustained, 500/min burst** (≈ 1,900× and ≈ 19,000× today's average).
  If a store runs a flash sale, 500/min is still ≈ 8 req/s.
- **Concurrency maths:** at p95 400 ms and 8 req/s, ≈ 3–4 FPM workers busy; `max_children = 20` gives
  5× headroom. Webhooks at 500 events/min with a healthy endpoint (≈ 300 ms) need < 3 worker-seconds per
  second → **2 workers** suffice; the second is for failure isolation, not throughput.
- **Storage:** `integration_requests` stores raw request and response per call. At 500 calls/min
  sustained that is 21 M rows/month — far more than real traffic, but the 30-day prune must **run and be
  monitored**, and its table needs `created_at` indexed (it is **[V]**).
- **Scale-out path (only if the design point is exceeded):** more FPM workers → more queue workers →
  read replica for the portal lists. No re-architecture.

## 4. Reliability design

**Logistic (platform side)**
- Keep: three-layer idempotency, signed webhooks, retries, `dropped` + replay, kill switches (global and per store).
- Add: **jitter** (V4), **circuit breaker** and **per-store concurrency cap** (V3), a **webhook poison guard**
  (a payload that always throws is parked after N crashes, not retried forever), and `X-Request-Id` on every
  call and webhook (V8).
- **Time budgets:** inbound requests must finish < 3 s; outbound webhooks connect 3 s / total 15 s (already 15 s).
  No external call (maps, geocoding) inside the intake transaction. Area resolution is an indexed query
  ([R] — confirm no per-request external call in `CityService::resolveForPoint`).
- **Degradation order when something is down:** Redis down → 503 + `Retry-After: 5` (fail closed);
  queue down → intake still succeeds (webhooks are rows first, delivered later); DB down → 503.

**Store side (the kapitano_web client)**
- Outbox row written in the same step as detecting the order; a sender claims rows with
  `SELECT … FOR UPDATE SKIP LOCKED`-style claiming so two workers never send one row.
- Deterministic idempotency keys (`create-{id}`), backoff with jitter, retry only network/429/5xx,
  a per-endpoint circuit breaker, a park-and-alert list for unmappable orders.
- A **reconciliation sweep** every 5 minutes (`updated_since` once it exists; per-order reads before).
- Clock on NTP (a 5-minute drift breaks every signature).

## 5. Security hardening

| Control | Status | Action |
|---|---|---|
| Per-store token + HMAC + replay nonce + IP allowlist | ✅ [V] | — |
| Token lifetime | ❌ V1 | Explicit TTL, expiry visibility, notifications |
| Secrets at rest (`signing_secret`, `webhook_secret`) | [R] stored `text` | Confirm the model casts them `encrypted`; add a test that the raw column is not plaintext |
| Credential events audit | [R] unconfirmed | Record who generated/rotated/revealed, when, from which IP; email the owner on each |
| SSRF on webhook URL | ⚠️ V10 | Resolve-and-pin at send time |
| TLS | ✅ nginx 1.2/1.3 | Add HSTS; certificate-expiry alert |
| Log redaction | [R] | Never log `Authorization`, signatures, secrets, or full phone numbers; test with a log-capture assertion |
| PII retention | ⚠️ | Payload logs 30 d (`integration:prune`); confirm legal retention; mask phones in logs |
| Dependency hygiene | — | `composer audit` in CI; Renovate/Dependabot |
| Tenant isolation | ✅ read/update 404 [V] | Add: same `external_order_id` in two stores; token A + webhook of B; idempotency key reuse across stores |
| Rate limiting | ✅ app-level | Add edge `limit_req` (V11) |
| Secrets management | — | Environment/secret store, never in git; rotate the token/keys that were pasted into chats or tickets |

## 6. Observability

**Request correlation:** one `X-Request-Id` → logs, `integration_requests`, webhook headers, support tickets.

**Metrics (RED + queue):** requests/s, error rate and latency by endpoint and by store; intake outcome by
`MessageDebug` key; queue depth **and age of the oldest job** per queue; webhook success rate, attempts
histogram, `dropped` count; Redis latency and errors; DB slow queries; FPM saturation.

**Alerts (proposed thresholds):**

| Alert | Condition | Severity |
|---|---|---|
| Intake 5xx | > 1 % over 5 min | Page |
| Redis unavailable | any `replay_check_unavailable` 503 | Page |
| Webhook queue age | oldest job > 2 min | Page |
| Dropped webhooks | any new `dropped` | Ticket (and notify the store) |
| Token expiring | < 30 d / < 7 d | Notify owner |
| Prune not run | no run in 26 h | Ticket |
| Cert expiry | < 14 d | Ticket |
| Synthetic check | signed `GET /health` fails 3× | Page |

**Synthetic probe:** a signed `GET /health` every minute from outside the network (spacing > 1 s because
of V13), plus the unauthenticated `/up` for load balancers (**[V]** exists).

**Existing:** `integration:health` every 15 minutes with a daily-deduplicated dashboard notification —
keep it as the in-app safety net, not the primary alerting.

## 7. Data and disaster recovery

- **Backups:** daily full + continuous binlog (point-in-time recovery). Verify a restore **monthly** on a
  scratch server and record the time it took.
- **Retention:** `orders` permanent; `integration_requests` / `webhook_deliveries` 30 d; confirm with the
  data owner. Prune is scheduled **[V]** (03:40) — add the "not run" alert.
- **Migrations:** additive, backwards compatible (expand → migrate → contract). Indexes online.
- **Redis:** persistence (AOF every second) for the nonce keys; losing them only opens a 5-minute replay
  window, so Redis loss is a *security* event to alert on, not data loss.
- **Secrets recovery:** lost signing secret → rotate (24 h overlap exists **[V]**); lost token → regenerate.

## 8. Release engineering

- **Contract tests:** every PR runs the OpenAPI-vs-real-response check (**[V]** exists) plus the
  conformance scenario (`integration:conformance`, from the review's P2).
- **Config checklist (production `.env`):** `QUEUE_CONNECTION=redis`, `CACHE_STORE=redis`,
  `SESSION_DRIVER=redis`, `APP_DEBUG=false`, `TRUSTED_PROXIES` set (runbook: security setting),
  `INTEGRATION_ALLOW_PRIVATE_WEBHOOK_URLS=false`, `INTEGRATION_ACCEPT_ORDERS=true`, JSON log channel.
- **Deploy:** `migrate` → `queue:restart` → cache warm; **never** deploy during a webhook-backlog alert.
- **Rollout:** per-store flag first (`is_active`), then global; the kill switches (**[V]**) are the rollback.
- **Rollback:** previous release + `queue:restart`; migrations are additive so no down-migration is needed.
- **Versioning:** additive changes in `v1`; breaking → `v2`; ≥ 6 months notice (P3 of the review).

## 9. Load and resilience test plan (prove the targets)

Run against a production-like staging (same FPM/Redis/MySQL sizing). Tool: k6 or Artillery. Each scenario
has a **pass criterion**; record evidence per the one-gate-at-a-time protocol.

| # | Scenario | Pass when |
|---|---|---|
| L1 | Steady 5 req/s `POST /orders` (unique ids), 30 min | p95 < 400 ms; 0 errors; no duplicate orders |
| L2 | Burst 50 req/s for 2 min | p95 < 800 ms; error < 0.1 %; queue age < 30 s after |
| L3 | Idempotent replay storm: same key × 1,000 | exactly 1 order; 999 × 200 `Idempotency-Replayed` |
| L4 | Two stores, same `external_order_id`, concurrently | two orders, correct isolation, correct webhooks to each URL |
| L5 | Store A's webhook endpoint hangs 15 s; store B healthy | store B's p95 delivery unaffected (V3) |
| L6 | Endpoint dead 30 min then recovers | retries resume with jitter, nothing lost, no stampede |
| L7 | Redis killed 60 s | signed calls 503 + `Retry-After`; auto-recover; alert fires |
| L8 | Queue workers killed | intake continues; queue-age alert fires within 2 min; no loss after restart |
| L9 | MySQL failover / restart | in-flight requests fail cleanly (no half orders); idempotent retries succeed |
| L10 | Token aged 200 d (V1) | still authenticates until its own `expires_at`; clear `token_expired` after |
| L11 | 24 h soak at 1 req/s | no memory growth in FPM/worker; prune keeps tables bounded |
| L12 | Clock skew ± 6 min | `signature_expired` with a correct skew hint in `/health` |
| L13 | Secret rotation under traffic | zero failed requests inside the overlap window |

## 10. Store-side (kapitano_web client) production checklist

- [ ] Shop → pickup coordinates for every active shop (0 of 67 have them **[V]**).
- [ ] Drop-off fallback decided (92 % of orders lack coordinates **[V]**) and "approximate location" flagged.
- [ ] COD amount rule signed off (USD base × 150, shipping rule).
- [ ] One logistic order per web order, or merged, decided (234 multi-seller checkouts **[V]**).
- [ ] Safe mode **off** only by deliberate flag; real-customer messaging verified impossible while on.
- [ ] Outbox, jitter, circuit breaker, reconcile sweep, park-and-alert list implemented and tested (L3, L6).
- [ ] Token expiry visible and renewal runbook written (V1).
- [ ] Webhook receiver public on 443, verifies every `v1`, dedupes `event_id`, orders by `sequence`.
- [ ] Dashboards/alerts for the client (outbox age, parked count, error rate).
- [ ] Runbook: "order not in logistic", "status not updated", "token expired", "keys rotated".

## 11. Go-live gate (all must be checked)

**Security** — V1 fixed and tested · secrets encrypted at rest (test) · credential audit trail · SSRF resolve-and-pin · edge rate limit · HSTS · log redaction test · `composer audit` clean.
**Reliability** — dedicated webhook queue + ≥ 2 supervised workers · circuit breaker · jitter · poison guard · Redis HA + persistence · kill switches rehearsed.
**Performance** — L1–L5 and L11 pass at the §2 targets · V6 indexes added and `EXPLAIN`ed · Redis for queue/cache/session/limiter · OPcache confirmed.
**Observability** — request-id end to end · JSON logs · metrics and alerts in §6 live · synthetic probe live · on-call owner named.
**Data** — backups + one successful restore drill · retention confirmed · prune alert.
**Contract and docs** — `store-api-ar.md` drift fixed (SAR → SYP, missing/extra fields, events) · OpenAPI regenerated · deprecation policy · onboarding guide updated · conformance command passes for kapitano_web.
**Ops** — runbooks for the five common faults · rollback rehearsed · staged rollout plan · incident contacts.

## 12. Deliberately **not** doing (to avoid over-engineering)

No Kafka or event bus, no microservices, no sharding or partitioning, no GraphQL, no service mesh, no
multi-region, no OAuth server, no separate "platform" proxy. Today's volume is ≈ 39 orders a day; each of
those adds failure modes and cost with no benefit at the design point in §3. Revisit only on the triggers
in the review's §9 or if load testing misses a §2 target.

## 13. Delivery order (one gated slice at a time)

| Slice | Work | Gate (evidence recorded) |
|---|---|---|
| **P0** | **V1 token lifetime** (callback, explicit TTL, `token_expired`, expiry notifications) | L10 + regression test with a 200-day-old token |
| P1-a | V4 retry schedule + jitter; V13 signature `v2` | Exact-schedule test; same-second reads pass |
| P1-b | V8 request-id + JSON logs + error tracking | One id traceable from a store call to its webhook |
| P1-c | V6 indexes + `updated_since` list | `EXPLAIN` uses the new indexes; cursor test |
| P1-d | V2/V3 Redis for queue/cache/limiter; dedicated webhook queue; circuit breaker; per-store cap | L5, L6, L8 |
| P1-e | Isolation tests (same id in two stores) | L4 |
| P2 | V9 Redis HA/readiness; V10 SSRF pin; V11 edge limits; audit trail; sandbox; conformance command; per-store counters | L7, L12, L13 |
| Client | kapitano_web client slices APIs 1→7 (as in the approach document) | Per-API gate |
| Final | Full §9 run + §11 checklist | Sign-off |

Each slice stops at its gate and waits for your acceptance before the next begins.