# RC1 Operations Runbook ## Persistent YAS runtime Run `php artisan gateway:smpp:run yas` with both YAS activation flags true. Bind-test and send-test remain short-lived diagnostics and do not drain durable work. The runtime publishes transactional outbox events internally before claiming submissions; no second publisher process is required. Stop through SIGINT/SIGTERM or the supervisor's console stop. Reconcile acknowledgement-unknown rows using safe identity/status/timestamp columns only; never export protected payload, address, credential, or ownership data. Public API fields, statuses, headers, and errors are governed by [API_V1_CONTRACT.md](api/API_V1_CONTRACT.md) and the authoritative [OpenAPI 3.1 document](api/openapi.yaml). Operational diagnosis must not expose fields those contracts keep internal. ## Safety rules - Work only in the intended environment and tenant scope. - Prefer read-only commands and queries during diagnosis. - Never put message bodies, API keys, SMPP passwords, webhook secrets, signature material, or unmasked recipient numbers into tickets or chat. - Use request correlation ID, public message ID, batch ID, tenant identifier, public status, and—only for authorized internal investigation—provider message identifier. - Do not claim external delivery from a queued API response. Public status is projected from durable evidence. ## Application and database health Repository-backed read-only checks: ```console php artisan about php artisan env php artisan migrate:status php artisan db:show php artisan route:list --path=api/v1 ``` Use `php artisan pail` only where the deployment permits an interactive log tail. Otherwise use the environment's approved log platform. Confirm `APP_DEBUG=false` in production without exposing the rest of the environment. Escalate immediately when application boot fails, the database is unavailable, pending migrations are unexpected, tenant isolation fails, secrets appear in output, or public responses expose internal details. ## Outbound processing ### Durable SMPP submission recovery boundary RC1-9A records provider command outcomes but does not add a persistent runtime command. `accepted` and `rejected` submissions are terminal and must never be returned to the pending queue. A `failed` record proves a sanitized local failure occurred before network submission and is also terminal until a separately approved retry policy exists. An expired claim with no `submit_started_at` evidence is safe for repository-controlled recovery to `pending`. Once `submit_started_at` exists, a missing durable provider response is ambiguous; recovery records `acknowledgement_unknown` and operators must not manually requeue or resend it. Reconciliation requires authorized provider evidence and a separately approved procedure. Historical claimed records encountered by the migration are deliberately treated as having possibly crossed the network boundary. Provider acceptance means the SMSC acknowledged `submit_sm`; it does not mean handset delivery. Only normalized DLR evidence can project delivered, failed, expired, or rejected delivery outcomes. Inspect pending outbox work without mutation: ```console php artisan gateway:publish-outbox --dry-run --limit=100 ``` Process at most one pending event only during an approved operational action: ```console php artisan gateway:publish-outbox --once ``` Relevant safe evidence includes public message ID, tenant ID, correlation ID, attempt outcome, provider code, and authorized internal provider reference. Do not log or copy the SMS body. If `YAS_OPERATIONAL_DISPATCH_ENABLED=false`, YAS operational publishing is deliberately inactive. Enabling it requires complete valid SMPP configuration and an approved change. A queued message alone does not prove an active sender process. ## YAS connectivity and process health The implemented bind diagnostic is: ```console php artisan gateway:smpp:bind-test yas ``` The implemented live-message diagnostic is: ```console php artisan gateway:sms:send-test yas --to= --from= --message= --request-dlr ``` Use these only with YAS authorization, approved test data, and an acceptance window. A successful bind diagnostic is not proof that a persistent runtime exists. The repository has no command to start, supervise, stop, or restart a persistent SMPP `RuntimeLoop`. Therefore: 1. Check the deployment platform's approved process inventory without guessing a service name. 2. If an expected externally managed process is stopped, preserve its sanitized logs and configuration state. 3. Use only the environment's approved supervisor action and runbook. 4. Re-run the bind diagnostic and one approved smoke test after restart. If no approved supervisor/runbook exists, this is a NO-GO and must be escalated; do not substitute an ad hoc shell loop. ## Inbound DLR processing Successful correlated receipts create immutable rows in `message_delivery_receipts`, update the message's internal delivery evidence when the transition is legal, and append a message event atomically. The public API uses `PublicMessageStatusProjector`; operators must not manually rewrite public status. Safe database inspection, subject to authorized read access: ```sql SELECT public_id, tenant_id, message_id, provider_code, provider_state, delivery_status, received_at FROM message_delivery_receipts WHERE message_id = ORDER BY received_at, id; ``` The internal message ID and provider message identifier are not public API fields. Keep them inside authorized operational evidence. Search application logs for `SMPP delivery receipt processed.` using correlation ID, outcome, tenant ID, and public status. Outcomes include applied, duplicate, uncorrelated, correlated, and retained conflict categories implemented by RC1-5. - Unknown or ambiguous provider correlation must not mutate another message. - Exact duplicate evidence must not add another receipt outcome or webhook attempt. - Conflicting terminal evidence is retained without rewriting the established terminal state. - Malformed receipt processing is isolated to its inbound envelope and must not stop unrelated processing. Escalate an uncorrelated or ambiguous receipt when YAS confirms the provider reference should uniquely match an acknowledged attempt. Capture masked time, provider code, authorized internal provider reference, correlation ID, and affected public message ID where known. ## Webhook operations Use tenant-authenticated `GET /api/v1/webhook` to inspect the safe URL and enabled state. It never returns a secret. Secret replacement requires `PUT /api/v1/webhook`; do not read or edit ciphertext directly. Authorized internal inspection: ```sql SELECT e.public_id, e.tenant_id, e.public_message_id, e.public_status, e.occurred_at, a.outcome, a.http_status, a.failure_category, a.attempted_at, a.completed_at, a.retain_until FROM tenant_webhook_events AS e LEFT JOIN tenant_webhook_delivery_attempts AS a ON a.tenant_id = e.tenant_id AND a.webhook_event_id = e.id WHERE e.public_message_id = ''; ``` `outcome=delivered` means the receiver returned a successful 2xx. `outcome=failed` retains a sanitized HTTP or connection category. `in_progress` means an attempt was reserved but no completion was durably recorded. RC1 performs no retry; do not manually replay an event because no approved replay command or API exists. A webhook HTTP failure must not reverse the message's terminal status. If it does, stop acceptance and escalate as a critical transaction-boundary defect. ## Authentication and idempotency diagnosis Authentication failures use the standardized public error boundary. Check credential state, tenant/application state, request prefix, and environment label through approved administrative access. Never request the full bearer credential in a ticket. For `IDEMPOTENCY_CONFLICT`: 1. Confirm the authenticated tenant. 2. Confirm whether the key was reused with a different semantic payload. 3. Confirm the client did not change ordered bulk contents. 4. Use correlation ID, message ID, or batch ID for evidence; do not expose the raw idempotency key beyond the authorized request trace. Equivalent single or bulk requests should replay the original business result. The response header identifies the current request; a single-message response body retains the original submission correlation ID. ## Log categories and identifiers | Signal | Safe search fields | | --- | --- | | Public API request/submission | current request correlation ID, tenant ID, public message ID, optional client reference | | Bulk submission | current request correlation ID, tenant ID, batch ID, accepted/rejected counts | | SMPP DLR processing | correlation ID, tenant ID, internal provider message identifier when authorized, public status, outcome | | Webhook attempt | correlation ID, tenant ID, public message ID, webhook URL, terminal public status, outcome | | Sanitized public failure | request ID, correlation ID, route, tenant/application ID, exception class | The webhook signature, active/previous secret, API credential, SMPP password, message body, and encrypted payload are never safe ticket fields. ## Escalation criteria Escalate and halt affected acceptance when any of these occurs: - cross-tenant data visibility; - plaintext secret or message-content exposure; - conflicting idempotent replay; - a DLR changes the wrong message; - a duplicate DLR creates another webhook attempt; - webhook network I/O occurs before commit; - webhook failure rolls back delivery state; - database constraint or migration failure; - missing or unknown YAS endpoint, credential, sender, TON/NPI, throughput, firewall, or support information; - an unhealthy SMPP process without an approved restart procedure; - a critical or major review finding. Record the evidence package defined in [ACCEPTANCE_CHECKLIST.md](ACCEPTANCE_CHECKLIST.md) and follow the approved incident/change process.