Troubleshooting

Locate the stage that failed before retrying the whole operation. Keep the request ID, safe error code, process role and task/delivery ID when available. Do not include tokens, OTPs, recipient payloads, environment dumps or connection credentials in a report. Configuration and transport errors deliberately omit sensitive values.

Pick the failing boundary#

Configuration or startup fails#

Use the same profile in each terminal. Existing shell values override dotenv; changing .env does not change a running process. APP_ENV_FILE='' disables file loading, while an explicit path must exist. Defaults and supported keys are in the configuration reference.

Read-only diagnostics, after the application can construct:

go run ./cmd/cli migrate status
go run ./cmd/cli route list
curl -i http://127.0.0.1:8080/healthz
curl -i http://127.0.0.1:8080/readyz

Run those commands with the same selected profile as the failing process. The quickstart disables authentication, push, and storage; mail is always constructed and connects on send. A quickstart check does not diagnose an enabled full profile. Do not use migrate refresh to repair a database containing data you need: it deletes the configured schema's objects/data. Choose a forward migration or an explicitly disposable database instead.

Authentication fails or a route is missing#

Serialize refresh calls per session. Persist a successful replacement before the next refresh; never keep retrying an old refresh token after an uncertain result. Recover with a new sign-in when the session is invalid. Access tokens expire after seven days by default, while refresh tokens and sessions have no absolute time limit by default. Revocation and account state still apply. See Identity and the API contract.

OTP send returns an accepted challenge; account creation occurs when OTP or social proof succeeds. Verification returns 201 for a new account and 200 for an existing account. After either status, inspect data.user.profile_complete and use PATCH /users/me for profile completion. There is no second registration request or verification-unavailable signup warning. Neither sign-in response confirms a background welcome reaction completed.

Queued work does not run#

Use the always-available queue commands to inspect the actual queue and task ID returned at enqueue:

go run ./cmd/cli queue list
go run ./cmd/cli queue inspect default TASK_ID

Inspection can expose saved payloads; keep output private. Check in this order:

1. Producer and worker use the same Redis address and database number; confirm both process profiles.
2. The worker consumes the task's queue; check QUEUE_DEFAULT, QUEUE_ROUTES and the named priorities. A per-task queue option can override routing.
3. A worker process is running, and the queue is not paused. HTTP-only processes enqueue work but do not execute it.
4. The task is due and its type is registered. Every consumer of app.command must register the requested command factory; every consumer of mail.send.v1 needs valid SMTP settings.
5. Inspect retry/archived state and the safe error. Fix dependencies or invalid payloads before choosing replay. Successful tasks normally disappear unless retained.

queue replay QUEUE ID moves eligible work to pending and can repeat effects. It does not reset retry history. First establish whether the previous attempt committed or sent anything. Do not delete tasks or flush Redis as a diagnostic step. See failures and retries.

For recurring work, run one cmd/scheduler process. cmd/worker never starts cron scheduling. Central schedules also have feature conditions; command handlers are registered independently of scheduler execution. Calendar expressions use UTC, and missed ticks are not replayed. See scheduling.

Notifications are pending, failed or duplicated#

Notification intent is stored in PostgreSQL before a dispatcher submits delivery IDs to Redis. A returned receipt inside a transaction still depends on commit. First inspect the ledger rather than assuming the notification is an ordinary task:

go run ./cmd/cli notifications deliveries --status pending
go run ./cmd/cli notifications deliveries --status awaiting_receipt
go run ./cmd/cli notifications deliveries --status failed
go run ./cmd/cli notifications delivery DELIVERY_ID

• Pending work needs a dispatcher and worker using the same database and queue. Workers run the dispatcher without cron. notifications dispatch is a manual enqueue operation; it does not deliver synchronously.
• Sending rows are leased. Let an expired lease recover; do not manually overwrite state while a worker may still own it.
• Pending rows can be waiting for their next automatic attempt; inspect the due time. Failed rows need operator review. Use notifications retry DELIVERY_ID after correcting the cause and accepting possible duplicate provider effects.
• An omitted key creates a new occurrence on every Enqueue call. Use the same business occurrence key across retries to reuse saved channel intents.
• A repeated key retains the first snapshot; editing a template does not rewrite existing delivery content. Database, email and push channels progress independently.

Email success means SMTP accepted the operation, not inbox placement. A network failure after acceptance leaves uncertainty. Ordinary Mailer.Enqueue is tracked in Asynq, not the notification ledger; use queue inspection for those sends.

Expo awaiting_receipt is normal after ticket acceptance. The dispatcher starts receipt checks after 15 minutes and polls missing receipts without resending. An ok receipt confirms provider handoff, not device display. After 24 hours an unconfirmed result becomes a failed uncertain delivery; manual retry may duplicate. Check the current subscription, project configuration and client permissions using the push guide. Explicit logout detaches session-bound subscriptions; already admitted sends cannot be retracted.

A local event disappeared#

GoChannel delivers only within the publishing process. A CLI process cannot notify subscribers in an API/worker process. Register local handlers before producers start, and keep subscribers with each producer role. Router retry middleware bounds one processing cycle; a subsequent Nack can trigger GoChannel redelivery while the process is alive. It is not a global attempt cap. There is no configured failed.events dead-letter route or durable replay store. Restart and process failure can lose in-memory events.

For the existing welcome example, an operator can enqueue the stable welcome task with identity welcome USER_ID. For business work that must survive failure, persist notification intent in the business transaction or design an explicit outbox. Adding a broker container alone changes none of these guarantees. See events and transactional notifications.

Media upload, download or cleanup fails#

Check both PostgreSQL metadata and the disk selected by the asset. A configured definition alone is not an opened disk: use the default or additional-disk list. Keep disk names stable for existing records; changing the default does not move previous objects. Authentication supplies user ownership; other model types need business-authorized handlers.

The default HTTP upload accepts exactly one file and optional visibility, with a 20 MiB file limit and bounded multipart overhead. Authenticated downloads require a ready asset belonging to the caller; the content route allows public assets or valid temporary links. A ready metadata row with a missing object points to backing-storage or restore consistency, not a reason to reassign ownership.

Interrupted uploads and deletions retain recoverable metadata. Run media recover or the central recovery schedule, with storage available. Recovery leaves recent uploads alone and processes bounded batches; it is not a bucket inventory cleaner. Back up metadata and objects together. See Media.

The returned asset URL does not work#

See Media for direct versus internal URLs and visibility. A direct provider URL bypasses application response headers and may have provider/CDN cache behavior independent of metadata deletion.

Request diagnostics are missing#

Start with the request's response header or body ID:

{service_name="labena"} | request_id="YOUR_REQUEST_ID"

Use your APP_NAME as service_name if it differs. In Grafana Explore's Builder, add a Label filter for request_id; it is structured metadata, not an indexed stream label. New OTLP records need no JSON parser.

Follow request → log → trace → SQL. Do not enable raw payload or SQL-value logging to diagnose these boundaries.

Readiness or shutdown does not finish#

Readiness is a startup signal unless dependency probing is enabled. A green ready response does not establish schema correctness, SMTP delivery or cloud health. During shutdown it becomes unavailable before admission stops. Worker-only processes have no business HTTP readiness endpoint.

Shutdown stops admission, drains workers then local subscribers, stops metrics and closes owned resources. A handler that ignores cancellation can exhaust the shared drain budget. The app then reports failure and refuses to close dependencies still in use; it cannot forcibly stop a goroutine. Give the deployment enough termination time for both shutdown and cleanup, and ensure the executable receives the signal.

Fix unbounded work or an unreachable dependency before increasing timeouts. Check application lifecycle and observability for correlation and bounded diagnostics. Recovery after forced termination must still tolerate incomplete operations and duplicate task delivery.

Prepare a useful issue report#

Include the operation and expected result, source revision, environment/role, relevant non-secret feature switches, HTTP status and safe error type, request ID, and task/delivery ID where applicable. State whether the problem reproduces after restart and which backend check failed.

Attach a minimal sanitized request or task shape only when needed. Remove access and refresh tokens, OTPs, connection URLs with credentials, provider payloads, and full configuration dumps. Report uncertain effects explicitly—for example, “SMTP acceptance is unknown”—so recovery does not accidentally duplicate them.