Troubleshooting

The canonical checklist for when something in the compliance stack isn't behaving. Walks through MDS endpoints, the Policy ingester, real-time enforcement, the city portal, and digest emails — in that order, because that's roughly the dependency chain.

MDS endpoints

401 on every request

Symptoms: city's validator returns 401 Unauthorized on /vehicles/status.

Checks:

1. The bearer token is missing from Authorization: Bearer <token>. Cities sometimes paste the token without the Bearer prefix.
2. The token was revoked. Open Settings -> API & Integrations -> MDS Tokens and confirm the token is active.
3. The token belongs to a different subaccount. One token per subaccount; cities accidentally testing across multiple jurisdictions will see this.

Validator fails on version field

Symptoms: mds-provider-validator complains the response envelope reports a version other than 2.0.1.

Checks: confirm src/lib/mds/v2/response.ts has const MDS_VERSION = '2.0.1'. If you deployed a stale build, redeploy.

MDS-JWT header missing on a response

Symptoms: city expects every response to carry MDS-JWT: <jwt> and the header is absent.

Checks:

1. mds_jwks_keys has no is_active = true row for the subaccount. Hit /api/mds/<subaccountId>/.well-known/jwks.json once to force a lazy mint, then retry.
2. The keypair was generated but the active row has private_key_pem corrupted (rare — usually a manual edit gone wrong). Look at Sentry for MDS sign failed breadcrumbs.

current_speed_limit_kph always 0

Symptoms: vehicles report current_speed_limit_kph: 0 even when they should be in a zone.

Checks:

1. The vehicle has no last-known GPS position. Inspect vehicles.last_lat/last_lng and vehicles.last_seen.
2. No operator zone or policy geofence contains the point. Run SELECT * FROM zones_containing_point('<subaccountId>', <lat>, <lng>) and confirm at least one row is returned.
3. The active rule's rule_type is no_ride rather than speed — that's a lock, not a speed limit. The status row should show is_disabled = true instead.

Policy ingester

City's Policy feed returns 404

Symptoms: audit log entries with status = 'failed' and errors containing HTTP 404.

Checks:

1. The URL in mds_jurisdictions.policy_feed_url is wrong. Test from your terminal: curl -i <url>. Cities sometimes change paths between provisional and production endpoints.
2. The auth token expired. Try with and without the token to isolate.

Feed validates but yields no policy_geofences

Symptoms: audit shows status = 'success' and mds_policies rows exist, but policy_geofences is empty.

Checks:

1. Every rule referenced geographies the city didn't publish. Inspect errors in the audit row — unresolved geography IDs are listed there.
2. The city published policies for a vehicle type your fleet doesn't operate. Confirm the rules' vehicle_types include at least one type you have vehicles for.

Same policy keeps appearing in every diff

Symptoms: every poll's diff shows the same policy in modified, even when nothing meaningful changed.

Checks:

1. The city is regenerating the feed payload on every request (different published_date, whitespace, ordering). The sha256 short-circuit fires only on byte-identical payloads. Ask the city to stabilize their serialization.
2. The city changed published_date automatically — some CMSes do this. The diff is harmless but noisy; can be filtered out in the audit log UI.

Activation never fires

Symptoms: a policy is pending, start_date has passed, but the status never flips to active.

Checks:

1. The mds-policy-activate cron is failing. Check /api/cron/mds-policy-activate in Vercel cron logs.
2. The start_date is in the city's local time but stored as UTC without conversion. The cron compares against now() UTC; a mis-set timezone in the feed causes activation to fire much later than expected.

Real-time enforcement

Rule activated but no commands sent

Symptoms: mds_policies.status = 'active' but policy_enforcement_events has zero rows for the rule.

Checks:

1. No vehicles are inside the geometry. SELECT COUNT(*) FROM vehicles WHERE ST_Contains('<rule_geometry>', ...) — if zero, there's nothing to enforce.
2. All vehicles have stale GPS (>5 min). The skip is by design. Look for error = 'stale_gps' events in the table.
3. The vehicles have no linked iot_devices row. Skipped with error = 'no_iot_device'.

Commands sent but vehicles don't slow down

Symptoms: policy_enforcement_events shows command_sent_at but no command_ack_at, and the vehicle continues at full speed.

Checks:

1. OEM-specific OEM offline or asleep. Open the vehicle in the dashboard and inspect last_seen.
2. The wrong command format reached the OEM. Inspect command_response for an error code — most OEMs return rejection codes in plain text.
3. The vehicle's IoT password is wrong. Confirm iot_devices.iot_password matches what the OEM expects (see OKAI integration, Queclink integration, etc.).

Sub-fleet sees commands, another sub-fleet doesn't

Symptoms: OKAI vehicles obey the slow-zone, Segway vehicles don't (or vice versa).

Checks:

1. The per-OEM dispatcher branch is missing or stale for the OEM. Confirm src/lib/iot/dispatch.ts has a case for that OEM and the command format matches the OEM's protocol version.
2. Segway BLE-relay latency — the command was sent but hasn't been acked yet because the vehicle is far from any phone. This is a known gap; latency for Segway can exceed 30 seconds.

City portal

Magic link returns "invalid token"

Symptoms: contact clicks the emailed link and lands on an error page.

Checks:

1. The token expired (15 minutes). Have them request a new one.
2. The contact row was deleted or portal_access = false.
3. Two links were minted in quick succession; the first consumed, the second appears stale.

Magic link email never arrives

Symptoms: contact submits the form, gets the "check your email" message, no email comes.

Checks:

1. Spam folder.
2. The email doesn't match a city_contacts row exactly (case-insensitive). Inspect the row.
3. The email infra rejected the send (Sentry will have the bounce). Look for spam-filter triggers on subject or body.

Portal shows no vehicles even though fleet is deployed

Symptoms: city contact logs in, the fleet map is empty.

Checks:

1. mds_jurisdictions.bbox is wrong — too tight or pointing elsewhere. Confirm against a map preview.
2. All vehicles have null last_lat/last_lng. Inspect at least one.

Digest emails

Digest didn't arrive on the expected day

Symptoms: city contact expected a Monday weekly digest, didn't get one.

Checks:

1. last_digest_sent_at is in the current week — the cron decided the cadence window hadn't elapsed. Likely a previous digest fired in the same window.
2. The cron didn't run at all. Vercel cron logs for /api/cron/compliance-digest.
3. The digest fired but the email send errored. Check city_compliance_reports — if the row exists but no email landed, the failure is downstream.

Digest shows a failing condition that you've already fixed

Symptoms: scoreboard in the digest shows red on complaint_sla even though all complaints were responded to.

Checks:

1. The digest covers a closed period (yesterday for daily, last week for weekly). It will not retroactively recompute. The current scoreboard at /dashboard/compliance/{id} is the live view.

PDF attachment missing on monthly digest

Symptoms: monthly digest arrives but pdf_url is null and the attachment is absent.

Checks:

1. The PDF renderer errored. Sentry will have the stack trace. The HTML/text body still fires; the PDF is best-effort.

Where to look in the database

When all else fails

Email support@levyelectric.com with:

• Subaccount ID
• Jurisdiction slug
• The audit run_id if the issue is in the ingester
• The policy_enforcement_events.id if the issue is in enforcement
• The city_compliance_reports.id if the issue is in a digest

We have direct access to logs and can usually pinpoint the failure within an hour.