Third-Party Installation Guide

1. Installation result

At the end of this procedure, the customer has:

• a customer-operated Kubernetes cluster inside the approved country or facility;
• Levy runtime images stored in the customer's private OCI registry;
• a PostgreSQL database with PostGIS and the Levy schema;
• customer-owned S3-compatible object storage for photos, exports, and evidence;
• local rider authentication and customer OIDC operator authentication;
• local payment and messaging adapters selected by the customer;
• local IoT and GPS ingestion, normalization, vehicle-state processing, and command routing;
• the rider API, operator web application, workers, schedulers, and installation controller;
• local metrics, dashboards, logs, alerts, backups, and recovery tooling;
• no required connection to Levy's hosted production environment; and
• no Levy source repository, original source archive, source map, or build credential on the customer network.

The application is delivered as signed OCI images and compiled command-line tools. The installer operates through levyctl; the controller performs the Kubernetes and database changes. This protects the product source while still providing a repeatable installation process.

The installed product is the micromobility rental platform: rider identity, fleet and vehicle management, reservations, rentals, pricing, wallet/payment records, operator functions, IoT/GPS, notifications, reporting, and the operational services needed to run them. Levy's hosted CRM, sales automation, marketing systems, internal support administration, Stripe integration, Twilio integration, and other Levy corporate services are not installed. An external AI customer-support bridge is optional, off by default, and isolated from the rental runtime.

2. The short version

The complete installation has four layers:

1. Servers and network: customer-provided Linux servers or virtual machines, storage, load balancers, DNS, certificates, firewall rules, and time synchronization.
2. Local platform: Kubernetes, container runtime, private registry, PostgreSQL/PostGIS, object storage, identity, secrets, monitoring, and backups.
3. Levy installation system: the signed Levy tools, release bundle, installation controller, policies, and site configuration.
4. Levy application: API, web, workers, schedulers, IoT/GPS services, adapters, database schema, dashboards, and acceptance tests.

For an already certified customer platform, the application portion is installed with this sequence:

levyctl package verify --package /opt/levy/packages/levy-core-onprem-1.0.0
levy-custody registry import --package /opt/levy/packages/levy-core-onprem-1.0.0 --registry registry.fleet.example.eg/levy
levyctl controller bootstrap --site /etc/levy/site.yaml --release 1.0.0
levyctl site validate --file /etc/levy/site.yaml
levyctl secrets verify --site /etc/levy/site.yaml
levyctl preflight run --site /etc/levy/site.yaml --release 1.0.0
levyctl install plan --site /etc/levy/site.yaml --release 1.0.0 --out /var/lib/levy/install-plan.json
levyctl install apply --plan /var/lib/levy/install-plan.json --approval "$LEVY_CHANGE_APPROVAL"
levyctl install status --watch
levyctl verify core --site /etc/levy/site.yaml
levyctl verify integrations --site /etc/levy/site.yaml
levyctl acceptance assemble --site /etc/levy/site.yaml --out /var/lib/levy/acceptance.json

The remaining sections explain how those inputs and prerequisites are created.

3. Installation roles

The installation is intentionally split between customer administrators and the third-party installer.

The certified installer does not receive Kubernetes credentials, registry pull credentials, database credentials, secret values, node access, or image-export access. Customer administrators complete privileged bootstrap steps before the installer identity is enabled.

4. Supported installation profiles

4.1 Production HA

Use this profile for a customer production launch.

The signed capacity report can increase these values based on vehicles, GPS frequency, active rides, image volume, retention, and recovery targets. Production uses at least three independent failure domains for control-plane and database placement and at least two for application, IoT, ingress, registry, and storage placement.

4.2 Compact production

For smaller fleets, control-plane and application roles can be converged onto three 16-vCPU, 64-GB Kubernetes servers. PostgreSQL remains a separate three-node service, and object storage remains redundant. The site packet records the shared-failure-domain risk and the signed capacity test proves the contracted peak.

4.3 Single-server pilot

A nonproduction pilot can run on one physical server or hypervisor with 16 CPU cores, 64 GB RAM, 2 TB usable encrypted SSD, and four VMs:

This profile is labeled pilot_single_site. It is not an HA production profile and does not satisfy a multi-failure-domain RPO or availability commitment.

5. Software installed on the servers

The exact patch versions are recorded in manifests/platform-lock.json inside every Levy release. The Levy Core On-Prem 1.0 reference profile uses the following software families.

The customer does not need Node.js, npm, Supabase, Stripe, Twilio, Vercel, Docker Compose, Git, or access to Levy's source repository on the production servers.

6. What the customer receives

The customer receives one encrypted release archive and one customer-specific site archive through the approved online or offline custody channel.

levy-core-onprem-1.0.0+20260901/
  README-FIRST.txt
  SHA256SUMS
  SHA256SUMS.sig
  manifests/
    release-index.json
    release-index.sig
    platform-lock.json
    component-bom.json
    image-index.json
    source-exclusion-receipt.json
    vulnerability-receipt.json
    sbom.spdx.json
  bin/linux-amd64/
    levyctl
    levy-sitecheck
    levy-custody
  bundles/
    levy-images.oci.zst
    dependency-images.oci.zst
    helm-charts.oci.zst
    mobile-artifacts/
  bootstrap/
    controller-bootstrap.yaml
    controller-policy.yaml
    installation-crds.yaml
    network-policies.yaml
    admission-policies.yaml
  database/
    baseline.tar.zst
    migrations/
    manifest.json
  conformance/
    platform/
    database/
    object-storage/
    providers/
    iot/
    residency/
    golden-journeys/
  examples/
    site.production-ha.yaml
    site.compact-production.yaml
    site.pilot.yaml
    secret-reference-ledger.yaml
  docs/
    install/
    operations/
    recovery/
    providers/
  licenses/
  signatures/

The separate site archive contains:

levy-site-cust_egypt_fleet_01-prod/
  SHA256SUMS
  SHA256SUMS.sig
  site.yaml
  site.sig
  entitlement.jwt
  entitlement.sig
  approvals/
  certificates/public/
  provider-manifests/
  device-manifests/
  evidence-template/

Secret values are not included. The site archive contains only references to entries created in the customer's secret manager.

7. Network, DNS, and firewall preparation

7.1 Required DNS names

Create these records before application installation:

7.2 Required inbound ports

7.3 Required internal flows

All other egress is denied. Levy's hosted application, Supabase, Stripe, Twilio, public package registries, and public error-reporting endpoints are not required destinations.

8. Before beginning

The installation window begins only after all items below are complete.

• Servers are racked or VMs created inside the approved country and facility.
• Every host has a unique hostname, static management address, current OS security patches, NTP, encrypted storage, and customer monitoring.
• Forward and reverse DNS work for all internal and public names.
• Public and internal TLS certificates have been issued with the correct SANs.
• Firewall rules match the signed network worksheet.
• The Kubernetes cluster, registry, database, object storage, identity, secrets, monitoring, and backup systems are installed.
• The release and site archives have been received inside the customer boundary.
• The Levy signing-root fingerprint has been verified through the separate onboarding channel.
• Customer and Levy change approvals are open for the exact release and site digest.
• Payment and messaging provider sandbox credentials are present in the customer secret manager.
• A test device or device simulator and staff rider accounts are available.
• The evidence directory is on customer-controlled encrypted storage.

9. Build or certify the local platform

The customer chooses one of two paths.

9.1 Customer-managed platform

Use this path when the customer already operates Kubernetes, PostgreSQL, object storage, identity, monitoring, and backup systems.

Copy the non-secret draft site configuration and the signed conformance bundle to the customer administrator workstation, then run:

sudo install -m 0755 ./bin/linux-amd64/levy-sitecheck /usr/local/bin/levy-sitecheck

sudo mkdir -p /etc/levy /var/lib/levy/evidence
sudo install -m 0640 ./examples/site.production-ha.yaml /etc/levy/site.yaml

levy-sitecheck run \
  --site /etc/levy/site.yaml \
  --platform-lock ./manifests/platform-lock.json \
  --catalog ./conformance/platform \
  --output /var/lib/levy/evidence/platform-readiness.json

Expected final output:

PASS platform.readiness
profile: production_ha
checks: 86 passed, 0 failed, 0 skipped
receipt: /var/lib/levy/evidence/platform-readiness.json

Any failure must be corrected in the customer platform. Do not bypass or edit the conformance catalog.

9.2 Levy reference platform

Use this path for a new environment using the offline reference stack. The customer platform administrator installs the exact packages from bundles/dependency-images.oci.zst and manifests/platform-lock.json; no server downloads software from the public internet.

The reference bootstrap performs these operations:

1. validates Ubuntu/RHEL version, kernel, CPU, memory, disks, NTP, DNS, firewall, and hostnames;
2. installs RKE2 and bundled containerd on the control-plane and worker nodes;
3. joins the nodes with customer-generated cluster tokens;
4. installs the CNI, default-deny policies, CSI integration, ingress, certificate integration, and admission policies;
5. registers the customer private registry and CA;
6. installs or connects PostgreSQL/PostGIS, object storage, OIDC, monitoring, and backup services; and
7. runs the same levy-sitecheck catalog used for a customer-managed platform.

On the first control-plane node:

sudo levy-sitecheck platform bootstrap \
  --profile production-ha \
  --inventory /etc/levy/platform-inventory.yaml \
  --bundle /opt/levy/packages/levy-core-onprem-1.0.0/bundles/dependency-images.oci.zst \
  --platform-lock /opt/levy/packages/levy-core-onprem-1.0.0/manifests/platform-lock.json \
  --output /var/lib/levy/evidence/platform-bootstrap.json

The command prints a one-time join command for the remaining control-plane and worker nodes. The join token is generated and retained by the customer and is never placed in the evidence output.

After all nodes join:

levy-sitecheck platform verify \
  --inventory /etc/levy/platform-inventory.yaml \
  --output /var/lib/levy/evidence/platform-readiness.json

Expected final output is PASS platform.readiness with every expected node and failure domain listed.

10. Receive and extract the Levy package

Run these commands on the customer administration workstation. The examples use an offline archive mounted at /media/levy-release.

export LEVY_RELEASE="1.0.0"
export LEVY_PACKAGE_ROOT="/opt/levy/packages/levy-core-onprem-${LEVY_RELEASE}"

sudo mkdir -p /opt/levy/packages
sudo tar --zstd -xf /media/levy-release/levy-core-onprem-1.0.0+20260901.tar.zst \
  -C /opt/levy/packages

cd "$LEVY_PACKAGE_ROOT"
sha256sum --check SHA256SUMS

Expected output lists every file as OK. A missing, extra, or changed file stops the installation.

Verify the aggregate signature with customer-controlled verification software:

cosign verify-blob \
  --key /etc/levy/trust/levy-release-root.pub \
  --signature SHA256SUMS.sig \
  SHA256SUMS

Expected output includes Verified OK and the release signer identity recorded in the custody worksheet.

Extract and verify the customer-specific site archive:

getent group levy-install >/dev/null || sudo groupadd --system levy-install
sudo mkdir -p /opt/levy/site /etc/levy
sudo tar --zstd -xf /media/levy-release/levy-site-cust_egypt_fleet_01-prod.tar.zst \
  --strip-components=1 \
  -C /opt/levy/site

cd /opt/levy/site
sha256sum --check SHA256SUMS
cosign verify-blob \
  --key /etc/levy/trust/levy-release-root.pub \
  --signature SHA256SUMS.sig \
  SHA256SUMS

sudo install -o root -g levy-install -m 0640 site.yaml /etc/levy/site.yaml

Expected output is OK for every site-package file and Verified OK for the signature. /etc/levy/site.yaml is now the canonical non-secret site configuration used by later commands.

11. Install the Levy command-line tools

Customer security installs the tools from the verified archive. The tools are compiled binaries; they do not install Node.js, Python, Git, or a compiler.

cd /opt/levy/packages/levy-core-onprem-1.0.0

sudo install -o root -g root -m 0755 bin/linux-amd64/levyctl /usr/local/bin/levyctl
sudo install -o root -g root -m 0755 bin/linux-amd64/levy-sitecheck /usr/local/bin/levy-sitecheck
sudo install -o root -g root -m 0755 bin/linux-amd64/levy-custody /usr/local/bin/levy-custody

levyctl version --verify --release-index manifests/release-index.json
levy-sitecheck version --verify --release-index manifests/release-index.json
levy-custody version --verify --release-index manifests/release-index.json

Expected output:

levyctl 1.0.0 linux/amd64 verified
levy-sitecheck 1.0.0 linux/amd64 verified
levy-custody 1.0.0 linux/amd64 verified
compatibility: PASS release 1.0.0

The certified installer is given execute permission for levyctl only. levy-custody and privileged levy-sitecheck platform operations remain customer-admin tools.

12. Import the application into the private registry

Customer security creates an immutable levy project in the local registry. The registry must preserve the current release, the previous supported release, and every digest needed by an open rollback window.

Online-to-local import:

levy-custody registry import \
  --package /opt/levy/packages/levy-core-onprem-1.0.0 \
  --registry registry.fleet.example.eg/levy \
  --ca /etc/levy/pki/registry-ca.pem \
  --credential-ref customer-secrets://levy/registry-import \
  --enforce-immutable \
  --receipt /var/lib/levy/evidence/registry-import.json

Offline archive import:

levy-custody offline import \
  --archive /opt/levy/packages/levy-core-onprem-1.0.0/bundles/levy-images.oci.zst \
  --index /opt/levy/packages/levy-core-onprem-1.0.0/manifests/image-index.json \
  --registry registry.fleet.example.eg/levy \
  --credential-ref customer-secrets://levy/registry-import \
  --receipt /var/lib/levy/evidence/registry-import.json

Verify the imported digests:

levy-custody registry verify \
  --registry registry.fleet.example.eg/levy \
  --index /opt/levy/packages/levy-core-onprem-1.0.0/manifests/image-index.json \
  --receipt /var/lib/levy/evidence/registry-verification.json

Expected result:

PASS registry.verification
images expected: 18
images matched: 18
mutable tags: 0
unexpected digests: 0

13. Prepare PostgreSQL and PostGIS

The customer DBA provides a PostgreSQL 16 service with TLS, PostGIS 3.5, PITR, monitoring, and tested backup storage. Levy supports a customer-managed external PostgreSQL cluster or the signed PostgreSQL operator profile included in the reference platform.

The DBA creates one database and three non-superuser roles using the customer secrets system:

The DBA creates the roles without superuser, replication, role-management, or database-creation privileges. Passwords or client certificates are assigned through the customer secrets platform rather than entered in the installation transcript:

CREATE ROLE levy_app LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION;
CREATE ROLE levy_worker LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION;
CREATE ROLE levy_migrate LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION;

The DBA creates the database and enables extensions:

CREATE DATABASE levy_core TEMPLATE template0 ENCODING 'UTF8';
\connect levy_core
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pgcrypto;
CREATE EXTENSION IF NOT EXISTS citext;
CREATE EXTENSION IF NOT EXISTS btree_gist;
GRANT CONNECT ON DATABASE levy_core TO levy_app, levy_worker, levy_migrate;

The DBA then runs the non-mutating Levy database check:

levy-sitecheck database verify \
  --site /etc/levy/site.yaml \
  --database-ref customer-secrets://levy/database-preflight \
  --catalog /opt/levy/packages/levy-core-onprem-1.0.0/conformance/database \
  --output /var/lib/levy/evidence/database-readiness.json

Expected result verifies version, extensions, encoding, collation, TLS, roles, connection capacity, disk latency, replication, WAL archiving, restore target, and absence of an incompatible Levy baseline. The schema itself is installed later by the Levy migration workload.

14. Prepare object storage

Create private buckets or prefixes for:

Enable server-side encryption, object versioning where required, lifecycle rules from the signed retention policy, and deny public access. Workloads receive separate least-privilege credentials for objects, exports, evidence, and backup operations.

Customer storage administrators can create the required buckets through their normal platform tooling or through the bounded Levy preparation command:

levy-sitecheck object-storage prepare \
  --site /etc/levy/site.yaml \
  --credential-ref customer-secrets://levy/object-storage/provisioner \
  --create-required-buckets \
  --apply-retention-policy \
  --output /var/lib/levy/evidence/object-storage-provisioning.json

The preparation command changes only the exact buckets and lifecycle rules named in the signed site file. Production workload credentials remain separate from the provisioning credential.

Verify the service:

levy-sitecheck object-storage verify \
  --site /etc/levy/site.yaml \
  --catalog /opt/levy/packages/levy-core-onprem-1.0.0/conformance/object-storage \
  --output /var/lib/levy/evidence/object-storage-readiness.json

The test creates, reads, ranges, signs, expires, versions, and removes synthetic objects in the dedicated conformance prefix. It never reads an existing customer object.

15. Prepare certificates and secrets

15.1 Certificates

The customer PKI creates:

• public TLS certificates for rider API, operator web, provider callbacks, and IoT endpoints;
• internal certificates for PostgreSQL, object storage, registry, OIDC, monitoring, and controller traffic;
• an mTLS client certificate for each time-bounded installer identity;
• workload certificates for payment, messaging, and IoT adapters when required; and
• separate offline keys for release custody, customer approval, backup recovery, and evidence countersigning.

Private keys never appear in the Levy package or site YAML.

15.2 Secret references

Create the entries named by site.yaml before running installation. A typical production site uses:

customer-secrets://levy/database/app
customer-secrets://levy/database/worker
customer-secrets://levy/database/migrate
customer-secrets://levy/object-storage/objects
customer-secrets://levy/object-storage/evidence
customer-secrets://levy/identity/operator-oidc
customer-secrets://levy/identity/rider-token-key
customer-secrets://levy/payment/provider
customer-secrets://levy/messaging/provider
customer-secrets://levy/iot/gateway
customer-secrets://levy/backup/database
customer-secrets://levy/alerts/route

Verify that references resolve without displaying their values:

levyctl secrets verify \
  --site /etc/levy/site.yaml \
  --redact-values \
  --receipt /var/lib/levy/evidence/secret-readiness.json

16. Configure identity

16.1 Operator authentication

The operator web application uses the customer's OIDC provider. Create a confidential OIDC client with:

• redirect URI https://fleet.fleet.example.eg/auth/callback;
• post-logout URI https://fleet.fleet.example.eg/;
• authorization-code flow with PKCE;
• customer groups mapped to Levy roles;
• MFA enforced by the customer identity provider; and
• short access-token and bounded refresh-token lifetime.

If the customer does not operate OIDC, install the Keycloak image included in the dependency bundle and import bootstrap/identity/levy-realm.json. Keycloak remains customer-operated and stores its data in a separate local database.

16.2 Rider authentication

The Levy local identity service stores rider identity and session data in the local Levy database. OTP delivery uses the customer's messaging adapter. No Supabase authentication or Levy-hosted identity service is used.

Run identity conformance:

levy-sitecheck identity verify \
  --site /etc/levy/site.yaml \
  --output /var/lib/levy/evidence/identity-readiness.json

The result verifies issuer, audience, redirect URIs, role claims, MFA signal, signing-key rotation, OTP submission, rate limits, expiration, and logout.

17. Configure payment and messaging gateways

Levy Core does not require Stripe or Twilio. Each customer installs or selects an in-country provider adapter implementing the signed Levy contract.

17.1 Payment adapter inputs

The customer supplies:

• base URL and mTLS/CA configuration;
• supported currency and minor-unit rules;
• authorization, capture, void, refund, and status capabilities;
• webhook URL and signature verification method;
• timeout, retry, idempotency, and reconciliation policy;
• sandbox credentials and production credential references; and
• settlement file or reconciliation API details.

The provider callback URL is normally:

https://api.fleet.example.eg/providers/payment/v1/webhook

17.2 Messaging adapter inputs

The customer supplies:

• provider endpoint, sender identity, and credential reference;
• approved OTP and transactional templates;
• delivery-receipt callback and signature verification;
• country code normalization;
• throughput and rate limits; and
• timeout, retry, suppression, and expiry behavior.

The messaging callback URL is normally:

https://api.fleet.example.eg/providers/messaging/v1/webhook

Both adapters are tested in sandbox before installation and again through the installed application before launch.

17.3 Maps and optional customer support

The rider and operator clients can use a customer-hosted tile, geocoding, and routing service or an explicitly approved in-country map provider. The site file records exact endpoints, CA identities, attribution, caching, and egress rules. Map access never falls back to an undeclared public provider.

The external AI support bridge is a separate optional adapter. When enabled, it receives only customer-approved support fields through a customer-controlled egress proxy. It is not used for rentals, GPS ingestion, payments, messaging, identity, device commands, or other core application processing.

18. Configure IoT, GPS, and client applications

18.1 IoT and GPS gateway

For each enabled vehicle/device family, the site packet contains:

• protocol and firmware family;
• listener protocol, port, TLS mode, and carrier/APN restrictions;
• device identifier and authentication method;
• telemetry fields, units, timestamps, and GPS normalization rules;
• heartbeat and offline thresholds;
• lock, unlock, locate, alarm, and configuration commands;
• acknowledgement, retry, timeout, and uncertain-state rules;
• safe behavior during database, provider, or network failure; and
• a canary device cohort.

The customer creates the IoT DNS record and load-balancer listener before installation. Devices are not repointed until Phase 26.

Validate the device manifest:

levy-sitecheck iot validate \
  --site /etc/levy/site.yaml \
  --device-manifest /opt/levy/site/device-manifests/enabled-devices.yaml \
  --output /var/lib/levy/evidence/iot-readiness.json

18.2 Rider mobile and operator clients

The release package includes signed Android and iOS rider applications configured for the customer package identifiers and local API, identity, map, support, and update endpoints. The package can also include an operator mobile application or kiosk build when purchased. The browser-based operator application is deployed automatically with the server runtime.

Customer mobile-release owners distribute the signed Android App Bundle/APK and iOS IPA through the customer's selected store, enterprise program, or MDM. The installer never receives mobile signing keys.

Verify the mobile release before distribution:

levyctl mobile verify \
  --manifest /opt/levy/site/mobile/mobile-release.json \
  --release-index /opt/levy/packages/levy-core-onprem-1.0.0/manifests/release-index.json \
  --api-origin https://api.fleet.example.eg \
  --identity-origin https://identity.fleet.example.eg \
  --receipt /var/lib/levy/evidence/mobile-release.json

The check verifies application identifiers, version/build numbers, signing identities, API and deep-link domains, certificate pins, supported OS versions, locale/RTL assets, map configuration, update policy, and absence of Levy hosted-production endpoints. Staff devices install the verified build before the golden rental journey.

19. Install the Levy installation controller

This is the privileged bootstrap that was missing from the earlier documentation. The customer platform administrator performs it before the certified installer receives access.

Create the namespaces:

kubectl create namespace levy-install-system
kubectl create namespace levy-prod
kubectl label namespace levy-prod \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

Bootstrap the controller from the verified package:

levyctl controller bootstrap \
  --package /opt/levy/packages/levy-core-onprem-1.0.0 \
  --site /etc/levy/site.yaml \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  --namespace levy-install-system \
  --target-namespace levy-prod \
  --registry registry.fleet.example.eg/levy \
  --registry-credential-ref customer-secrets://levy/registry-pull \
  --receipt /var/lib/levy/evidence/controller-bootstrap.json

This command installs, by signed digest:

• the LevyInstallationRequest custom resource definition;
• the installation controller Deployment and Service;
• controller and workload service accounts;
• the customer target policy and release allowlist;
• signature and digest admission policies;
• default-deny network policies and exact required flows;
• the controller HTTPS API on TCP 9443;
• audit routing and typed redacted status; and
• evidence-repository integration.

Verify the controller:

levyctl controller verify \
  --endpoint https://controller.fleet.internal:9443 \
  --expected-release 1.0.0 \
  --receipt /var/lib/levy/evidence/controller-readiness.json

Expected result:

PASS controller.readiness
controller: 1.0.0
api: install.levy.example/v1
target namespace: levy-prod
registry access: verified
evidence repository: verified
open cases: 0

Customer security then provisions the installer with a short-lived OIDC identity and mTLS certificate scoped to the controller endpoint and one installation case. The installer receives no kubeconfig.

20. Create the site configuration

Install the verified customer site file extracted in Section 10. If release engineering is preparing a brand-new customer package, it starts from the matching profile example and replaces every value before customer and Levy signing:

sudo install -m 0640 \
  /opt/levy/site/site.yaml \
  /etc/levy/site.yaml

The file contains non-secret configuration only. Its major sections are:

apiVersion: install.levy.example/v1
kind: LevySite
metadata:
  customerId: cust_egypt_fleet_01
  environmentId: env_prod_cairo_01
spec:
  profile: production_ha
  country: EG
  timezone: Africa/Cairo
  currency: EGP
  runtime:
    namespace: levy-prod
    registry: registry.fleet.example.eg/levy
  domains:
    api: api.fleet.example.eg
    operator: fleet.fleet.example.eg
    iot: iot.fleet.example.eg
  database:
    endpoint: postgres.fleet.internal:5432
    database: levy_core
    appCredentialRef: customer-secrets://levy/database/app
    workerCredentialRef: customer-secrets://levy/database/worker
    migrationCredentialRef: customer-secrets://levy/database/migrate
  objectStorage:
    endpoint: https://objects.fleet.internal
    credentialRef: customer-secrets://levy/object-storage/objects
  identity:
    operatorIssuer: https://identity.fleet.example.eg/realms/fleet
    operatorClientId: levy-operator
    riderTokenKeyRef: customer-secrets://levy/identity/rider-token-key
  providers:
    paymentManifest: /opt/levy/site/provider-manifests/payment.yaml
    messagingManifest: /opt/levy/site/provider-manifests/messaging.yaml
  iot:
    deviceManifest: /opt/levy/site/device-manifests/enabled-devices.yaml
  observability:
    metricsEndpoint: https://prometheus.fleet.internal
    logsEndpoint: https://loki.fleet.internal
    alertRouteRef: customer-secrets://levy/alerts/route
  backup:
    policy: production-standard
    credentialRef: customer-secrets://levy/backup/database

Validate and sign it:

levyctl site validate \
  --file /etc/levy/site.yaml \
  --release-index /opt/levy/packages/levy-core-onprem-1.0.0/manifests/release-index.json \
  --strict \
  --receipt /var/lib/levy/evidence/site-validation.json

levyctl site digest --file /etc/levy/site.yaml

Customer security and Levy approve the printed digest. Any later site-file change creates a new digest and requires new approvals.

21. Run full preflight

The certified installer connects only to the controller API:

export LEVY_CONTROLLER="https://controller.fleet.internal:9443"
export LEVY_SITE="/etc/levy/site.yaml"
export LEVY_RELEASE="1.0.0"

levyctl auth login \
  --controller "$LEVY_CONTROLLER" \
  --oidc-issuer https://identity.fleet.example.eg/realms/fleet \
  --client-cert customer-identity://installer/current

levyctl case open \
  --site "$LEVY_SITE" \
  --release "$LEVY_RELEASE" \
  --evidence /var/lib/levy/evidence

The second command returns the case identifier used by later commands:

case: ins_01J8C7F3M4V7E0AN2XQ9
status: OPEN
site digest: sha256:...
release digest: sha256:...

Run preflight:

levyctl preflight run \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --site "$LEVY_SITE" \
  --release "$LEVY_RELEASE" \
  --receipt /var/lib/levy/evidence/preflight.json

Preflight verifies:

• package signatures, revocation state, entitlement, and imported image digests;
• Kubernetes version, nodes, failure domains, quotas, storage classes, ingress, DNS, NTP, and policy enforcement;
• registry immutability and rollback-digest retention;
• PostgreSQL/PostGIS, roles, TLS, performance, replication, backup, and PITR;
• object-storage buckets, encryption, lifecycle, and permissions;
• all certificate chains, SANs, expiry thresholds, and mTLS identities;
• every secret reference without returning values;
• operator OIDC and rider identity configuration;
• payment, messaging, maps, and optional provider manifests;
• IoT listeners, device manifests, commands, and canary cohort;
• monitoring, logging, alert routing, evidence storage, and support controls; and
• the exact egress allowlist with no Levy cloud, Supabase, Stripe, or Twilio dependency.

Required final output:

PASS installation.preflight
checks: 214 passed, 0 failed, 0 skipped
readyForPlan: true

22. Generate the installation plan

Generate a deterministic, redacted plan:

levyctl install plan \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --site /etc/levy/site.yaml \
  --release 1.0.0 \
  --out /var/lib/levy/install-plan.json \
  --receipt /var/lib/levy/evidence/install-plan.json

The plan lists component digests, namespaces, database migration range, resources, DNS names, providers, policies, jobs, and checks without exposing image layers or secret values.

Expected summary:

PLAN installation
release: 1.0.0
namespace: levy-prod
components: 18 create, 0 update, 0 remove
database: empty -> core_onprem_1_0
network policies: 14 create
scheduled jobs: 11 create, disabled until verification
estimated duration: 24 minutes
approval required: customer change approval + Levy release approval

Customer platform, database, security, provider, and launch owners review the plan and issue the change approval identifier.

23. Install the Levy application

Create a pre-install recovery point:

levyctl recovery create \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --type pre-install \
  --receipt /var/lib/levy/evidence/pre-install-recovery.json

Apply the approved plan:

export LEVY_CHANGE_APPROVAL="apr_customer_01J8C9D2"

levyctl install apply \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --plan /var/lib/levy/install-plan.json \
  --approval "$LEVY_CHANGE_APPROVAL" \
  --receipt /var/lib/levy/evidence/install-apply.json

Watch progress:

levyctl install status \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --watch

The controller performs these operations in order:

1. re-verifies approvals, release digests, entitlement, registry, and recovery point;
2. creates workload service accounts, quotas, and network policies;
3. creates only secret references and CSI mounts, never secret values;
4. runs the database baseline and forward migrations under the migration role;
5. deploys rider identity, core API, operator web, worker, scheduler, outbox, and reconciliation services;
6. deploys payment, messaging, object-storage, and enabled optional adapters;
7. deploys enabled IoT/GPS listeners, normalizers, state processors, and command routers;
8. deploys metrics, dashboards, alerts, synthetic checks, and backup jobs;
9. configures ingress and readiness probes; and
10. leaves external traffic, production payment, device commands, and scheduled jobs gated until their acceptance phases pass.

Successful status ends with:

INSTALLATION SUCCEEDED
release: 1.0.0
database baseline: core_onprem_1_0
workloads ready: 18/18
ingress ready: 3/3
undeclared egress: 0
next: levyctl verify core

24. Services installed by the controller

The exact list comes from manifests/image-index.json and the signed feature manifest. Disabled modules are not deployed.

25. Verify core readiness

levyctl verify core \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --receipt /var/lib/levy/evidence/core-readiness.json

This verifies:

• every workload is available and uses the approved image digest;
• API and operator health endpoints are reachable through customer ingress;
• the database schema checksum matches the release manifest;
• application, worker, and migration roles have only their intended grants;
• object storage supports signed upload, read, and lifecycle behavior;
• queues, leases, retries, idempotency keys, and dead-letter handling work;
• default-deny network policy blocks undeclared destinations;
• metrics, structured logs, traces, and alerts arrive locally; and
• no source map, build credential, debug route, shell utility, or prohibited cloud endpoint is present in the runtime profile.

Required result: PASS core.readiness.

26. Create the first administrator and sign in

Map the first customer OIDC subject to the platform_admin role:

levyctl identity bootstrap-admin \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --oidc-subject "oidc:customer-idp:12345678" \
  --email "fleet-admin@example.eg" \
  --role platform_admin \
  --approval "$LEVY_CHANGE_APPROVAL" \
  --receipt /var/lib/levy/evidence/first-admin.json

The administrator opens:

https://fleet.fleet.example.eg

They authenticate through the customer identity provider and verify:

• the correct customer and environment banner;
• Arabic/English locale and Africa/Cairo timezone;
• EGP currency formatting;
• role and permission mapping;
• fleet, vehicle, rider, ride, map, payment, messaging, and settings navigation; and
• audit recording of the login and role assignment.

No local default password is created.

27. Run integration and rental acceptance

27.1 Provider tests

levyctl verify payment \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --mode sandbox \
  --receipt /var/lib/levy/evidence/payment-sandbox.json

levyctl verify messaging \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --mode sandbox \
  --receipt /var/lib/levy/evidence/messaging-sandbox.json

levyctl verify identity \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --receipt /var/lib/levy/evidence/identity-acceptance.json

Payment verification covers authorization/sale, capture, void, refund, duplicate submission, timeout, webhook signature, settlement record, and reconciliation. Messaging verification covers OTP, transactional delivery, callback signature, expiry, duplicate callback, suppression, and rate limiting.

27.2 IoT canary

Point only the approved canary devices at iot.fleet.example.eg, then run:

levyctl verify iot-canary \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --cohort cairo-canary-01 \
  --receipt /var/lib/levy/evidence/iot-canary.json

The test verifies connection, authentication, heartbeat, GPS, battery, lock status, unlock, lock, locate, duplicate command handling, timeout, uncertain-state reconciliation, and safe ride-end behavior.

27.3 Golden rental journey

levyctl verify golden-journeys \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --catalog standard-micromobility-v1 \
  --fixture install-fixture-cairo-01 \
  --receipt /var/lib/levy/evidence/golden-journeys.json

The catalog performs:

1. rider registration and OTP;
2. payment-method token handoff;
3. vehicle discovery and availability;
4. reservation, rental start, unlock, and active-ride telemetry;
5. route and GPS display;
6. ride end, lock confirmation, pricing, and receipt;
7. wallet/payment ledger and provider reconciliation;
8. operator lookup, audit, adjustment authorization, and export; and
9. retention and deletion behavior for synthetic data.

Production credentials or real monetary value are used only under a separate signed staff-only authorization.

28. Enable schedules and production traffic

After core, provider, IoT, recovery, observability, residency, and golden-journey checks pass:

levyctl jobs enable \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --catalog production-jobs-v1 \
  --approval "$LEVY_CHANGE_APPROVAL" \
  --receipt /var/lib/levy/evidence/jobs-enabled.json

levyctl rollout advance \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --cohort staff-and-canary \
  --approval "$LEVY_CHANGE_APPROVAL" \
  --receipt /var/lib/levy/evidence/rollout-staff.json

The normal rollout is:

1. staff riders and simulator devices;
2. one small physical device cohort;
3. one operating zone;
4. a limited rider cohort; and
5. the approved production fleet.

Each wave remains observable for the duration in the site packet before the launch authority approves the next wave.

29. Backup and disaster-recovery acceptance

Before final handoff, prove all of the following:

• PostgreSQL base backup and continuous WAL archiving complete successfully;
• object-storage configuration and required bucket metadata are backed up;
• Kubernetes configuration, policies, and controller state are backed up;
• encryption and recovery keys remain under customer custody;
• backups are stored in a second approved failure domain inside the residency boundary; and
• a restore completes in the isolated recovery environment within the declared RPO and RTO.

Run:

levyctl verify backup \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --receipt /var/lib/levy/evidence/backup.json

levyctl verify restore \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --recovery-environment env_recovery_cairo_01 \
  --receipt /var/lib/levy/evidence/restore.json

The restore check validates schema, row counts, financial-ledger balance, object references, device state, jobs, and application startup without connecting the recovery environment to production devices or providers.

30. Observability and operations acceptance

The customer operations team verifies dashboards and alerts for:

• API availability, latency, error rate, and saturation;
• active rides and ride-start/ride-end failures;
• IoT connections, telemetry lag, command acknowledgement, and uncertain state;
• payment authorization, webhook, reconciliation, and settlement variance;
• messaging and OTP delivery;
• worker queues, retries, dead letters, leases, and schedule delay;
• database connections, replication lag, locks, disk, WAL, and backup age;
• object-storage capacity, errors, and lifecycle failures;
• certificate and entitlement expiry;
• undeclared network egress or denied policy attempts; and
• backup and restore freshness.

Run the alert catalog:

levyctl verify operations \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --catalog production-operations-v1 \
  --receipt /var/lib/levy/evidence/operations.json

The test generates synthetic conditions and requires the local on-call route to acknowledge each alert. It does not send telemetry to Levy unless the customer later enables an explicit support session.

31. Residency and security acceptance

levyctl verify residency \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --duration 30m \
  --receipt /var/lib/levy/evidence/residency.json

levyctl verify security-profile \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --receipt /var/lib/levy/evidence/security-profile.json

Acceptance requires:

• all operational data, logs, metrics, traces, objects, evidence, and backups remain in the declared local boundary;
• zero undeclared DNS or network destinations;
• no mandatory Levy cloud connection;
• external AI support integration remains disabled unless explicitly approved;
• the installer cannot access Kubernetes, registry layers, database rows, secrets, nodes, or raw logs;
• runtime images match signed digests and exclude original source, source maps, build caches, repository metadata, and build credentials;
• workload and administrator access is auditable; and
• temporary installation credentials have a bounded expiration.

32. Final handoff

Dispose of or quarantine installation fixtures according to the customer-approved policy:

levyctl fixtures dispose \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --fixture install-fixture-cairo-01 \
  --policy delete-synthetic-retain-financial-receipts \
  --receipt /var/lib/levy/evidence/fixture-disposition.json

Assemble and verify the acceptance record:

levyctl acceptance assemble \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --evidence /var/lib/levy/evidence \
  --out /var/lib/levy/acceptance.json

levyctl acceptance verify \
  --receipt /var/lib/levy/acceptance.json \
  --strict

Final output:

PASS installation.acceptance
customer: cust_egypt_fleet_01
environment: env_prod_cairo_01
release: 1.0.0
receipts: complete
exceptions: 0
installer access: ready for revocation
production handoff: APPROVED

Customer security then revokes the installer OIDC session and mTLS certificate:

levyctl access close-installer \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --receipt /var/lib/levy/evidence/installer-access-closed.json

The customer receives the as-built site file, release and image digests, database manifest, DNS and certificate inventory, provider/device manifests, dashboards, alert routes, backup/restore evidence, runbooks, responsibility matrix, exceptions, and signed final acceptance record.

33. Routine operations after installation

Daily

• Review active alerts, failed safety/revenue jobs, database replication, backup age, IoT uncertainty, and payment variance.
• Confirm scheduled reconciliation completed with no unexplained balance.
• Confirm the local synthetic rental and provider checks passed.

Weekly

• Review capacity, queue growth, GPS volume, object growth, certificate horizon, and denied egress.
• Review privileged access and support-session audit records.
• Verify a sampled backup and evidence-repository signature.

Monthly

• Reconcile provider settlement and wallet ledger.
• Test one selected recovery procedure.
• Review OS, platform, database, identity, registry, and Levy security advisories.
• Recalculate 90-day CPU, memory, storage, IOPS, and network capacity.

Quarterly

• Complete an isolated restore.
• Exercise IoT gateway failover and payment/messaging provider degradation.
• Rotate due certificates and secrets.
• Re-certify the current installer/controller access policy.

34. Upgrade procedure

Receive, verify, and import the new release exactly as described in Sections 10–12. Then:

levyctl upgrade preflight \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --target-release 1.1.0 \
  --receipt /var/lib/levy/evidence/upgrade-preflight-1.1.0.json

levyctl upgrade plan \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --target-release 1.1.0 \
  --out /var/lib/levy/upgrade-plan-1.1.0.json

levyctl recovery create \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --type pre-upgrade \
  --receipt /var/lib/levy/evidence/pre-upgrade-recovery-1.1.0.json

levyctl upgrade apply \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --plan /var/lib/levy/upgrade-plan-1.1.0.json \
  --approval "$LEVY_CHANGE_APPROVAL"

levyctl upgrade status --case ins_01J8C7F3M4V7E0AN2XQ9 --watch

The controller runs compatibility checks, expand-and-contract migrations, canary workloads, core and integration verification, and the release-specific acceptance catalog. The previous release digests remain immutable until the rollback window closes.

35. Rollback and failed installation

The controller stops automatically on a failed signature, prerequisite, migration, workload, provider, safety, financial, security, residency, or recovery gate.

View the typed failure without raw cluster access:

levyctl operation status \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --redacted

Rollback is permitted only when the release manifest declares the database and application pair rollback-compatible:

levyctl rollback plan \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --to-release 1.0.0 \
  --out /var/lib/levy/rollback-plan.json

levyctl rollback apply \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --plan /var/lib/levy/rollback-plan.json \
  --approval "$LEVY_CHANGE_APPROVAL"

If the migration is forward-only, the controller preserves the failed environment, stops new ride starts at the signed safety boundary, maintains ride-end/lock/refund/reconciliation operations, and restores into the isolated recovery environment before customer-authorized cutover.

36. Uninstall and data disposition

Uninstallation is a customer data-governance operation, not a raw namespace deletion.

First create and verify the final export and backup:

levyctl uninstall plan \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --data-policy customer-retain-7-years \
  --out /var/lib/levy/uninstall-plan.json

levyctl uninstall apply \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --plan /var/lib/levy/uninstall-plan.json \
  --approval "$LEVY_CHANGE_APPROVAL" \
  --receipt /var/lib/levy/evidence/uninstall.json

The plan separately addresses:

• ending or transferring active rides safely;
• disabling new rentals and device commands;
• payment settlement, refund, wallet, and reconciliation completion;
• database and object export;
• legal retention and deletion schedules;
• backup and evidence retention;
• device endpoint and credential rotation;
• DNS and certificate retirement;
• registry image retention or deletion; and
• removal of Levy workloads, controller, policies, identities, and secrets.

The customer DBA and data owner execute final database/object deletion only after the signed retention period and approval. The installer never deletes customer data directly.

37. Troubleshooting

Create an encrypted support bundle when Levy assistance is approved:

levyctl support bundle \
  --case ins_01J8C7F3M4V7E0AN2XQ9 \
  --redact customer-standard \
  --encrypt-to /etc/levy/support/levy-support-public.pem \
  --out /var/lib/levy/support/support-bundle.enc

The bundle contains typed component status, digests, check results, redacted configuration structure, and synthetic diagnostics. It excludes secret values, raw consumer records, full database output, image layers, source, and unrestricted logs.

38. Complete installation checklist

An installation is complete only when every line below is true.

Package and platform

• Release archive checksum and signature verified.
• Levy trust root independently confirmed.
• levyctl, levy-sitecheck, and levy-custody versions verified.
• Every required OCI image imported by digest into an immutable local registry.
• Platform conformance passes with zero failed or skipped mandatory checks.
• Capacity profile and failure-domain placement approved.

Data and security

• PostgreSQL/PostGIS, roles, TLS, replication, backup, and PITR pass.
• Object-storage encryption, lifecycle, and access tests pass.
• All certificates and secret references pass without secret disclosure.
• Default-deny networking and exact egress manifest pass.
• Installer has controller-only access and no Kubernetes, registry, database, node, or secret access.

Application and integrations

• Installation controller is healthy and version-compatible.
• Installation plan is approved and applied successfully.
• All expected application services are ready on signed digests.
• Database schema checksum matches the release.
• Operator login and first administrator work through customer OIDC.
• Rider OTP and local session management pass.
• Payment and messaging sandbox conformance pass.
• IoT/GPS canary and command-safety tests pass.
• Golden rental journeys and financial reconciliation pass.

Operations and handoff

• Local dashboards, logs, traces, alerts, and on-call routing pass.
• Backup and isolated restore meet RPO/RTO.
• Residency capture shows zero undeclared egress.
• Controlled rollout completes through the approved production cohort.
• Synthetic installation fixtures are disposed according to policy.
• Final acceptance receipt is complete and signed.
• Installer identity, certificate, session, and temporary media access are revoked.
• Customer operations owns the as-built record and all runbooks.

39. Installer handoff summary

A developer or infrastructure engineer evaluating this deployment should expect the following work:

1. provision and secure a normal production Linux/Kubernetes environment;
2. provide PostgreSQL/PostGIS, object storage, registry, identity, secrets, ingress, monitoring, and backup services;
3. receive and verify one signed Levy release archive plus one customer site package;
4. install three compiled Levy command-line tools;
5. import approximately twenty signed OCI images into the local registry;
6. bootstrap one restricted installation controller;
7. populate one non-secret site YAML and customer-owned secret references;
8. run preflight, plan, apply, and status commands through levyctl;
9. configure and verify local payment, messaging, identity, and device integrations;
10. complete rental, financial, safety, recovery, observability, residency, and security tests; and
11. sign the acceptance record and revoke installer access.

No source checkout or customer-specific application build occurs at the installation site. Customer variation is configuration and certified adapters; the installed Levy application is the same signed, versioned product release used across on-premise customers.