Independent Installation Certification

1. What this certification proves

This protocol is the gate between “the software was built” and “the installation package can actually be handed to someone.” It proves that:

• the package contains every prerequisite, value, artifact, command, expected result, stop condition, and escalation path;
• a qualified installer can complete Installation Guide without repository, Kubernetes, registry, database, Secret, node, runtime, raw-log/event, or provider-admin access;
• the controller/CLI contracts behave exactly as documented, including failure and retry behavior;
• installation evidence is sufficient for another person to verify what happened;
• the system survives upgrade, rollback, restore, dependency outages, and access revocation;
• no hidden hosted service, foreign destination, undocumented manual correction, or Levy engineer action is required;
• the final customer operational team can assume responsibility from the handoff package.

It does not prove that every future customer/provider/device combination works. Those require release/site-specific certification under Release Package Contract. It proves that the reference product and handoff mechanism are repeatable and complete.

2. Certification types and triggers

2.1 Required certification runs

Both runs include adversarial installer-access tests, residency capture, provider/IoT simulators, fixture disposition, evidence assembly, and access/media/cache closure.

For the first released LTS with no predecessor, D4 records nMinusOneUpgrade and applicationRollback as notApplicableNoPredecessor; it still certifies greenfield online/offline, restore, adversarial denial, and handoff. The first successor release must certify N-1 upgrade and application rollback before any customer package for that successor can reach D5. Later receipts may not inherit those scopes by release name alone.

2.2 Recertification triggers

Full two-run recertification is required for:

• controller or installer API major change;
• levyctl major change;
• levy-custody encryption/decryption/import/export, signature, compatibility, or receipt-semantic change;
• controller bootstrap, CRD conversion/stored-version, controller rollback, or lifecycle-security semantic change;
• installation request/target policy/signature/canonicalization/trust model change;
• baseline database or migration framework replacement;
• new deployment/protection profile;
• new reference orchestrator/database/object/identity distribution family;
• change to installer privileges or diagnostic surface;
• change to artifact custody/encryption/signing/revocation/entitlement model;
• addition or semantic change of a cryptographic profile (certification is profile-scoped);
• change to evidence schema/counter-signing/redaction/fixture disposition;
• a severity-1 installation/certification escape.

A bounded one-run delta certification may be approved for an additive minor release when compatibility tests prove no affected lifecycle/security surface. The approval names exact changed requirements and expires at the next full LTS certification.

3. Independence and personnel rules

3.1 Installer team

Each installer team must:

• have passed the general installer qualification and release-specific delta exam;
• not have authored the controller, levyctl, release schemas, Helm chart, baseline/migrations, or Installation Guide steps being tested;
• not have participated in the immediately prior certification run;
• receive only the same package/access a real third party would receive;
• record every ambiguity, help request, undocumented action, and elapsed blocked interval.

3.2 Observer team

Observers include Levy QA, release engineering, security, customer-role actors, and a certification lead. They:

• provision only actions assigned to their roles in the package;
• do not volunteer hints or operate installer steps;
• record protocol deviations and system evidence;
• may stop immediately for safety, security, money, residency, data-loss, or evidence-integrity risk;
• classify assistance and defects without changing results during the run.

3.3 Customer-role simulation

Different named people or isolated test identities perform customer platform, security, DBA, provider, operations, and launch roles. The installer cannot possess or share those identities. Two-person approvals must be genuinely separate signatures.

4. Clean-room and no-assistance policy

4.1 Allowed installer inputs

Before the installer session, customer roles perform Installation Guide Customer Readiness Phase −1. The installer receives only:

• the exact signed, expiring certification_candidate Release Package Contract package with handoffReady: false, a synthetic lab binding, and certification.status: pending;
• the exact compatible levyctl already independently verified/provisioned by customer security, plus its offline help and distribution receipt;
• customer-issued controller API OIDC/mTLS identity;
• approved application test accounts/devices and dashboards;
• the in-boundary evidence repository;
• documented support/escalation contact capable only of answering through the controlled assistance process.

The installer does not receive source, repository access, internal Slack/email history, unpublished architecture diagrams, developer environment files, cluster/registry/database credentials, kubeconfig, Secrets, raw logs/events, node access, image archives/decryption keys, or a Levy engineer paired at the keyboard.

4.2 Assistance classes

A narrowly defined documentation erratum is not A4 only when it changes generated README/runbook explanatory prose and documentationDigest but leaves machineContractDigest, commands, arguments, schemas, API/receipt contracts, catalogs, expected values, security controls, stop conditions, and operational semantics byte-for-byte unchanged. It is recorded as S3/S4, may be issued between runs, and requires an independent documentation check plus rerun of every affected exercise; it does not reset already passing unaffected exercises. Any doubt is treated as A4.

Search engine/public vendor documentation may be used only when Installation Guide or an included runbook explicitly routes the installer there and the content/version is captured in the package's external-reference ledger. Generative AI or private Levy knowledge is prohibited during certification because it would hide documentation incompleteness.

5. Certification lab

5.1 Minimum topology

Each run uses a new environment with:

• at least three Kubernetes control-plane nodes or a certified HA control plane;
• at least three worker nodes across declared failure domains;
• customer-local immutable registry;
• HA PostgreSQL/PostGIS plus pooler;
• S3-compatible local object storage;
• customer OIDC and local rider OTP service/simulator;
• customer secret manager/workload identity;
• local DNS/NTP/PKI/ingress/load balancing;
• local logs/metrics/alerts/audit/on-call sink;
• local evidence repository/counter-signer;
• backup/PITR target and isolated recovery capacity;
• payment/messaging simulators implementing success, decline, timeout-after-success, duplicate/out-of-order/missing callback, settlement/delivery lookup;
• IoT gateway simulator plus at least one physically supervised reference device where the release supports real hardware;
• rider/operator production-candidate clients;
• customer-owned CNI/firewall/flow metadata capture;
• no general outbound Internet path except exact destinations declared for the chosen site profile.

Run B's release media arrives through an offline encrypted archive and is imported without the lab contacting a Levy registry/key service. Customer security generates the selected crypto-profile recipient key inside the lab boundary, delivers only its public key through the authenticated certification channel, confirms the fingerprint out of band, and tests dual-custody rotation/loss handling.

5.2 Clean-environment proof

Before Phase −1, customer platform/security capture a signed clean-environment receipt proving:

• no Levy namespace/workload/CRD/controller/schema/database/object/registry artifact exists;
• no prior installer identity or evidence cache exists;
• infrastructure versions/topology match the selected supported-platform entry;
• clocks, audit, backup target, evidence repository, and default-deny policy are active;
• test providers/devices contain only the declared certification fixtures.

Snapshots/images preloaded with a successful Levy installation are prohibited.

Customer roles then complete Phase −1 and, before the installer opens the package, produce a separate readiness receipt proving Levy roots came from the authoritative out-of-package channel; levyctl, levy-custody, and levy-sitecheck were independently verified/provisioned; the release was imported; and the controller, target policy, installer API identity, evidence counter-signer, and audit route are active at their frozen digests. Both the earlier clean receipt and later readiness receipt are required so bootstrap does not erase proof that the lab began clean.

6. Candidate prerequisites

Certification starts only when:

1. The release is D3_RELEASE_QUALIFIED under Engineering and Release Reference.
2. The complete Release Package Contract certification_candidate validates with no unresolved non-template placeholder/secret/prohibited content, is synthetic/lab-bound, and cannot enable live value or D5.
3. All release scans/signatures/revocation data are fresh for the scheduled window.
4. The exact controller, levyctl, API/schema, platform, mobile, provider, IoT, catalog, and runbook versions are frozen.
5. If the candidate declares a predecessor, a clean certified N-1 release/package exists for upgrade/rollback testing; otherwise the release index and certification scope declare notApplicableNoPredecessor.
6. All known issues have severity, affected test, workaround status, owner, and release decision.
7. Test payment/message/device actions are bounded and no real external customer/consumer is involved.
8. Observers approve stop/escalation, evidence, privacy, and physical safety arrangements.

Changing any signed candidate artifact after start invalidates the run.

7. Certification execution

Exercise 0 — Package comprehension and readiness

Without oral briefing, the installer:

1. opens README-FIRST.md;
2. verifies the Phase −1 trust/tool/controller receipts and retrieves the authoritative root fingerprint through the documented out-of-package path;
3. runs package validation;
4. identifies all roles, prerequisites, windows, stop conditions, and evidence locations;
5. produces a written execution checklist using only the package;
6. correctly declares any deliberately seeded missing prerequisite before mutation.

Pass: no unexplained placeholder/conflict and every seeded blocker is detected by the documented mechanism.

Exercise 1 — Artifact custody

Run A uses customer mirror/import; Run B uses encrypted media. Customer custodians—not the installer—obtain/import/decrypt artifacts. The installer verifies typed receipts only.

Faults include altered digest, expired scan, stale revocation list, mutable tag, revoked component, extra registry artifact, missing N-1 retention lock when a predecessor exists, and installer pull attempt.

Pass: every invalid case stops with the stable result class; valid import succeeds; installer never obtains registry/archive/key access; media custody is complete.

Exercise 2 — Controller bootstrap verification and authorization

Customer roles prove the signed bootstrap/target policy installed during Phase −1, controller/API/schema/CRD versions, conversion health, and zero open mutating cases. The installer authenticates only to the controller API and executes the full Installation Guide Section 33 suite; the exercise never grants bootstrap authority to the installer.

Adversarial attempts include:

• Kubernetes discovery/read/write/log/event/CR/exec/port-forward/debug/token/impersonation;
• registry/image/archive/database/object/backup/Secret access;
• undocumented API path/method;
• free-form command, SQL, URL, path, environment, image, manifest, unknown field;
• wrong customer/environment/case/release/config/catalog/cohort;
• unsigned/partially signed/modified/expired/replayed request;
• excessive rate/body/concurrency and parallel mutations;
• raw log/event/source/image/secret/database/support-bundle retrieval;
• direct CR creation/mutation or target-policy replacement.

Pass: every attempt is denied/audited without privileged detail; allowed verbs still operate; denial lockout/recovery follows policy.

Exercise 3 — Preflight and deliberate prerequisites

The lab seeds at least these failures one at a time:

• unsupported platform version;
• insufficient CPU/memory/storage/IOPS or shared failure domain;
• DNS/NTP/certificate chain/expiry failure;
• database/PostGIS/role/TLS/schema/object/backup failure;
• missing/expired secret reference;
• wrong provider CA/endpoint/capability;
• undeclared egress or wildcard destination;
• evidence/on-call route unavailable;
• entitlement/config/release mismatch.

The installer must identify the typed owner/runbook and stop. Customer actors correct the prerequisite through documented customer procedures; the installer reruns the exact check.

Exercise 4 — Configuration, signatures, and dry run

The installer renders/populates only allowed non-secret values. The harness attempts a secret literal, hidden default, disabled module with live job, provider mismatch, unsupported currency/minor unit, wrong pause policy, unapproved host, unknown field, post-signature edit, and target substitution.

Pass: validation catches every case; the final complete spec is dual-signed; dry-run output is deterministic/redacted and matches the customer target policy.

Exercise 5 — Installation and failure recovery

The controller installs from the immutable request. The harness injects failures at:

• pre-mutation revalidation;
• recovery-point verification;
• migration lock/baseline/forward migration;
• image pull/admission;
• database/object/identity/provider readiness;
• worker/IoT deployment;
• evidence upload/counter-signing;
• controller restart and duplicate client submission.

Pass: reconciliation is idempotent, no duplicate migration/action occurs, FailedSafe receipt is complete, hold deadlines/automatic actions execute, and the documented retry/rollback/forward-fix/restore class returns the environment to the declared state.

Exercise 6 — Core, providers, jobs, IoT, and mobile

Execute every enabled release catalog:

• core readiness and tenant/auth isolation;
• payment/messaging/identity/optional gateway conformance;
• job idempotency, leader loss, retry/backlog/poison recovery;
• IoT ownership/fencing/telemetry/command/canary/failover;
• client endpoint/update/signature/version/fallback scan;
• 21 golden journeys including pause-billing and ride end/lock under dependency outage.

Pass: all expected receipts are produced; no undocumented operator action is required; safety/financial failures block progression.

Exercise 7 — Residency, security, and support boundary

Customer security captures flow metadata while the harness exercises normal and failure paths. Inject a prohibited destination, support-bridge attempt to reach core data/IoT/money, test-auth bypass, unsigned image, privileged workload, source-map artifact, and evidence/support prohibited content.

Pass: zero undeclared allowed destination, all injected violations denied/detected, installer receives only typed metadata/receipt, and customer alone decrypts the synthetic redacted support bundle.

Exercise 8 — Live-value simulation and fixture disposition

Use the certification payment provider in production-equivalent settlement mode and staff fixtures only. Execute capture/refund/webhook/settlement across three simulated cycles, then disposition every fixture.

Pass:

• zero unexplained ride/payment/wallet/provider/settlement variance;
• no non-staff cohort opens early;
• every test row/object/command has the immutable fixture ID;
• normal analytics/reports/payouts exclude synthetic fixtures;
• real-value-equivalent capture/refund/fee entries remain separately identified/reconciled;
• deletion/quarantine/retention/backup-expiry follows the signed policy.

Exercise 9 — Backup, isolated restore, and DR

Trigger encrypted backup/PITR, restore into an isolated environment under recovery_nonproduction entitlement, run schema/object/identity/synthetic verification, prove live egress/device/provider actions are disabled, then destroy/expire the recovery fixture.

The harness includes an unavailable latest backup, wrong key/reference, insufficient recovery capacity, expired entitlement, and a corrupted restore candidate.

Pass: backup alone never passes; a valid restore meets RPO/RTO for the profile; failures produce correct owner/recovery path; production remains unaffected.

Exercise 10 — N-1 upgrade and rollback

When a predecessor exists, Run B begins on the certified N-1 release or installs it before the target. Customer security adds target digests while retaining installed/N-1. For a no-predecessor first release, verify the signed notApplicableNoPredecessor declaration and omit the remaining steps; the D4 receipt records both lifecycle scopes as not applicable. Otherwise execute:

1. upgrade plan and compatibility/migration review;
2. recovery point;
3. expand migration and canary rollout;
4. controller restart/duplicate status observation;
5. target conformance/golden/residency/reconciliation;
6. application rollback while schema is compatible;
7. re-upgrade;
8. rollback-window closure and contract migration;
9. retired digest removal only after signed closure.

Inject a target digest missing from admission, N-1 image missing from registry, incompatible mobile/gateway, failed expand migration, canary regression, and rollback request without authority.

Pass: every issue stops correctly, N-1 remains available, rollback/forward-fix/restore classes match the release, and no manual tag/SQL/manifest action occurs.

Exercise 11 — Operations handoff and access closure

Customer operations, without engineering help:

• uses dashboards/alerts/runbooks for seeded API/job/DB/object/provider/IoT/cert/backup/egress/entitlement issues;
• performs one safe maintenance and one incident exercise;
• verifies RACI/SLO/RPO/RTO/escalation/support access;
• accepts as-built digests and evidence;
• revokes installer API/VPN/cert/session/dashboard access;
• re-runs API/Kubernetes/registry denial proof;
• verifies receipt cache deletion and media/workspace return/destruction;
• assembles and verifies the final acceptance index.

Pass: installer has no remaining access/material; operations can diagnose and recover through documented surfaces; all evidence is counter-signed and in-boundary.

8. Seeded-fault minimum catalog

Each run randomizes at least one fault from every class so the installer cannot memorize order:

Fault injection uses signed lab controls inaccessible to the installer and never alters a real customer/provider/device.

9. Documentation and product defect classification

Every issue records requirement ID, exercise/step, environment/release/package digest, assistance class, observed evidence, expected behavior, impact, owner, fix, regression test, and recertification scope. Raw consumer/secret data is never attached.

10. Timing and operational acceptance

Time is measured to find unusable documentation, not to force unsafe speed. Stop/approval/customer-remediation time is separated from installer active time.

Reference objectives for an infrastructure-ready site:

• package comprehension/readiness: 4 active hours;
• custody/controller/preflight/config/dry-run: 2 business days;
• recovery point/core install/readiness: 2 business days;
• provider/IoT/jobs/mobile/operations/security: 3 business days;
• golden/live-value simulation/fixture disposition: 2 business days plus settlement clock;
• restore/upgrade/rollback: 2 business days;
• handoff/evidence/access closure: 1 business day.

Missing customer prerequisites pause the clock and are not “worked around.” Exceeding an objective is not itself failure, but every delay must map to customer wait, release defect, documentation defect, tool performance, or installer error and feed the readiness report.

11. Evidence produced by certification

Each run produces an in-boundary signed index containing:

• clean-environment receipt;
• exact package/release/site/config/controller/CLI/custody-tool/sitecheck/platform/crypto-profile, machine-contract, and documentation digests;
• certification scope vector and predecessor release ID or notApplicableNoPredecessor declaration;
• installer qualifications and independence attestations;
• Customer Readiness Phase −1, Phase 0–19 receipts, and I0–I11 gate results;
• seeded-fault catalog and typed observed results;
• API/Kubernetes/registry/secret/image denial matrix;
• assistance log and defects/dispositions;
• financial/fixture/residency/security/restore/upgrade/rollback results;
• as-built and timing report;
• installer access/media/cache/workspace closure;
• installer, observer, QA, security, release, and certification-lead signatures.

Screen recordings/raw terminal logs are not certification evidence when they could expose operational data. Machine receipts and concise observer attestations are authoritative.

12. Certification decision

A run passes only when:

1. Every required exercise and Installation Guide gate passes against the same release/profile/machine-contract digest; documentation digests may differ only under a recorded compliant erratum and affected-exercise rerun.
2. No A2, A3, or A4 event occurred.
3. No open S1/S2/S3 defect remains.
4. Every seeded failure was detected, stopped, assigned, and safely recovered exactly as documented.
5. All required receipts/signatures/counter-signatures verify and contain no prohibited content.
6. Installer Kubernetes/registry/database/Secret/node/runtime/image denial holds throughout and after revocation.
7. No hidden hosted dependency or undeclared egress appears.
8. Financial, safety, fixture, backup/restore, upgrade/rollback, and lifecycle invariants pass.
9. Customer operations completes the handoff exercises without engineering intervention.
10. The environment and installer artifacts are closed according to policy.

Product certification requires two consecutive passing runs A and B on different clean environments and independent teams. A failure resets the consecutiveness requirement for the affected candidate machine-contract version; after an A4/S1/S2 correction, the new release/machine-contract digest starts a new run. A compliant documentation-only erratum follows Section 4.2 and does not invalidate unaffected completed exercises.

13. Certification receipt and validity

The final receipt states:

• certification ID, issue/expiry, release/package/controller/CLI/custody-tool/sitecheck/API/schema/platform/crypto-profile, machine-contract, and final documentation digests;
• Run A and B evidence-index digests and installer-team identities/independence;
• supported install, upgrade, rollback, restore, delivery, and protection profiles;
• explicit scope vector for greenfieldOnline, greenfieldOffline, restore, adversarialDenial, handoff, nMinusOneUpgrade, and applicationRollback, with predecessor ID or notApplicableNoPredecessor;
• known limitations and exact non-blocking exceptions;
• recertification triggers and support window;
• Levy QA, security, release, product, and certification-lead signatures.

Certification expires at the earliest of:

• declared certification/support end date;
• a full-recertification trigger in Section 2.2;
• revocation of a bound key, digest, entitlement, API/schema, controller/CLI, or platform profile;
• discovery of a certification-escaping S1 defect.

Customer package generation must verify certification validity and exact digest compatibility; it cannot rely on a release name alone.

14. Final handoff-readiness declaration

The statement “a qualified third party can install Levy Core On-Prem from the supplied instructions” may be made only when:

• the exact release/profile/machine-contract digest holds a current D4 receipt whose scope vector covers the proposed customer handoff;
• Release Package Contract produces a D5_SITE_READY package for the exact customer/environment;
• the installer is currently qualified for the release delta;
• no revocation, scan-freshness, compatibility, known-issue, or customer prerequisite blocks the scheduled installation.

The signed D4 certification and D5 customer site package provide the release- and site-specific evidence used to authorize installation.