OEM Verification

Collection counts as done when we have high confidence in it, not when it is running. This page defines that bar, tier by tier, and the procedure for reaching it — for new OEMs and for OEMs already collecting alike.

Raw data cannot be re-collected

The public APIs only show what is on the market right now. If today's collection is wrong, today's data is lost for good. Everything downstream — transformed, consolidated, derived — can always be backfilled from a properly-collected raw tier. Raw collection therefore comes first for every OEM; downstream depth is scheduled by consumer need.

Key concepts

Verification uses gates and asset checks, and records results as artifacts. They differ by when they run and whether their result is persisted:

• Gates — deterministic pass/fail validations run while a skill or workflow executes. A gate blocks the run when it fails. Two kinds:

  • Structural gates check code and config shape. Current per-tier validators included in the table below.
  • Data-quality gates — EDA (exploratory data analysis) profiles the collected data and fails the run on any 🔴 finding.

  Gates are not wired into the DAG; they fire when we run discovery or implementation, not on every materialization, though rerunning certain skills on a regular cadence is worth pursuing for additional confidence.

• Asset checks — validations wired into the Dagster DAG that run automatically after every materialization (fetch_attempts, unresolved_refs, …; see Design → Asset Checks). Most are non-blocking warnings. A check that fires for a known, deferrable reason can be quieted via the exception config.

• Artifacts — Committed, machine-readable records of a gate or check failure, so a later fix can be validated deterministically against it. The discovery contract gate (discovery-contract.gate.json) is the canonical example.

• Exception config — a shared, committed config recording a deliberately-quieted check or accepted gap (reason + scope). Both the asset checks and any relevant gates read it, so an exception is declared once and honored everywhere.

Definition of done

Every tier clears three bars — one per column of the table below:

• Automated gates pass — the tier's gates and EDA run green. They encode failure modes we have already seen, keeping past classes of issues from recurring.
• Human review is complete — a person finds the issues we have not seen before: working the APIs directly, stress-testing the agent's conclusions and inherited assumptions, and checking the collected data against reality.
• The refinement loop closes — each new finding becomes a new gate or asset check, so the same class of issue is caught automatically next time. The automated share grows with every OEM we take through this; the manual burden shrinks.

Clearing all three gives us high confidence that discovery was performed correctly and that the raw sources it uncovered are comprehensive enough to build a useful history.

Tier	Automated bar	Human bar	Refinement loop
Discovery	Contract probes green against the live API (pagination, response shape, enumerators, freshness, identity keys) per check_contract_gate.py	Engineer works the APIs directly: stress-tests the reported findings and inherited assumptions, probes for endpoints or entities that were missed	Failures and misses feed back into the discovery skills and new deterministic gates
Raw	EDA + Structural gates (eg, check_hardcoded_fanout.sh) pass; Runs clean on branch deployment: no runtime errors, stable record counts, no unquieted asset-check failures	Collected data reviewed against the live site: totals, spot-checked records, field semantics	Findings feed back into the raw implementation, the EDA skills, and new asset checks
Transformed	EDA + Structural gates (eg, validate_registry.py) pass	Asset checks triaged; extracted-field semantics and dedup/tiebreak policies reviewed	Findings feed back into the transformed implementation, the EDA skills, and new asset checks
Consolidated	EDA + Structural gates (eg, validate_yaml_refs.py) pass	Join/resolution rates reviewed; identity keys confirmed in the data	Findings feed back into the consolidated implementation, the EDA skills, and new asset checks

The procedure

One procedure covers new onboarding and existing-OEM review. For OEMs already collecting, completed steps are skipped and recent production runs stand in for the branch deployment.

Phase 1 — raw public collection

1. Discovery — the public-api-discovery workflow: explore → document (Postman) → live contract probes → contract gate. Existing OEMs: re-run with current tooling — it cheaply catches everything the gates already know how to catch.
2. Hands-on vetting — an engineer works the APIs directly: validates the reported findings, stress-tests inherited assumptions (e.g. filters copied from the frontend that were never confirmed droppable), and looks for what was missed. This is where new kinds of issues get found; the contract-gate approval is the floor, not the whole job.
3. Implementation — the public-api-implement workflow: gated build of the raw tier plus provisional transformed and consolidated tiers. Fixtures come from captured live samples. The fetch layer raises on empty, blocked, or partial responses rather than continuing.
4. Live-run validation — 3 consecutive clean daily runs with stable counts and live-site parity. Recurring runtime failures (OOMs, timeouts, bot-blocks) block raw verification: a raw run that dies is a missed day, and missed days are permanent. Existing OEMs: recent production runs stand in for the branch deployment.
5. Data review — counts and fields against the live site and any independent reference (legacy BigQuery where it exists). Findings become new EDA rules and asset checks.
6. Check triage — anything quieted is recorded as an exception in the shared config (reason + scope), which the daily asset checks also consume.
7. Wrap-up — review + merge, with the ledger entry part of the PR. Existing OEMs: record the trusted-history-since date and any permanent gaps in the ledger instead.

Phase 2 — public downstream (scheduled per consumer)

1. Asset checks quieted in Phase 1 are resolved — the underlying issue is fixed and its exception removed from the config.
2. Consolidated-tier identity and FK resolution is fine-tuned.
3. Derived-tier work is completed as consumers need it.

Phase 3 — private data source integration

1. Private-system upstreams are already available in BigQuery (current workflow).
2. Data shape is assessed and differently named but equivalent fields are correlated.
3. Public data is integrated via consolidation, adjusting entity definitions as needed.

Trusted history

Each review records a trusted-history-since date: the date from which the accumulated raw history is known good. Where earlier history has known gaps, that is recorded rather than silently consumed downstream.

Alert triage

Alerts come from failed or erroring runs, failing asset checks, and freshness alarms. Every alert is classified as exactly one of:

1. Raw collection at risk — wrong or missing data, including raw runs that fail or error out and transformed-tier checks signalling missing or invalid upstream data. A missed day is a permanent gap, so these are top priority, further ordered by OEM.
2. Miscalibrated check or threshold — fix the check. Three or more spurious fires means the check itself is the bug; the added noise drowns out real alerts.
3. OEM-side outage — confirm the outage is genuine against the live public website to rule out category 1, then document and monitor.
4. Known transformation/consolidation issue — recoverable by rematerializing once we invest time in the fix. Quiet it with an exception in the shared config; revisit when the OEM reaches downstream-refinement priority.

See Viewing Asset Check Results for how to inspect check output.

Verification ledger

Per-OEM verification state is tracked on the team's OEM Verification Ledger Confluence page — one row per OEM, one column per tier, refreshed at milestones. The repo is the source of truth on granular progress; the ledger keeps everyone informed at a glance.