Skip to main content

Enhancement Ideas

A running list of cross-cutting ingestion enhancements we want to pursue. Each entry states the problem, the proposed approach, and why it matters. OEM-specific follow-ups belong in that OEM's source docs — this page is for platform-wide ideas.

Cross-source consistency checks​

When the same entity is collected from more than one source, those sources overlap in known, predictable ways. We should encode that expected relationship and flag deviations from it automatically, rather than discovering gaps by hand after they have already polluted downstream data.

Example: public vs. private inventory​

A public scrape and a private dealer feed cover overlapping but non-identical slices of inventory. The expected relationship looks like this:

RegionMembershipContains
Public onlyin public, not privatePost-production vehicles at non-onboarded dealers
Private onlyin private, not publicPre-production vehicles at onboarded dealers
Public ∩ Privatein both feedsPost-production vehicles at onboarded dealers

The intersection is the region we can reason about most confidently: a post-production vehicle at an onboarded dealer should appear in both feeds. A sudden, structured absence there — e.g. "the private feed is missing every vehicle of a given model code that the public feed still lists" — is a strong signal of a feed gap, not normal churn, and should raise an alert.

Relationship to the existing check​

The consolidated tier already ships a cross_source_overlap asset check that compares set membership between paired inventory feeds within configurable coverage bands (see Component Reference → ConsolidatedComponent). This idea generalizes it:

  • Beyond inventory — apply the same expected-overlap modeling to any entity collected from multiple sources (dealers, models, features), not just inventory.
  • Structured deviation detection — group the keys missing from a feed by an attribute (model code, brand, dealer) so an alert points at what dropped out, not just how many. A whole-segment dropout (an entire model code vanishing from one feed) is far more actionable than an aggregate count.
  • Declarative expectations — let each entity declare its expected cross-source relationship in YAML, so the check is configured alongside the merge spec rather than hand-written per OEM.

Discovery gates the GM rediscovery exposed​

Every item below is a defect that shipped through a successful discovery contract on GM and was only caught by hand afterwards. Each is generic — none is GM-specific — so each belongs in the discovery skills (public-api-contract-test, public-api-raw) as a deterministic gate rather than in an OEM's code. See GM discovery contract for the original findings.

1. Identifier distinctness, not just nonempty-ness​

The data-quality probe accepted headerId as GM's feature_code because it was mostly nonempty. It is a CMS section id: 91 of 108 items shared one value, so keying features on it discarded ~91% of them. Gate: any field nominated as a record identifier must be checked for distinctness within its parent group, not just presence. A near-constant identifier should fail outright.

2. Flatten doubly-nested record paths​

The same probe assessed only data.categories[0].items — the first parent — so the reuse of headerId across categories was invisible. Gate: a records path with two levels of nesting must be flattened across all parents before assessment.

3. Omitted-filter narrowing​

Omitting driveTypeId from GM's /trim does not return all drive types; it silently returns the default one. That cost 6 LYRIQ styles and would have cost more had the model list been larger. Probe: for each optional filter parameter, compare the unscoped response against the union of scoped responses. If unscoped ⊂ union, the parameter is mandatory for completeness and must be recorded as such in the contract. This is the highest-value item here — it is silent, and it under-collects.

4. Fan-out key under-enumeration​

/line advertises body types in bodyTypes[].id, but the dual-rear-wheel siblings live only in groupedIds and trimMeta[]. Reading the obvious field cost 46 styles across two brands. Gate: cross-check the fan-out key set an enumerator yields against every key the downstream response references. A downstream response naming a key the enumerator never produced is a fan-out gap.

5. Suppressed errors erase the work list​

A SUPPRESS error handler emits no event, so events_to_raw_fetch_result writes no raw row. Any downstream asset deriving its work list from that asset silently drops those items. GM's LYRIQ vanished from /trim entirely, and the "fallback when a model has no body types" branch could never fire because the model was not in the frame. Gate: an asset whose work list comes from an upstream that can suppress must either source that list from the enumerator instead, or assert raw_row_count == expected_request_count.

6. Derived identifiers where an authoritative one exists​

GM trim codes were being reverse-engineered from image URLs with a regex, producing TK30953D_3SA where the authoritative trims[].trimCode said TK30953_3SA — minting phantom model codes that then failed downstream. Gate: flag any identifier produced by string/URL parsing when the same entity exposes an authoritative field for it. (Parsing is legitimate when there is no other source — see GM inventory RPO codes — so this is a warning that demands a recorded justification, not a hard fail.)

7. FK namespace overlap​

Nothing verified that a referencing field's values were drawn from the same namespace as the target's key. RPO codes and CMS section ids coexisted in one feature_code column without complaint. Gate: when declaring a foreign key, sample both sides and require non-zero overlap. Zero overlap means the join can never resolve and should fail the contract, not merely report a 100% unresolved rate later.