Enhancement Ideas
A running list of cross-cutting ingestion enhancements we want to pursue but have not yet scheduled. Each entry states the problem, the proposed approach, and why it matters. OEM-specific follow-ups belong in that OEM's source docs — this page is for platform-wide ideas.
Cross-source consistency checks
When the same entity is collected from more than one source, those sources overlap in known, predictable ways. We should encode that expected relationship and flag deviations from it automatically, rather than discovering gaps by hand after they have already polluted downstream data.
Example: public vs. private inventory
A public scrape and a private dealer feed cover overlapping but non-identical slices of inventory. The expected relationship looks like this:
| Region | Membership | Contains |
|---|---|---|
| Public only | in public, not private | Post-production vehicles at non-onboarded dealers |
| Private only | in private, not public | Pre-production vehicles at onboarded dealers |
| Public ∩ Private | in both feeds | Post-production vehicles at onboarded dealers |
The intersection is the region we can reason about most confidently: a post-production vehicle at an onboarded dealer should appear in both feeds. A sudden, structured absence there — e.g. "the private feed is missing every vehicle of a given model code that the public feed still lists" — is a strong signal of a feed gap, not normal churn, and should raise an alert.
Relationship to the existing check
The consolidated tier already ships a cross_source_overlap asset check that compares
set membership between paired inventory feeds within configurable coverage bands (see
Component Reference → ConsolidatedComponent).
This idea generalizes it:
- Beyond inventory — apply the same expected-overlap modeling to any entity collected from multiple sources (dealers, models, features), not just inventory.
- Structured deviation detection — group the keys missing from a feed by an attribute (model code, brand, dealer) so an alert points at what dropped out, not just how many. A whole-segment dropout (an entire model code vanishing from one feed) is far more actionable than an aggregate count.
- Declarative expectations — let each entity declare its expected cross-source relationship in YAML, so the check is configured alongside the merge spec rather than hand-written per OEM.