Merge Spec
The merge block controls how records are matched across sources and how field values are resolved when sources disagree.
merge:
match_on:
- [vin] # tier 1: tried first
- [order_number] # tier 2: tried when tier 1 finds no match
coalesce:
use_latest: [price] # prefer the value from the source with the most recent last_seen
evict_stale: [offer] # ignore a source's value if its last_seen is not today
match_on
An ordered list of field sets used to match right-side rows against the accumulated left side. Each right row is tried against tiers in order, and the first tier whose fields are non-empty and equal on both rows merges them. A right row that a tier does not match is tried on the next tier, but only against accumulated rows that lack a value in every earlier tier where the right row has one. Two rows with different values of an earlier tier's fields, such as two different VINs, are therefore never merged by a later tier.
Left (accumulated) wins on shared columns unless coalesce overrides the policy for specific fields.
merge:
match_on:
- [vin] # tier 1: tried first
- [order_number] # tier 2: tried when tier 1 finds no match
A tier field that only one side has is treated as null on the other side. A source row populated only in that tier is still matched on it, and is appended as a new row when no accumulated row matches. A tier field that neither side has raises an error. A configuration with a single tier raises when either side lacks its field.
coalesce
Field-level overrides for how values are selected across sources. By default every field is resolved by source priority order alone.
Each field listed under use_latest or evict_stale, and each target and source field of
append, must be a column of at least one source group. The merge raises an error naming
any field that no group has.
use_latest
Selects the value from the source with the most recent last_seen date among the sources
that have a value for the field. A source whose value is null is not compared. If two
sources share the same last_seen, priority order breaks the tie.
coalesce:
use_latest: [price, dealer_code, estimated_ship_date]
Use this for fields that should always reflect the freshest observation regardless of source priority — for example, a price that may have been updated in a lower-priority source after the higher-priority source last reported it.
evict_stale
Ignores a source's value for this field if that source's last_seen is not the current partition date. Priority order then chooses among the values that remain.
coalesce:
evict_stale: [offer_status]
Use this for fields that are only meaningful when fresh. If all sources are stale for a given field, the field is null on the merged row.
append
Maps a target list field to a source list field. After the merge, on a row where the source field has a value, the target holds the target's elements and then the source field's elements, with each distinct element once. A row whose source field is null keeps its target unchanged. On a row whose target is null, the target holds the source field's elements, with each distinct element once. The source field is not in the merged frame.
The target and the source field must be list columns of one dtype in every group that has
them. A source field whose target is in evict_stale must be in evict_stale too.
coalesce:
prefer:
feature_refs: [onegraph, accessaudi]
append:
feature_refs: accessory_refs
append adds the source field's elements to the target value that priority order, prefer
or use_latest selected. Use it when one source group keeps part of its list in a field of
its own, and the merge selects another group's list for the target.