Run Metrics
RunMetricsComponent measures what an ordering run recommended. It needs no vehicle outcome, so its
metrics are final once the run's events are written. See
The reporting layer and
Run modes.
Recommendation metrics
<oem>/<market>/reporting/recommendation_metrics has one row per recommendation_id, in one
partition per ordering run. It covers every run mode, and run_mode is a column.
| Columns | Source |
|---|---|
score, module_scores | The recommendation event |
final_score, scorer_outputs, misalignment_by_attribute_type, candidate_attributes, cnt_pre_restriction_prune, cnt_post_restriction_prune | The recommendation's metadata |
skipped, skip_reason | The recommendation event |
attempt_id, submittable, refusal_reason, has_cured_build | The validated event of the recommendation's attempt |
placed | placed() over the attempt's outcome event |
confirmed | The success of the attempt's outcome event. An attempt in a SIMULATED or DRY_RUN run can be confirmed, but it is never placed. |
dealer_id, base_model_id | consolidated/dealers and consolidated/models |
A skipped recommendation has no attempt, so its attempt columns are null, and has_cured_build,
placed and confirmed are false.
run_metrics
<oem>/<market>/reporting/run_metrics has one row per dealer of an ordering run, in the same
ordering-run partition as recommendation_metrics. It reads the run's partition of
recommendation_metrics and the run's rows of ordering/runs, which give run_id, run_mode and
phase_timings. A dealer with no recommendations still has its row. Each row's metrics are
computed over the recommendations of the row's dealer.
An ordering/runs row with no dealer_code or no partition_date names no dealer to look up.
The first was written before ordering/runs had a dealer_code column, and the second belongs to
a run that stopped before the partition probe answered. Such a run has one row, and its
dealer_id is the one its recommendations name, or null when it has no recommendations.
| Column | Meaning |
|---|---|
dealer_id | The dealer's dealer_id in consolidated/dealers at the run's partition_date. recommendation_metrics names dealers by the same id. |
recommendation_count | The dealer's recommendations |
target_count | Distinct target_index values of the dealer's recommendations. A retried target has several recommendations with the same target_index. |
skipped_count | Recommendations whose skipped is true |
overpruned_count | Recommendations whose skip_reason is overpruned |
submittable_count | Recommendations whose submittable is true |
placed_count | Recommendations whose placed is true |
confirmed_count | Recommendations whose confirmed is true |
restriction_pruned_candidate_rate | The mean, over the dealer's candidate-selection scopes, of 1 minus cnt_post_restriction_prune / cnt_pre_restriction_prune. The recommendations of one scope have the same counts, so each scope counts once. Null when no recommendation has a cnt_pre_restriction_prune above 0. |
overpruned_rate | The share of targets with a recommendation whose skip_reason is overpruned. Null for a dealer with no recommendations. |
first_weighted_misalignment, weighted_misalignment_reduction, weighted_misalignment_reduction_rate | Described under Alignment |
attribute_types | One entry per attribute type, sorted by type |
run_duration_seconds, mean_recommendation_seconds, phase_durations | Described under Duration |
The counts are per recommendation, so a retried target counts once for each of its
recommendations. The alignment and ranking metrics are computed only over the recommendations whose
skipped is false. When a dealer has no such recommendation, attribute_types is empty and the
three weighted misalignment columns are null.
Alignment
Each recommendation's misalignment_by_attribute_type has one entry per attribute type. The
entry's total_misalignment is the sum of the absolute differences between demand and supply take
rates. For each dealer_id, model_id and attribute type, recommendations are sorted by
target_index. The reduction is the first recommendation's total_misalignment minus the last
recommendation's.
A recommendation's total_misalignment is computed before its order is added to supply. The
reduction therefore excludes the last recommendation's order.
The reduction is exact when the run has one candidate-selection SelectionScope per dealer and
model. When a model has several, such as one per production date, the first and last values come
from different scopes' supply, so the reduction is approximate.
first_weighted_misalignment multiplies each model's first total_misalignment by the
first recommendation's attribute_weight. weighted_misalignment_reduction multiplies each
reduction by the same weight. Both sum the products over models and attribute types, and
both are null when no weight is set. The run-level columns are weighted only, because candidate
selection gives many attribute types a weight close to 0.
Both sums grow with the number of the dealer's models in the run.
weighted_misalignment_reduction_rate is weighted_misalignment_reduction divided by
first_weighted_misalignment, which compares across runs of different sizes. It is null when
first_weighted_misalignment is null or 0.
Ranking
Each value in candidate_attributes has a rank among the values of its attribute type. Rank 1 is
the value whose demand take rate exceeds its supply take rate by the most. For each recommendation
and attribute type, the ranking metrics use the value with the best rank. A type with several values
in one candidate therefore counts once per recommendation.
attribute_types
| Field | Meaning |
|---|---|
first_misalignment | The first recommendation's total_misalignment, summed over models |
misalignment_reduction | The reduction described under Alignment, summed over models |
misalignment_reduction_rate | misalignment_reduction divided by first_misalignment. Null when first_misalignment is null or 0. |
mean_attribute_weight | The first recommendation's attribute_weight, averaged over models |
ranked_recommendation_count | Recommendations with a ranked value of the type |
ranked_first_rate | The share of those recommendations whose best rank is 1 |
top_quartile_rate | The share whose best rank is at most a quarter of num_values_in_type, rounded up. For a type with at most 4 values, it equals ranked_first_rate. |
mean_normalized_rank | The mean of 1 minus (best rank − 1) / (num_values_in_type − 1). A type with one value counts as 1. |
Duration
The ordering workflow writes phase_timings to the run's ordering_runs document. It has one
entry for each workflow phase the run entered. An entry's seconds is the time the run spent in
the phase, summed over every time the run entered it. An entry's entry_count is the number of
times the run entered the phase. The workflow measures the time with workflow.now().
| Column | Meaning |
|---|---|
run_duration_seconds | The sum of seconds over every phase. It ends when the run enters its final phase, such as COMPLETED, so it excludes the run-completed notification and the PDF report. It includes the time in PAUSED. |
mean_recommendation_seconds | The seconds in RECOMMENDING, DRY_RUN_VALIDATING, ORDERING and PERSISTING, divided by the entry_count of RECOMMENDING. It is the mean time per get_next_recommendation call. The run enters RECOMMENDING before each call, including a call that returns no recommendation. The time counted for a call ends when the run next enters RECOMMENDING or a phase outside these four. Null when the run never entered RECOMMENDING. |
phase_durations | The phase, seconds and entry_count of each entry of phase_timings, in the order the run first entered each phase |
A run recorded before the workflow wrote phase_timings has null in all three columns.
Asset checks
missing_metadata is non-blocking. It counts the recommendations with no metadata.
missing_take_rates is non-blocking. It counts the recommendations that are not skipped and whose
values in candidate_attributes all have neither a supply nor a demand take rate. A recommendation
with only some values missing both take rates is not counted. Candidate selection gives a value no
take rate when neither its demand data nor its supply data contains the value.
The component raises when:
- a
dealer_codeof a recommendation or of anordering/runsrow has no row inconsolidated/dealers; - a recommended
model_idhas no row inconsolidated/models; - the run's
ordering/runsrows have differentpartition_datevalues; - a run with no
dealer_codeor nopartition_datehas more than oneordering/runsrow, or recommendations for more than one dealer; consolidated/dealersorconsolidated/modelshas no partition at the run'spartition_date.