Skip to main content

Run Metrics

RunMetricsComponent measures what an ordering run recommended. It needs no vehicle outcome, so its metrics are final once the run's events are written. See The reporting layer and Run modes.

Recommendation metrics​

<oem>/<market>/reporting/recommendation_metrics has one row per recommendation_id, in one partition per ordering run. It covers every run mode, and run_mode is a column.

ColumnsSource
score, module_scoresThe recommendation event
final_score, scorer_outputs, misalignment_by_attribute_type, candidate_attributes, cnt_pre_restriction_prune, cnt_post_restriction_pruneThe recommendation's metadata
skipped, skip_reasonThe recommendation event
attempt_id, submittable, refusal_reason, has_cured_buildThe validated event of the recommendation's attempt
placedplaced() over the attempt's outcome event
confirmedThe success of the attempt's outcome event. An attempt in a SIMULATED or DRY_RUN run can be confirmed, but it is never placed.
dealer_id, base_model_idconsolidated/dealers and consolidated/models

A skipped recommendation has no attempt, so its attempt columns are null, and has_cured_build, placed and confirmed are false.

run_metrics​

<oem>/<market>/reporting/run_metrics has one row per dealer of an ordering run, in the same ordering-run partition as recommendation_metrics. It reads the run's partition of recommendation_metrics and the run's rows of ordering/runs, which give run_id, run_mode and phase_timings. A dealer with no recommendations still has its row. Each row's metrics are computed over the recommendations of the row's dealer.

An ordering/runs row with no dealer_code or no partition_date names no dealer to look up. The first was written before ordering/runs had a dealer_code column, and the second belongs to a run that stopped before the partition probe answered. Such a run has one row, and its dealer_id is the one its recommendations name, or null when it has no recommendations.

ColumnMeaning
dealer_idThe dealer's dealer_id in consolidated/dealers at the run's partition_date. recommendation_metrics names dealers by the same id.
recommendation_countThe dealer's recommendations
target_countDistinct target_index values of the dealer's recommendations. A retried target has several recommendations with the same target_index.
skipped_countRecommendations whose skipped is true
overpruned_countRecommendations whose skip_reason is overpruned
submittable_countRecommendations whose submittable is true
placed_countRecommendations whose placed is true
confirmed_countRecommendations whose confirmed is true
restriction_pruned_candidate_rateThe mean, over the dealer's candidate-selection scopes, of 1 minus cnt_post_restriction_prune / cnt_pre_restriction_prune. The recommendations of one scope have the same counts, so each scope counts once. Null when no recommendation has a cnt_pre_restriction_prune above 0.
overpruned_rateThe share of targets with a recommendation whose skip_reason is overpruned. Null for a dealer with no recommendations.
first_weighted_misalignment, weighted_misalignment_reduction, weighted_misalignment_reduction_rateDescribed under Alignment
attribute_typesOne entry per attribute type, sorted by type
run_duration_seconds, mean_recommendation_seconds, phase_durationsDescribed under Duration

The counts are per recommendation, so a retried target counts once for each of its recommendations. The alignment and ranking metrics are computed only over the recommendations whose skipped is false. When a dealer has no such recommendation, attribute_types is empty and the three weighted misalignment columns are null.

Alignment​

Each recommendation's misalignment_by_attribute_type has one entry per attribute type. The entry's total_misalignment is the sum of the absolute differences between demand and supply take rates. For each dealer_id, model_id and attribute type, recommendations are sorted by target_index. The reduction is the first recommendation's total_misalignment minus the last recommendation's.

A recommendation's total_misalignment is computed before its order is added to supply. The reduction therefore excludes the last recommendation's order.

The reduction is exact when the run has one candidate-selection SelectionScope per dealer and model. When a model has several, such as one per production date, the first and last values come from different scopes' supply, so the reduction is approximate.

first_weighted_misalignment multiplies each model's first total_misalignment by the first recommendation's attribute_weight. weighted_misalignment_reduction multiplies each reduction by the same weight. Both sum the products over models and attribute types, and both are null when no weight is set. The run-level columns are weighted only, because candidate selection gives many attribute types a weight close to 0.

Both sums grow with the number of the dealer's models in the run. weighted_misalignment_reduction_rate is weighted_misalignment_reduction divided by first_weighted_misalignment, which compares across runs of different sizes. It is null when first_weighted_misalignment is null or 0.

Ranking​

Each value in candidate_attributes has a rank among the values of its attribute type. Rank 1 is the value whose demand take rate exceeds its supply take rate by the most. For each recommendation and attribute type, the ranking metrics use the value with the best rank. A type with several values in one candidate therefore counts once per recommendation.

attribute_types​

FieldMeaning
first_misalignmentThe first recommendation's total_misalignment, summed over models
misalignment_reductionThe reduction described under Alignment, summed over models
misalignment_reduction_ratemisalignment_reduction divided by first_misalignment. Null when first_misalignment is null or 0.
mean_attribute_weightThe first recommendation's attribute_weight, averaged over models
ranked_recommendation_countRecommendations with a ranked value of the type
ranked_first_rateThe share of those recommendations whose best rank is 1
top_quartile_rateThe share whose best rank is at most a quarter of num_values_in_type, rounded up. For a type with at most 4 values, it equals ranked_first_rate.
mean_normalized_rankThe mean of 1 minus (best rank − 1) / (num_values_in_type − 1). A type with one value counts as 1.

Duration​

The ordering workflow writes phase_timings to the run's ordering_runs document. It has one entry for each workflow phase the run entered. An entry's seconds is the time the run spent in the phase, summed over every time the run entered it. An entry's entry_count is the number of times the run entered the phase. The workflow measures the time with workflow.now().

ColumnMeaning
run_duration_secondsThe sum of seconds over every phase. It ends when the run enters its final phase, such as COMPLETED, so it excludes the run-completed notification and the PDF report. It includes the time in PAUSED.
mean_recommendation_secondsThe seconds in RECOMMENDING, DRY_RUN_VALIDATING, ORDERING and PERSISTING, divided by the entry_count of RECOMMENDING. It is the mean time per get_next_recommendation call. The run enters RECOMMENDING before each call, including a call that returns no recommendation. The time counted for a call ends when the run next enters RECOMMENDING or a phase outside these four. Null when the run never entered RECOMMENDING.
phase_durationsThe phase, seconds and entry_count of each entry of phase_timings, in the order the run first entered each phase

A run recorded before the workflow wrote phase_timings has null in all three columns.

Asset checks​

missing_metadata is non-blocking. It counts the recommendations with no metadata.

missing_take_rates is non-blocking. It counts the recommendations that are not skipped and whose values in candidate_attributes all have neither a supply nor a demand take rate. A recommendation with only some values missing both take rates is not counted. Candidate selection gives a value no take rate when neither its demand data nor its supply data contains the value.

The component raises when:

  • a dealer_code of a recommendation or of an ordering/runs row has no row in consolidated/dealers;
  • a recommended model_id has no row in consolidated/models;
  • the run's ordering/runs rows have different partition_date values;
  • a run with no dealer_code or no partition_date has more than one ordering/runs row, or recommendations for more than one dealer;
  • consolidated/dealers or consolidated/models has no partition at the run's partition_date.