Skip to main content

Dealer website platform classification

Names the vendor that runs each dealer's website, and the parameters that vendor's inventory API needs to list the dealer's vehicles.

Asset​

<oem>/<market>/dealers/website_platform, one row per dealer that publishes a website, keyed by dealer_code.

ColumnMeaning
dealer_codeThe OEM's dealer code, from the upstream dealers table
domainHost that served the homepage after redirects; the upstream website's host when no response arrived
requested_domainHost of the upstream website, which the asset requested
status_codeHTTP status of the final response, null on a connection error
platformVendor, or unknown
evidencevendor_assets or vendor_mention, null when platform is unknown
platform_paramsJSON object of the parameters that vendor's inventory API needs
built_by_textThe page's "Powered by", "Website by", "Designed by" or "Developed by" phrase and up to 60 characters after it, null when the page has none

platform_params holds every parameter its platform needs. It is an empty object on an unknown row and for a platform whose patterns define no parameters.

On an unknown row, built_by_text is the lead for adding a vendor the patterns file does not know.

The classified_rate check warns when the share of dealers with a known platform falls below min_classified_rate.

Upstream​

dealers_asset is a table with dealer_code and website columns: a transformed dealers table, or for MBUSA and Audi US the <oem>/<market>/dealers/website asset. The asset fetches website as given, so it must reach the dealer's own site; a host that redirects there is fine.

An empty upstream partition raises: it means the upstream has not materialized that partition yet.

A raw asset that reads the rows of one platform from a website_platform partition writes no rows when the partition has none of that platform, and raises when the partition is empty.

Such an asset leaves out a dealer whose website redirected to a site that another dealer owns, and logs a warning naming both hosts. The owner of a site is a dealer, on any platform, whose requested_domain equals its domain. Hosts are compared ignoring case. A dealer group can retire one store's site and redirect it to a sister store's site. That site lists the sister store's inventory under the sister store's provider ids.

The asset keeps a dealer whose website redirected to a site that no dealer owns, such as a bare host that redirects to its www. host. It also keeps a row whose requested_domain is null.

The asset logs how many upstream rows it read, how many had no website, how many websites held no host, and how many sites it fetched. On completion the fetch logs its elapsed time, requests issued, request rate, refusals and requests per dealer.

Configured OEMs​

An OEM market is added with a defs.yaml under projects/ai_public/src/ai_public/defs/<oem>/<market>/dealers/website_platform/, setting dealers_asset and zyte_key_prefix: DEALER_WEBSITE.

OEM marketdealers_asset
stellantis/usstellantis/us/transformed/dealer_locator_dealers
mercedes/mbusamercedes/mbusa/dealers/website (MBUSA dealer websites)
audi/usaudi/us/dealers/website (Audi US dealer websites)

Evidence​

evidence records which kind of page content identified the platform.

  • vendor_assets: the page configures the vendor's API, or loads code the vendor hosts. Either shows the vendor runs the site now.
  • vendor_mention: the vendor's name or domain appears in the page, or the page links images the vendor hosts. A dealer that leaves a vendor can keep both, so this is the weaker evidence.

Every vendor_assets pattern is tried before any vendor_mention pattern. A matching platform whose patterns define parameters is chosen only when every parameter extracts, because those parameters are what the inventory collector calls with. At each evidence value:

  1. The first matching platform in the patterns file whose parameters all extract wins.
  2. Otherwise, the first matching platform whose patterns define no parameters wins.
  3. Otherwise, the next evidence value is tried. When none is left, the platform is unknown.

The patterns live in packages/ai_public_api/src/ai_public_api/dealer_providers/website_platform_patterns.toml, one entry per platform. The module validates the file when it loads: a parameter pattern needs exactly one capture group, and a platform may be listed only once.

A parameter's value comes from the first of its patterns that matches the page: that pattern's first match, or, for a parameter listed in the platform's most_common_params, its most frequent match. DealerOn's dealer_id is the one such parameter, because a group site's homepage also loads files from the account that hosts the group's shared logo. A team_velocity homepage can repeat a Google Ads campaignid more often than it names its own campaignId, so other parameters keep the first match.

Only a page served with a 2xx status is classified. A row with any other status_code is unknown, whatever its body names: a 404 page or a block page can contain a vendor's markup without the requested domain serving that vendor's inventory.

Fetching homepages​

The fetch uses AkamaiSession for browser TLS impersonation and ZyteSession as a fallback; component-reference describes both. This section covers only how this asset configures them.

Impersonation profiles​

Each vendor's edge servers accept a different set of impersonation profiles, so IMPERSONATION_PROFILES holds only profiles that Dealer Inspire, Dealer.com and Dealer eProcess all serve. A session draws a new profile every REQUESTS_PER_PROFILE requests.

A refusal is retried on a profile the dealer has not used yet. A refusal means the edge server rejected that profile, so a retry on another profile can succeed. The pool must hold at least REFUSAL_ATTEMPTS + 1 profiles: one per attempt, and one for a rotation forced by the request cap during those attempts.

Refusals​

A refusal is a 403, or a 2xx whose body is a bot check in place of the homepage. These bot checks are recognized from the start of the body:

  • Dealer Inspire's challenge page, titled Dealer Website, which the Dealer Inspire source describes.
  • Akamai's challenge page, containing a sec-if-cpt-container element.
  • A page titled Client Challenge.
  • Google's reCAPTCHA challenge page, whose <base> element points at https://www.google.com/recaptcha/challengepage/. Motive sites serve it.

Retry timing​

  • A refusal is retried with no wait. The retry wait pauses the whole pool, and the next attempt already uses a different profile.
  • A 429 or 5xx is retried after its Retry-After value or the exponential backoff, capped at MAX_RETRY_WAIT_SECONDS, so one long Retry-After does not pause every worker for as long as it asks.
  • When every attempt on a dealer is refused, the dealer's result holds the body of the last refusal.

Pacing​

A shared Throttle spaces requests across the whole pool. Dealer domains are separate hosts behind shared vendor edge servers, so the combined request rate is what draws a 429.

MAX_WORKERS is high enough that the throttle, not the worker count, limits the request rate: the worker count must exceed the throttle's permitted rate times the time a homepage takes to fetch.

The only header the asset adds is Referer: https://www.google.com/, from HOMEPAGE_HEADERS. curl_cffi sends the rest of the header set for the browser it impersonates; a header of our own, such as a random User-Agent, would contradict the TLS fingerprint. The Dealer Inspire source describes the Referer as the header that passes its challenge page.

Zyte fallback​

A dealer whose final direct attempt was refused or rate limited is fetched again through Zyte, whose requests come from other addresses. A caller of fetch_homepages can also send a dealer whose direct result it cannot use. The Motive homepages asset sends each dealer whose homepage is a 2xx HTML page without a dealer id.

  • The Zyte result replaces the direct one only when Zyte receives a 2xx that is not a bot check.
  • Zyte requests do not send the Referer.
  • Each Zyte request has oem and market tags, so its cost appears in org/vendors/zyte/oem_market_stats.

Zyte retries​

ZyteFallback.session_factory builds the Zyte session factory for the homepage fetch on this page and for the Dealer.com search-pages and listings assets. A request is sent through Zyte up to 4 times (ZYTE_ATTEMPTS). A transient failure, such as a 429 or a 5xx from Zyte, is retried after waits of 3, 6 and 12 s (ZYTE_BACKOFF_SECONDS, doubled at each retry). When the origin site's response, relayed through Zyte, has a Retry-After header, the wait is that value instead, capped at MAX_RETRY_WAIT_SECONDS as for a direct homepage request.

Zyte's error guide describes two kinds of retryable response:

  • A 520, /download/temporary-error: a temporary error, usually a ban, prevents the download of the requested URL. The guide says to retry it with an exponential backoff, and its example waits start at 3 s. Zyte does not charge for a 520.
  • A 429 or a 503: a rate-limiting response. A 429 is a limit on the user, on requests to one domain, or on the organization's requests to one domain. A 503 is a limit across all of Zyte. The guide says to retry a rate-limiting response as many times as needed.

The guide does not say whether a 520 has anything to do with the API key. We treat a 520 as concerning only the requested URL. We treat every status of the origin site that Zyte relays, and every failed connection to Zyte, the same way. Before retrying one of these, the worker that received it sleeps, and other requests continue.

A rate limit concerns more than the requested URL, so only a 429 or a 503 from the Zyte API itself pauses the key's Zyte throttle, which all requests sent with the key share. ZyteSession raises such a response as a ZyteAPIError, and ZYTE_RATE_LIMIT_STATUSES lists the two statuses.

Credentials​

VariableSource
DEALER_WEBSITE_ZYTE_API_KEY_SECRET_IDSecrets Manager dagster/public/dealer-website-zyte-api-key, set in projects/ai_public/container_context.yaml

The key is labeled ai-tier3-dealer-sites in the Zyte dashboard. The secrets Terraform solution creates the secret with no value, and the value is entered as plaintext in the AWS console. The user-code execution role reads it through the public-secrets-read policy in deployments/aws/cloudformation/ecs-agent-vpc-private.yaml. For a local run, export DEALER_WEBSITE_ZYTE_API_KEY directly.