Dealer website platform classification
Names the vendor that runs each dealer's website, and the parameters that vendor's inventory API needs to list the dealer's vehicles.
Asset
<oem>/<market>/dealers/website_platform, one row per dealer that publishes a
website, keyed by dealer_code.
| Column | Meaning |
|---|---|
dealer_code | The OEM's dealer code, from the upstream dealers table |
domain | Host that served the homepage after redirects; the upstream website's host when no response arrived |
requested_domain | Host of the upstream website, which the asset requested |
status_code | HTTP status of the final response, null on a connection error |
platform | Vendor, or unknown |
evidence | vendor_assets or vendor_mention, null when platform is unknown |
platform_params | JSON object of the parameters that vendor's inventory API needs |
built_by_text | The page's "Powered by", "Website by", "Designed by" or "Developed by" phrase and up to 60 characters after it, null when the page has none |
platform_params holds every parameter its platform needs. It is an empty
object on an unknown row and for a platform whose patterns define no
parameters.
On an unknown row, built_by_text is the lead for adding a vendor the patterns
file does not know.
The classified_rate check warns when the share of dealers with a known
platform falls below min_classified_rate.
Upstream
dealers_asset is a table with dealer_code and website columns: a transformed
dealers table, or for MBUSA and Audi US the <oem>/<market>/dealers/website asset. The asset fetches website as given, so it must reach the dealer's own
site; a host that redirects there is fine.
An empty upstream partition raises: it means the upstream has not materialized that partition yet.
A raw asset that reads the rows of one platform from a website_platform
partition writes no rows when the partition has none of that platform, and raises
when the partition is empty.
Such an asset leaves out a dealer whose website redirected to a site that another dealer
owns, and logs a warning naming both hosts. The owner of a site is a dealer, on any
platform, whose requested_domain equals its domain. Hosts are compared ignoring case.
A dealer group can retire one store's site and redirect it to a sister store's site. That
site lists the sister store's inventory under the sister store's provider ids.
The asset keeps a dealer whose website redirected to a site that no dealer owns, such as
a bare host that redirects to its www. host. It also keeps a row whose
requested_domain is null.
The asset logs how many upstream rows it read, how many had no website, how many websites held no host, and how many sites it fetched. On completion the fetch logs its elapsed time, requests issued, request rate, refusals and requests per dealer.
Configured OEMs
An OEM market is added with a defs.yaml under
projects/ai_public/src/ai_public/defs/<oem>/<market>/dealers/website_platform/,
setting dealers_asset and zyte_key_prefix: DEALER_WEBSITE.
| OEM market | dealers_asset |
|---|---|
stellantis/us | stellantis/us/transformed/dealer_locator_dealers |
mercedes/mbusa | mercedes/mbusa/dealers/website (MBUSA dealer websites) |
audi/us | audi/us/dealers/website (Audi US dealer websites) |
Evidence
evidence records which kind of page content identified the platform.
vendor_assets: the page configures the vendor's API, or loads code the vendor hosts. Either shows the vendor runs the site now.vendor_mention: the vendor's name or domain appears in the page, or the page links images the vendor hosts. A dealer that leaves a vendor can keep both, so this is the weaker evidence.
Every vendor_assets pattern is tried before any vendor_mention pattern. A
matching platform whose patterns define parameters is chosen only when every
parameter extracts, because those parameters are what the inventory collector
calls with. At each evidence value:
- The first matching platform in the patterns file whose parameters all extract wins.
- Otherwise, the first matching platform whose patterns define no parameters wins.
- Otherwise, the next
evidencevalue is tried. When none is left, the platform isunknown.
The patterns live in
packages/ai_public_api/src/ai_public_api/dealer_providers/website_platform_patterns.toml,
one entry per platform. The module validates the file when it loads: a
parameter pattern needs exactly one capture group, and a platform may be listed
only once.
A parameter's value comes from the first of its patterns that matches the page:
that pattern's first match, or, for a parameter listed in the platform's
most_common_params, its most frequent match. DealerOn's dealer_id is the one
such parameter, because a group site's homepage also loads files from the account
that hosts the group's shared logo. A team_velocity homepage can repeat a Google
Ads campaignid more often than it names its own campaignId, so other
parameters keep the first match.
Only a page served with a 2xx status is classified. A row with any other
status_code is unknown, whatever its body names: a 404 page or a block page
can contain a vendor's markup without the requested domain serving that vendor's
inventory.
Fetching homepages
The fetch uses AkamaiSession for browser TLS impersonation and ZyteSession
as a fallback; component-reference describes
both. This section covers only how this asset configures them.
Impersonation profiles
Each vendor's edge servers accept a different set of impersonation profiles, so
IMPERSONATION_PROFILES holds only profiles that Dealer Inspire, Dealer.com and
Dealer eProcess all serve. A session draws a new profile every
REQUESTS_PER_PROFILE requests.
A refusal is retried on a profile the dealer has not used yet. A refusal means
the edge server rejected that profile, so a retry on another profile can succeed. The
pool must hold at least REFUSAL_ATTEMPTS + 1 profiles: one per attempt, and
one for a rotation forced by the request cap during those attempts.
Refusals
A refusal is a 403, or a 2xx whose body is a bot check in place of the homepage. These bot checks are recognized from the start of the body:
- Dealer Inspire's challenge page, titled
Dealer Website, which the Dealer Inspire source describes. - Akamai's challenge page, containing a
sec-if-cpt-containerelement. - A page titled
Client Challenge. - Google's reCAPTCHA challenge page, whose
<base>element points athttps://www.google.com/recaptcha/challengepage/. Motive sites serve it.
Retry timing
- A refusal is retried with no wait. The retry wait pauses the whole pool, and the next attempt already uses a different profile.
- A 429 or 5xx is retried after its
Retry-Aftervalue or the exponential backoff, capped atMAX_RETRY_WAIT_SECONDS, so one longRetry-Afterdoes not pause every worker for as long as it asks. - When every attempt on a dealer is refused, the dealer's result holds the body of the last refusal.
Pacing
A shared Throttle spaces requests across the whole pool. Dealer domains are
separate hosts behind shared vendor edge servers, so the combined request rate is what
draws a 429.
MAX_WORKERS is high enough that the throttle, not the worker count, limits the
request rate: the worker count must exceed the throttle's permitted rate times
the time a homepage takes to fetch.
The only header the asset adds is Referer: https://www.google.com/, from
HOMEPAGE_HEADERS. curl_cffi sends the rest of the header set for the browser it
impersonates; a header of our own, such as a random User-Agent, would
contradict the TLS fingerprint. The Dealer Inspire source
describes the Referer as the header that passes its challenge page.
Zyte fallback
A dealer whose final direct attempt was refused or rate limited is fetched again
through Zyte, whose requests come from other addresses. A caller of
fetch_homepages can also send a dealer whose direct result it cannot use. The
Motive homepages asset sends each dealer whose homepage is a 2xx HTML page
without a dealer id.
- The Zyte result replaces the direct one only when Zyte receives a 2xx that is not a bot check.
- Zyte requests do not send the Referer.
- Each Zyte request has
oemandmarkettags, so its cost appears inorg/vendors/zyte/oem_market_stats.
Zyte retries
ZyteFallback.session_factory builds the Zyte session factory for the homepage
fetch on this page and for the Dealer.com search-pages and listings assets. A
request is sent through Zyte up to 4 times (ZYTE_ATTEMPTS). A transient failure,
such as a 429 or a 5xx from Zyte, is retried after waits of 3, 6 and 12 s
(ZYTE_BACKOFF_SECONDS, doubled at each retry). When the origin site's response,
relayed through Zyte, has a Retry-After header, the wait is that value instead,
capped at MAX_RETRY_WAIT_SECONDS as for a direct homepage request.
Zyte's error guide describes two kinds of retryable response:
- A 520,
/download/temporary-error: a temporary error, usually a ban, prevents the download of the requested URL. The guide says to retry it with an exponential backoff, and its example waits start at 3 s. Zyte does not charge for a 520. - A 429 or a 503: a rate-limiting response. A 429 is a limit on the user, on requests to one domain, or on the organization's requests to one domain. A 503 is a limit across all of Zyte. The guide says to retry a rate-limiting response as many times as needed.
The guide does not say whether a 520 has anything to do with the API key. We treat a 520 as concerning only the requested URL. We treat every status of the origin site that Zyte relays, and every failed connection to Zyte, the same way. Before retrying one of these, the worker that received it sleeps, and other requests continue.
A rate limit concerns more than the requested URL, so only a 429 or a 503 from the
Zyte API itself pauses the key's Zyte throttle, which all requests sent with the
key share. ZyteSession raises such a response as a ZyteAPIError, and
ZYTE_RATE_LIMIT_STATUSES lists the two statuses.
Credentials
| Variable | Source |
|---|---|
DEALER_WEBSITE_ZYTE_API_KEY_SECRET_ID | Secrets Manager dagster/public/dealer-website-zyte-api-key, set in projects/ai_public/container_context.yaml |
The key is labeled ai-tier3-dealer-sites in the Zyte dashboard. The secrets
Terraform solution creates the secret with no value, and the value is entered as
plaintext in the AWS console. The user-code execution role reads it through the
public-secrets-read policy in
deployments/aws/cloudformation/ecs-agent-vpc-private.yaml. For a local run,
export DEALER_WEBSITE_ZYTE_API_KEY directly.