Valhalla routing tiles
osm/us/valhalla/tiles is a Valhalla routing graph of the US for pyvalhalla.
It is built from Geofabrik's OpenStreetMap extract of the US and stored as one tile
archive. The archive is the indexed .tar file that valhalla.Actor reads
from the path in its mjolnir.tile_extract config setting.
Asset
The asset is defined in defs/osm/us/valhalla_tiles/defs.yaml of ai_public.
The component's country sets the key osm/<country>/valhalla/tiles, and its
region names the country's Geofabrik extract, such as north-america/us.
It is not an Iceberg table. Each materialization writes one object to the
ai-app-public-artifacts-prod bucket, at a path that includes the asset key:
s3://ai-app-public-artifacts-prod/osm/us/valhalla/tiles/<tiles_version>/valhalla_tiles.tar
On a branch deployment the path starts with branches/<git branch>/, so a
materialization on a branch deployment never writes a version that prod reads.
A published tiles_version never changes. The upload uses exclusive creation
(mode="create"), which fails when an object already exists at the path.
When an object already exists at the path, the asset records a materialization
of that object without downloading or building. It sets archive_reused to
true in the materialization's metadata.
The existing object is a complete archive that passed the isochrone check, step 4 under Build. S3 makes an object visible only after its upload completes, and the asset uploads the archive only after the check.
A run can end after the upload but before Dagster records the materialization,
for example when the run reaches its dagster/max_runtime limit. Launching the
partition again records a materialization of the uploaded archive.
The bucket holds files that ai_public builds and that are not Iceberg tables.
Terraform defines it in
deployments/aws/terraform/solutions/dagster-agent/artifacts_s3.tf, with these
lifecycle rules:
- a multipart upload still incomplete after 7 days is aborted;
- an object under
branches/expires after 30 days.
The user-code task role's public-artifacts-s3-access policy, in
deployments/aws/cloudformation/ecs-agent-vpc-private.yaml, allows:
s3:GetObject,s3:PutObjectands3:AbortMultipartUploadon the bucket's objects;s3:ListBucketands3:GetBucketLocationon the bucket.
s3fs uploads the archive in parts and aborts the upload when a part fails, so
the role needs s3:AbortMultipartUpload. The policy does not allow
s3:DeleteObject, so no run can delete a published archive.
The dagster-agent Terraform solution creates the bucket and deploys that
CloudFormation template as a stack. Its apply must finish before the first
materialization, including one on a branch deployment.
Versions
The asset has one partition per year, starting at first_year. The partition
key is January 1 of the year, in the format YYYY-MM-DD. The archive's
tiles_version is the partition key. A year's partition exists from January 1
of that year.
A partition is built from Geofabrik's January 1 extract of its year,
<region>-YY0101.osm.pbf. Geofabrik keeps the January 1 extract of every year
since 2021, but it keeps the .md5 file only for the current year's extracts.
The asset checks the download against the .md5 when Geofabrik has one, and
otherwise against the Content-Length of a HEAD request.
osm_extract_md5_checked in the materialization's metadata records which check
ran.
Geofabrik can publish the January 1 extract as late as January 2, so a build on January 1 can fail with an HTTP 404.
Launching a build
The osm_us_valhalla_tiles_schedule schedule builds the current year's
partition at the time in cron_schedule, 06:00 UTC on January 3. Its default
status on a branch deployment is stopped.
A build of the US runs longer than the 3-hour
run_monitoring.max_runtime_seconds in dagster_cloud_settings.yaml. The
osm_us_valhalla_tiles_job job sets the run tag dagster/max_runtime to
max_runtime_hours, converted to seconds. The tag overrides that limit for the
job's runs. Launch any other build, such as an earlier year or a test on a
branch deployment, from that job. A run launched with Materialize on the
asset does not have the tag.
Build
- Download the extract and check its MD5, or its size when Geofabrik has no
.md5file for it. The asset retries 429 and 5xx responses and dropped connections. A retried download resumes from the last byte written. - Build the tiles with pyvalhalla's
valhalla_build_tiles, with one thread per CPU thatos.process_cpu_count()reports in the step's task. The materialization records the count asbuild_threads. Valhalla writes its log to the step's stdout. - Pack the tiles into the archive with
valhalla_build_extract. - Route a 5-minute
autoisochrone from the point atcheck_latitude,check_longitude. The asset fails when the isochrone has no polygon. - Upload the archive.
The build uses no admin, timezone, landmark, transit or traffic data. An auto
isochrone without a departure time reads none of them. Without the admin
database, Valhalla applies its defaults for country-specific access rules and
for border crossings.
The tiles have no bicycle or pedestrian edges, so they support only the auto
costing model.
Edge speeds come from OSM maxspeed tags and from Valhalla's default speed for
each road class, so drive times are free-flow times.
Replacing a bad archive
An archive that passes the isochrone check can still be wrong. No run can delete it. A new run of its partition records a materialization of the existing archive instead of building a new one. To rebuild a partition:
- Delete the object at its
archive_uriwith credentials that allows3:DeleteObjecton the bucket, such as an administrator's:aws s3 rm s3://ai-app-public-artifacts-prod/osm/us/valhalla/tiles/<tiles_version>/valhalla_tiles.tar - Launch the partition from
osm_us_valhalla_tiles_job.
Task size
The task asks for:
- 8 vCPU;
- 60 GB of memory;
- 100 GiB of ephemeral storage, set with
op_tags: {ecs/ephemeral_storage: "100"}.
The US extract is about 12 GB and its archive about 15 GB. A build's working
directory uses at most about 80 GiB of disk, during valhalla_build_tiles.
The task asks for more than that because Fargate's ephemeral storage also holds
the container image.