Census Blocks
census/tiger/tabblock20 holds the housing units, population and internal point of every
2020 census block of the 50 states, the District of Columbia and Puerto Rico. The table
has about 8.17 million rows. It has one partition, 2020.
market/us/dealers/isochrone_tracts reads
it to compute the share of a tract's housing units inside an isochrone.
Source
The attribute table (.dbf) of the TIGER/Line 2020 TABBLOCK20 shapefile of each state,
https://www2.census.gov/geo/tiger/TIGER2020/TABBLOCK20/tl_2020_<state FIPS>_tabblock20.zip.
The table holds the block's GEOID20, HOUSING20, POP20 and internal point
(INTPTLAT20, INTPTLON20) from the 2020 census. No API key is needed.
The zips hold the block outlines too, about 10 GB in all. The asset reads only the .dbf
member, with HTTP range requests:
- A
HEADrequest gives the zip's size. - Range requests read the zip's central directory, which gives the offset and size of the
.dbfmember. - Range requests of at most 8 MiB read the member's compressed bytes.
zipfiledecompresses them and checks the member's CRC.
Each range request is retried on its own, on a 429 or 5xx response, on a connection error
and on a response shorter than the range requested. The compressed .dbf members are
about 4.5% of the size of the zips. The asset downloads about 260 MB and writes 1.1 GB of
.dbf files to a temporary directory.
The 2020 PL 94-171 redistricting files have the same counts. The asset does not read them,
because each state's zip holds every summary level and all three data segments.
California's zip is 80 MB, against 17 MB for the compressed .dbf member of its
TABBLOCK20 zip. The asset does not read the Census Data API either, because the API
needs a key.
TIGER/Line also publishes TABBLOCK20 for American Samoa, Guam, the Northern Mariana
Islands and the US Virgin Islands. Their HOUSING20 and POP20 fields are blank, so the
asset does not read them.
Columns
| Column | Type | Meaning |
|---|---|---|
geo_id | String | 15-digit block GEOID (GEOID20) |
tract_geo_id | String | The block's 2020 tract GEOID, the first 11 digits of geo_id |
housing_units | Int64 | Housing units in 2020 (HOUSING20) |
population | Int64 | Population in 2020 (POP20) |
latitude, longitude | Float64 | The block's internal point (INTPTLAT20, INTPTLON20), NAD83 |
year | Int64 | 2020 |
_partition | String | 2020, the partition key |
The internal point is a point inside the block, near its center. The asset does not
transform it to EPSG:4326. pyproj's default NAD83 to WGS84 transformation, which
census/tiger/cb_tract applies, leaves
coordinates in the contiguous states, the District of Columbia and Puerto Rico unchanged
and moves coordinates in Alaska and Hawaii by less than 1 m.
tract_geo_id is the tract of the 2020 census. In 2022 Connecticut's counties were
replaced by nine planning regions. Its tracts kept their outlines and their 6-digit tract
codes, and took the planning region's code in their GEOID, so a Connecticut block's
tract_geo_id differs from the GEOID of its tract in census/tiger/cb_tract 2024.
Checks
A materialization fails when any of these holds:
- The
.dbflacksGEOID20,HOUSING20,POP20,INTPTLAT20orINTPTLON20. - The fields' widths do not add up to the record length, or the file holds fewer records than its header states.
- A record is marked deleted.
- A GEOID is not 15 digits.
HOUSING20orPOP20is blank or negative.- The internal point does not parse.
- A server answers a range request with a status other than 206.
Resources
The asset reads the .dbf files in batches of 50,000 records and streams them to the
Iceberg table. Locally, downloading the tables of every state's zip takes about 90
seconds. Reading and writing the tables takes about 2 seconds, at a peak of about 840 MB
of memory. The table takes about 180 MB in Parquet. ecs_vcpu 1 and ecs_memory_gb 2 in
the asset's defs.yaml set its ECS task size.
eager automation materializes the partition once. To turn automation off, set
automation: null in the asset's defs.yaml.