ACS demographic features
ai_core.acs defines the demographic features derived from Census ACS 5-year variables.
The definitions are in packages/ai_core/src/ai_core/acs/feature_defs.json, and they cover
both a single geography and several geographies, such as the census tracts inside a
dealer's isochrone.
These consumers use the features:
DealerDemographicsComponentinai_dagstercopies the features of one ZCTA to each dealer, for example inbmw/us/dealers/demographics.market/us/dealers/demographicscomputes the features of each dealer location's census tract and of each of its isochrones fromcensus/acs/acs5_tract.
Features of one geography
feature_defs.json has two lists:
scalars: each feature is one ACS variable, such asmedian_hh_incomefromB19013_001E.ratios: each feature is the sum of itsnumeratorsvariables divided by itsdenominatorvariable, such aspct_bachelors_plus.
derive_acs_features() pivots long-format ACS rows, such as those of
census/acs/acs5_zcta, to one row per geo_id and computes both lists.
Absent variables
A variable can be absent from the ACS rows, for example when a vintage does not publish it
or a geography has no value for it. DealerDemographicsComponent computes its features with
derive_acs_features(), which treats an absent variable as follows:
- A numerator variable absent from one geography counts as 0 in that geography.
- A ratio is null in a geography whose denominator variable is absent or not positive.
- A numerator variable absent from every geography is left out of the sum, and a warning is logged.
- A ratio whose denominator variable, or every numerator variable, is absent from every geography is a null column.
- A scalar whose variable is absent from every geography has no column.
Features of several geographies
The definitions state how to compute each feature over several geographies. The value of a ratio over several geographies is the sum of its numerators over the geographies divided by the sum of its denominator.
The aggregation of each scalar states how to compute the scalar over several
geographies:
kind | Computation | Scalars |
|---|---|---|
sum | The sum of the scalar's own variable | total_population |
ratio | Like a ratio feature: summed numerators divided by the summed denominator | per_capita_income |
bracket_median | A median interpolated from bracket counts summed over the geographies | median_hh_income, median_age, median_gross_rent, median_home_value |
lognormal_mixture_gini | The Gini index of a mixture of one log-normal income distribution per geography | gini_index |
A median or a Gini index of several geographies is not a function of the geographies'
medians or Gini indexes. A bracket_median scalar is computed from the summed
distribution, and a lognormal_mixture_gini scalar from a model of each geography's
distribution.
acs_variables_for_aggregation() returns every ACS variable needed to compute a set of
features over several geographies:
- the variable of each scalar;
- the numerators and the denominator of each ratio;
- the bracket variables of each
bracket_medianaggregation; - the
householdsandaggregatevariables and thefallbackbracket variables of eachlognormal_mixture_giniaggregation; - the numerators and the denominator of each
ratioaggregation.
aggregate_acs_features() in ai_core.acs.aggregation computes the features of groups
of geographies. It takes long-format ACS rows and a frame of weights, with one row per
group and geography. The weight is the share of the geography's values that belongs to
the group. Each sum, including the numerators and denominator of a ratio and the bracket
counts of a median, is a sum of values times weights. A lognormal_mixture_gini scalar
weights each geography by its weight times its households. In a group whose only
geography of positive weight has weight 1, it is that geography's own published value
where that is not null.
aggregate_acs_features() also computes the
CBP features of groups of counties,
with the weights that County features
describes.
census/acs/acs5_tract fetches every variable that acs_variables_for_aggregation()
returns for the features in feature_defs.json, and some variables that no feature
uses.
Null and absent values
The ACS API reports a missing estimate with a negative sentinel, such as -666666666.
CensusAcsComponent stores the value as null. aggregate_acs_features() treats a null
value, and a geography with no row for a variable, as follows:
- The value counts as 0 in a sum, including the bracket counts of a median.
- A ratio leaves out each geography whose denominator is null.
- A ratio is null when its summed denominator is not positive.
- A median is null when its summed bracket counts are all 0.
- A
lognormal_mixture_giniscalar leaves out each geography that has no log-normal distribution, as listed in Excluded geographies. It is null when it leaves out every geography of the group. - In a group whose only geography of positive weight has weight 1, a
lognormal_mixture_giniscalar whose published value is null is the mixture's value, as it is when that geography has a lower weight.
aggregate_acs_features() raises ValueError when a variable the features need has no
row for any geography.
per_capita_income (B19301_001E) is aggregate income divided by total population. The
numerator is B19313_001E, the aggregate income of the population 15 years and over. The
denominator is B01001_001E, the total population, which is the universe of B19301.
Weights of partial tracts
market/us/dealers/demographics
weights each tract by the share of its 2020 housing units inside the isochrone.
The eval in projects/ai_public/evals/partial_geography/ compares housing-unit weights
with area and population weights on ZCTAs, state house districts and congressional
districts, which split tracts. For each geography, the eval aggregates the ACS 2024 values
of the tracts the geography overlaps under each weight. It compares the result with the
values the ACS publishes for the geography. Every weight goes through the same Gini
estimator.
Housing-unit weights bring household and population counts much closer to the published
counts than area weights do. They bring gini_index closer by less, because most of its
error comes from the log-normal mixture estimate itself. Against the ACS 2024 values, the
mean absolute error of household counts is 1.2% with housing-unit weights and 5.8% with
area weights on state house districts, and 4.5% and 16.0% on ZCTAs. The mean absolute
error of gini_index is 0.0059 and 0.0066 on state house districts, and 0.0164 and
0.0177 on ZCTAs. Population weights are slightly closer than housing-unit weights on
population counts, and no closer on household counts or gini_index. Every feature
therefore takes the housing-unit weight.
Bracket medians
Each bracket_median lists the brackets of a distribution. A bracket has a lower and an
upper bound and a list of variables, whose sum is the bracket's count. The last bracket
has no upper bound. The upper bound of each bracket is the lower bound of the next
bracket. For example, the bracket labeled "$10,000 to $14,999" is the interval from 10,000
to 15,000.
| Scalar | Published median | Bracket variables | Universe |
|---|---|---|---|
median_hh_income | B19013_001E | B19001_002E to B19001_017E, from "Less than $10,000" to "$200,000 or more" | Households |
median_age | B01002_001E | B01001_003E to B01001_025E (male) plus B01001_027E to B01001_049E (female), from "Under 5 years" to "85 years and over" | Total population |
median_gross_rent | B25064_001E | B25063_003E to B25063_026E, from "Less than $100" to "$3,500 or more" | Renter-occupied housing units paying cash rent |
median_home_value | B25077_001E | B25075_002E to B25075_027E, from "Less than $10,000" to "$2,000,000 or more" | Owner-occupied housing units |
The gross rent brackets leave out B25063_027E, "No cash rent", because B25064 is the
median over units that pay cash rent. Each age bracket sums the male and the female
variable of the same age range.
Interpolation
ai_core.acs.median.bracket_median() returns a Polars expression that computes the median
from the bracket count columns:
- Half the total count is the median position.
- The median bracket is the first bracket whose cumulative count reaches the median position.
- The median is the bracket's
lowerbound plus the bracket's width times the fraction of the bracket's count below the median position.
The interpolation is linear, so it assumes that values are uniformly distributed within a bracket. The Census Bureau computes its published ACS medians from finer distributions than the published brackets, so an interpolated median of a single geography differs slightly from that geography's published median.
When the median position falls in the open-ended top bracket, the median is that bracket's
lower bound, such as 200,000 for household income. The value is a lower bound on the true
median. Interpolation in the top bracket would need an assumed upper bound or an assumed
distribution within the bracket, such as a Pareto distribution.
The median is null when the total count is zero or any bracket count is null.
Medians in the top bracket
A published median can be greater than the lower bound of the top bracket, and the
interpolated median is never greater than that bound. For example, the top bracket of
B19001 is "$200,000 or more", and the published B19013 median household income is
top-coded at "250,000+". In an area whose median household income is above 200,000, the
published median is between 200,000 and 250,000 or is "250,000+", and the interpolated
median is 200,000.
Vintages
In each ACS 5-year vintage from 2015 through 2024, the label of every bracket variable matches the bounds of its bracket.
Before 2015, some of the same variable codes have other meanings. For example, in the 2014
vintage B25063_023E is "$2,000 or more" and B25063_024E is "No cash rent", and
B25063_025E and B25063_026E do not exist. B25075_025E is "$1,000,000 or more", and
B25075_026E and B25075_027E do not exist.
Each bracket_median, and the fallback of each lognormal_mixture_gini, has a
min_vintage of 2015 and a max_vintage of 2024, the vintages whose labels were
checked. bracket_median() and bracket_gini() raise ValueError for a vintage outside
them. To add a later vintage, check that the labels of every bracket variable in that
vintage match the bounds, then raise max_vintage in feature_defs.json.
Gini index of several geographies
gini_index of a group of tracts, such as the tracts inside a dealer's isochrone, is the
Gini index of the group's household income distribution. The ACS publishes each tract's
Gini index B19083_001E, but the Gini index of a union of tracts is not a function of the
tracts' Gini indexes. It also depends on how far apart the tracts' incomes are.
aggregate_acs_features() models each tract's income distribution as a log-normal
distribution and computes the Gini index of the mixture of those distributions.
Model
Each tract t has a log-normal household income distribution with the tract's own mean income and Gini index:
- N_t, the number of households, is
B19001_001E. - μ_t, the mean household income, is the aggregate household income
B19025_001Edivided by N_t. - G_t is the published Gini index
B19083_001E. Where that is null, G_t is thebracket_gini()estimate from the tract's own income brackets and aggregate income, as described in Bracket Gini index. - σ_t = √2 · Φ⁻¹((G_t + 1) / 2), where Φ is the standard normal distribution function. A log-normal distribution whose logarithm has standard deviation σ has the Gini index G = 2Φ(σ / √2) − 1 (Aitchison and Brown, The Lognormal Distribution, 1957), so the distribution with σ_t has the Gini index G_t.
- m_t = ln μ_t − σ_t² / 2 is the mean of the logarithm, which makes the distribution's mean μ_t.
A group's income distribution is the mixture of its tracts' distributions in which tract
t has the mixture weight w_t · N_t, where w_t is the tract's weight in the group:
- The group's mean income is μ = Σ w_t N_t μ_t / Σ w_t N_t.
- The survival function S(y), the share of the group's households with income above y, is S(y) = Σ w_t N_t S_t(y) / Σ w_t N_t, where S_t(y) = 1 − Φ((ln y − m_t) / σ_t).
- The group's Gini index is G = 1 − (∫₀^∞ S(y)² dy) / μ.
Computation
ai_core.acs.gini.lognormal_curves() evaluates each tract's S_t at grid_points (257)
incomes, evenly spaced in log income from lowest_income ($1) to highest_income
($100,000,000). A group's S at those incomes is the weighted mean of its tracts' values.
lognormal_mixture_gini() integrates S² over the grid with Simpson's rule. The integral
below the lowest income L and above the highest income H comes from closed forms of each
tract's expected shortfall below L and expected excess above H:
- ∫₀^L S² dy ≈ L − 2P(L) + F(L) P(L), where F = 1 − S and P(L) is the mixture's expected shortfall E[(L − Y)⁺].
- ∫_H^∞ S² dy ≈ S(H) C(H), where C(H) is the mixture's expected excess E[(Y − H)⁺].
For a single log-normal with a Gini index from 0.2 to 0.8 and a mean from $8,000 to $400,000, the computed Gini index is within 0.00001 of the closed form. The grid spacing is 0.072 in log income. A tract whose Gini index is below about 0.05 has a distribution narrower than the grid resolves, so a group made only of part of such a tract has a Gini index about 0.03 too high.
Each row of the join of the weights to the tracts holds the tract's S_t at every grid
income, so aggregate_acs_features() computes the mixture with the Polars streaming
engine in batches of 10,000 rows.
Excluded geographies
A tract has no log-normal distribution, and is left out of every group, when:
- its number of households is null or 0;
- its aggregate income is null or not positive;
- its G_t is null or outside (0, 1), the range of a log-normal distribution's Gini index.
Accuracy
The eval packages/ai_core/evals/acs_gini/partial_tracts.py estimates the Gini index of
each ZIP Code Tabulation Area (ZCTA), state legislative district (lower, SLDL; upper,
SLDU) and 119th congressional district (CD) from the 2024 tracts it overlaps. Each tract's
weight is the share of its area inside the geography. The eval compares each estimate
with the geography's published Gini index, for geographies with at least 500 households.
Run it by hand from packages/ai_core:
uv run python -m evals.acs_gini.partial_tracts --data-dir <directory>
"Bracket" is the method of Bracket Gini index, applied to the area-weighted sums of the tracts' brackets and aggregate income.
| Geographies | Method | Bias | Mean absolute error | 95th percentile |
|---|---|---|---|---|
| ZCTA (21,181) | Bracket | +0.0030 | 0.0191 | 0.0651 |
| ZCTA | Log-normal mixture | +0.0032 | 0.0178 | 0.0635 |
| SLDL (4,722) | Bracket | −0.0016 | 0.0077 | 0.0232 |
| SLDL | Log-normal mixture | −0.0012 | 0.0066 | 0.0214 |
| SLDU (1,922) | Bracket | −0.0023 | 0.0054 | 0.0150 |
| SLDU | Log-normal mixture | −0.0019 | 0.0044 | 0.0126 |
| CD (432) | Bracket | −0.0024 | 0.0038 | 0.0074 |
| CD | Log-normal mixture | −0.0030 | 0.0031 | 0.0055 |
The bracket method's bias depends on income. It overestimates the Gini index of geographies with a high mean income or a large share of households in the top bracket. In the districts, it underestimates the Gini index of geographies with a low mean income. The log-normal mixture has no such trend in the districts.
Bias by the geography's mean household income:
| Mean income | ZCTA bracket | ZCTA mixture | SLDL bracket | SLDL mixture | SLDU bracket | SLDU mixture |
|---|---|---|---|---|---|---|
| Under $60,000 | +0.0020 | +0.0073 | −0.0061 | −0.0003 | −0.0085 | −0.0022 |
| $60,000 to $100,000 | +0.0023 | +0.0048 | −0.0034 | −0.0010 | −0.0039 | −0.0017 |
| $100,000 to $150,000 | +0.0007 | +0.0012 | −0.0015 | −0.0014 | −0.0021 | −0.0021 |
| $150,000 and over | +0.0120 | −0.0010 | +0.0068 | −0.0022 | +0.0060 | −0.0028 |
Bias by the share of households with an income of $200,000 or more:
| Top-bracket share | ZCTA bracket | ZCTA mixture | SLDL bracket | SLDL mixture | SLDU bracket | SLDU mixture |
|---|---|---|---|---|---|---|
| Under 5% | +0.0060 | +0.0094 | −0.0036 | −0.0003 | −0.0044 | −0.0011 |
| 5% to 15% | −0.0016 | +0.0002 | −0.0028 | −0.0012 | −0.0033 | −0.0017 |
| 15% to 30% | +0.0015 | −0.0004 | +0.0001 | −0.0021 | −0.0007 | −0.0030 |
| 30% and over | +0.0201 | +0.0009 | +0.0115 | −0.0021 | +0.0104 | −0.0027 |
The log-normal mixture takes each tract's inequality from its published Gini index, which the Census Bureau computes from finer income data than the published brackets.
Error sources
Sampling error. ACS values are survey estimates. The published B19083_001E of a
tract has a median margin of error of 0.056 at 90% confidence, and B19025_001E has a
relative margin of error of about 19% at the tract level. In a group of many tracts,
these errors partly cancel. In a group of few tracts, they do not. The published Gini
index of a ZCTA has a median margin of error of 0.014, so part of the ZCTA error in the
tables above is the published value's own error.
Area weighting. An area weight assumes that a tract's households are spread evenly over its area. Households are not spread evenly, so a geography that contains a share of a tract's area can contain a different share of its households. A partial tract has less than 99% of its area inside the geography. The log-normal mixture's mean absolute error grows with the share of a geography's households that come from partial tracts:
| Households from partial tracts | ZCTA | SLDL | SLDU |
|---|---|---|---|
| Under 10% | 0.0030 | 0.0028 | 0.0028 |
| 10% to 50% | 0.0079 | 0.0056 | 0.0044 |
| 50% and over | 0.0233 | 0.0124 | 0.0122 |
Bracket Gini index
ai_core.acs.gini.bracket_gini() estimates a geography's Gini index from its household
income brackets B19001_002E to B19001_017E and its aggregate household income
B19025_001E. lognormal_mixture_gini uses it, as its fallback, for a tract whose
published Gini index is null. The method is the mean-matching approach of
von Hippel et al. 2017, with these choices:
-
The mean household income μ is the aggregate divided by the number of households.
-
Within each bounded bracket, income has a uniform density.
-
In the open-ended top bracket, "$200,000 or more", income has a Pareto density. Its mean m is the one that makes the whole distribution's mean equal μ: the aggregate minus each bounded bracket's count times its midpoint, divided by the top bracket's count. The Pareto index is α = m / (m − 200,000). m is at least
min_top_mean_ratio(1.001) times 200,000, which keeps α finite and positive when the bounded brackets alone account for the aggregate. -
The Gini index is 1 − ∫S(y)² dy / mean, where S is the share of households with income above y and the mean is that of the fitted distribution. The mean equals μ except in two cases:
- When m is raised to its lower limit, the mean is above μ.
- When the top bracket's count is 0, the mean is that of the bounded brackets' midpoints, and the aggregate is not used.
Over a bounded bracket of width w, where S falls linearly from a to b, the integral is w(a² + ab + b²)/3. Over the top bracket it is S(200,000)² × 200,000 / (2α − 1).
Against each 2024 tract's published B19083_001E, the estimate has a mean absolute error
of 0.008 and a bias of +0.001. It is sensitive to the aggregate. A 2% change in
B19025_001E moves it by about 0.010.