Skip to main content

Dataset Loading

DatasetAssembler loads the datasets declared in PipelineConfig.datasets. Without gcs_cache_prefix, it calls each loader directly. With a prefix, it reads Parquet exports from GCS. parallel_io enables concurrent dataset operations.

Export before loading​

The bmw_direct_rate.yaml configuration enables GCS reads. Export the datasets before calling create_pipeline() or create_batch_pipelines() with this configuration.

Given a loaded config and the complete list of scopes for the batch:

from candidate_selection.data_loaders.base import DatasetAssembler
from candidate_selection.factory import create_batch_pipelines

assembler = DatasetAssembler(
config.datasets,
parallel=config.parallel_io,
gcs_cache_prefix=config.gcs_cache_prefix,
)
assembler.export_all_bulk(scopes)
pipelines = create_batch_pipelines(config, scopes)

Export every scope the batch needs. Export paths identify the dataset and date; they do not identify the scope list. Batches using the same prefix and date share the exports. Use different prefixes for batches that need separate exports.

Export paths​

The assembler writes and reads:

{gcs_cache_prefix}/{scope_date}/{dataset_name}/*.parquet

scope_date is the SelectionScope.date of the scopes being exported or loaded. With gcs_cache_prefix set, the scopes passed to one export_all_bulk() or load_all_bulk() call must all have the same date. Both methods raise ValueError when the scopes have different dates or no date.

A configured prefix selects GCS reads immediately. Missing or unreadable exports raise FileNotFoundError; the assembler does not query BigQuery to populate them. Omit gcs_cache_prefix to load directly from the configured sources.

Export methods​

A loader that implements build_export_query() exports through BigQuery EXPORT DATA. All five loaders in bmw_direct_rate.yaml support this method. Other loaders run load_bulk() and write 000000000000.parquet in the export directory.