Dataset Loading
DatasetAssembler loads the datasets declared in PipelineConfig.datasets.
Without gcs_cache_prefix, it calls each loader directly. With a prefix, it reads
Parquet exports from GCS. parallel_io enables concurrent dataset operations.
Export before loading
The bmw_direct_rate.yaml configuration enables GCS reads. Export the datasets
before calling create_pipeline() or create_batch_pipelines() with this
configuration.
Given a loaded config and the complete list of scopes for the batch:
from candidate_selection.data_loaders.base import DatasetAssembler
from candidate_selection.factory import create_batch_pipelines
assembler = DatasetAssembler(
config.datasets,
parallel=config.parallel_io,
gcs_cache_prefix=config.gcs_cache_prefix,
)
assembler.export_all_bulk(scopes)
pipelines = create_batch_pipelines(config, scopes)
Export every scope the batch needs. Export paths identify the dataset and date; they do not identify the scope list. Batches using the same prefix and date share the exports. Use different prefixes for batches that need separate exports.
Export paths
The assembler writes and reads:
{gcs_cache_prefix}/{scope_date}/{dataset_name}/*.parquet
scope_date is the SelectionScope.date of the scopes being exported or loaded.
With gcs_cache_prefix set, the scopes passed to one export_all_bulk() or
load_all_bulk() call must all have the same date. Both methods raise ValueError when
the scopes have different dates or no date.
A configured prefix selects GCS reads immediately. Missing or unreadable exports
raise FileNotFoundError; the assembler does not query BigQuery to populate them.
Omit gcs_cache_prefix to load directly from the configured sources.
Export methods
A loader that implements build_export_query() exports through BigQuery
EXPORT DATA. All five loaders in bmw_direct_rate.yaml support this method.
Other loaders run load_bulk() and write 000000000000.parquet in the export
directory.