EOEfe Peter Oroh
← All work

LFS Indigenous Data Pipeline

One pipeline that prepares 5M+ rows of Indigenous labour market data for several recurring reports.

Automation · R
Role
Built the pipeline
Stack
R
Data
Labour Force Survey
Scale
5M+ rows
Context

Why it mattered

The Labour Force Survey identifies Indigenous people living off reserve in the provinces, and several recurring analytical reports draw on that data.

Each report used to prepare its own extract by hand: the same filtering, weighting and aggregation repeated every cycle. I automated the extraction and processing into one pipeline that every report now reads from.

The question
Can one pipeline prepare Indigenous labour market data for every recurring report, so analysts start from finished tables instead of raw extracts?
  • Which steps were repeated across reports?
  • How do we keep every report on the same definitions?
Approach

How I solved it

01 · Extract

Pull once.

Survey extracts for every reference period are read in a single step.

Labour Force Survey extractsIndigenous identity fieldsReference periodsRaw extract5M+ rows
02 · Process

Same rules, every time.

Filtering, survey weighting, aggregation and quality checks are written once.

Filteridentity groupsWeightsurvey weightsAggregateby period,groupCheckquality rules
03 · Reuse

Many reports, one source.

Each report reads finished tables from the pipeline instead of building its own.

Sharedpipelineone templateone output set per reportreused every cycle
Outcome

What it delivers

5M+rows extracted and processed automatically
90%+less time spent preparing data for reports
1 pipelinefeeding several recurring reports
Demo

Demo

This pipeline runs on internal data, so there is no public demo.