08 — Conditional & Coalescing Sources

Real intake data often records the same fact in more than one way. One survey asks for weight in pounds, another in kilograms, and stores whichever the respondent chose in a separate column. Height comes in as feet and inches, or as meters and centimeters. Age might be a number of years for adults and a number of months for infants. In each case the value you want is spread across several columns, only one of which is filled in for any given row — and the column you pick may need its own conversion before it fits the target.

Chapter 06 combined several columns with a single fixed rule. Here the choice of which column to use, and how to convert it, changes from row to row. This example introduces the two primitives that make that choice:

• case picks a branch by looking at a flag column that records the answer ("this row used pounds", "this row used the metric form").
• coalesce has no flag to consult, so it simply uses the first column that has a value in it.

What it teaches

• Choosing a branch from a flag with case. When the data already records which form was used, case reads that flag and follows the matching branch. A flag value with no matching branch (or a missing flag) falls through to a default.
• Choosing the populated column with coalesce. When there is no flag, coalesce walks the branches in order and uses the first whose column isn't empty.
• A different conversion per branch. Each branch carries its own steps, so the kilograms branch can convert to pounds while the pounds branch is left untouched. This is the part chapter 06's reduce couldn't do — it applies one set of steps to every input.
• Combining several columns inside one branch. A branch can read more than one column and fold them together. The height branches add a feet value (converted to inches) to an inches value, and the metric branch adds converted meters and centimeters.

How case and coalesce work

As with the multi-source rules in chapter 06, the framework hands the transformation the values from every column listed in sources. Both primitives refer to those columns by name, so the branches read clearly and don't break if the sources list is reordered.

• case looks at the selector column and runs the first branch whose when list contains that value.
• coalesce runs the first branch whose column has a non-empty value.
• A branch is either one column with its own steps (source + operations), or several operands — each a column with its own steps — folded together with combine. combine is one of the chapter-06 reductions (sum, any, none, all, one-hot); leave it off when a branch reads a single column.

A note on flag values. when values are compared as text. A flag column with some blank cells gets read as numbers with decimals (so 2 arrives as 2.0), and case accounts for this — a whole-number flag like 2.0 still matches when: ["2"], so you don't have to worry about how the file was loaded.

The data

input.csv — each row picks units via a flag, or reports age in one of two columns:

person_id	weight_units	weight_lbs	weight_kgs	height_units	height_ft	height_in	height_m	height_cm	age_years_raw	age_months_raw
P1	2 (lbs)	150		1 (ft+in)	5	7			34
P2	1 (kg)		68	2 (m+cm)			1	70		18
P3	2 (lbs)	200		1 (ft+in)	6	0			52
P4	1 (kg)		90	2 (m+cm)			1	80	7

The harmonization choices (and why)

Sources	Target	Pipeline	Why
weight_units + weight_lbs/weight_kgs	weight_lb	case on weight_units	Flag says which unit was used. "2" → lbs as-is; "1" → convert kg→lbs.
height_units + 4 height columns	height_inches	case, branches with combine: sum	Each branch sums two converted operands: ft→in + in, or m→in + cm→in.
age_years_raw / age_months_raw	age_years	coalesce	No flag; only one is filled. Years as-is, else months/12 rounded.

When to use case, and when to use coalesce

Both choose one column to read per row; what differs is how they decide. Reach for case when the data tells you which form was used — the weight and height rows each carry a unit flag, so there's no guessing. Reach for coalesce when nothing records the choice and the rule is just "use whatever was filled in" — the age column has no flag, so the populated field speaks for itself. case stays unambiguous even if a stray extra column happens to be filled, because the flag decides; coalesce settles that case by taking the earliest branch.

Why not the reduce from chapter 06?

Chapter 06's reduce won't fit here for two reasons. First, it treats an empty input as an error, but in this data exactly one of the candidate columns is empty on every row — that's normal, not a problem. Second, the two unit forms need different handling: kilograms convert to pounds, pounds are already correct. reduce (and map_each) apply the same steps to every input, so they can't make that distinction. case and coalesce give each branch its own steps, which is exactly what this needs.

The rules, serialized

As in earlier chapters, the saved file is the mapping, and the same rule set serializes to JSON (the default) or YAML — the extension decides which, and both load identically.

rules.yaml:

- sources: [weight_units, weight_lbs, weight_kgs]
  target: weight_lb
  operations:
  - operation: case
    sources: [weight_units, weight_lbs, weight_kgs]
    selector: weight_units
    branches:
    - when: ['2']
      combine: null
      operands:
      - source: weight_lbs
        operations:
        - {operation: do_nothing}
    - when: ['1']
      combine: null
      operands:
      - source: weight_kgs
        operations:
        - {operation: convert_units, source_unit: kg, target_unit: lb}
        - {operation: round, precision: 0}
    default: null
  metadata: {rationale: 'Respondent chose units (weight_units: 1=kg, 2=lbs)...'}

- sources: [height_units, height_ft, height_in, height_m, height_cm]
  target: height_inches
  operations:
  - operation: case
    sources: [height_units, height_ft, height_in, height_m, height_cm]
    selector: height_units
    branches:
    - when: ['1']
      combine: sum
      operands:
      - {source: height_ft, operations: [{operation: convert_units, source_unit: feet, target_unit: inch}]}
      - {source: height_in, operations: [{operation: do_nothing}]}
    - when: ['2']
      combine: sum
      operands:
      - {source: height_m,  operations: [{operation: convert_units, source_unit: m, target_unit: inch}, {operation: round, precision: 0}]}
      - {source: height_cm, operations: [{operation: convert_units, source_unit: cm, target_unit: inch}, {operation: round, precision: 0}]}
    default: null
  metadata: {rationale: 'Respondent chose units (height_units: 1=feet+inches, 2=meters+cm)...'}

- sources: [age_years_raw, age_months_raw]
  target: age_years
  operations:
  - operation: coalesce
    sources: [age_years_raw, age_months_raw]
    branches:
    - {combine: null, operands: [{source: age_years_raw, operations: [{operation: do_nothing}]}]}
    - combine: null
      operands:
      - {source: age_months_raw, operations: [{operation: scale, scaling_factor: 0.08333333333333333}, {operation: round, precision: 0}]}
    default: null
  metadata: {rationale: 'Age reported in years OR months, no flag. Coalesce takes the first non-null...'}

(The full, exact JSON and YAML are in rules.json / rules.yaml, regenerated by build_rules.py. A single-source branch normalizes to a one-operand list with combine: null.)

Running it

../../../harmonization-framework/venv/bin/python build_rules.py
../../../harmonization-framework/venv/bin/python run_python.py
../../../harmonization-framework/venv/bin/python run_yaml.py   # same, from rules.yaml
bash run_cli.sh

Expected output

...,weight_lb,height_inches,age_years,...
P1,...,150,67,34,...   # lbs as-is; 5ft7in; years as-is
P2,...,150,67,2,...    # 68kg -> 150lb; 1m70cm -> 67in; 18mo -> 2yr
P3,...,200,72,52,...   # 6ft0in -> 72in
P4,...,198,70,7,...    # 90kg -> 198lb; 1m80cm -> 70in; years as-is

Data files

Rendered from the committed CSV files in this example.

input.csv

person_id	weight_units	weight_lbs	weight_kgs	height_units	height_ft	height_in	height_m	height_cm	age_years_raw	age_months_raw
P1	2	150		1	5	7			34
P2	1		68	2			1	70		18
P3	2	200		1	6	0			52
P4	1		90	2			1	80	7

expected_output.csv

person_id	weight_units	weight_lbs	weight_kgs	height_units	height_ft	height_in	height_m	height_cm	age_years_raw	age_months_raw	weight_lb	height_inches	age_years	source dataset	original_id
P1	2	150.0		1	5.0	7.0			34.0		150.0	67.0	34.0	measurements	0
P2	1		68.0	2			1.0	70.0		18.0	150.0	67.0	2.0	measurements	1
P3	2	200.0		1	6.0	0.0			52.0		200.0	72.0	52.0	measurements	2
P4	1		90.0	2			1.0	80.0	7.0		198.0	70.0	7.0	measurements	3