04 — Dates¶

Dates are ambiguous to parse. 01/05/2025 could be the 5th of January or the 1st of May, and a parser that guesses can guess wrong, producing a date that is not the one in the source. The framework does not guess: you state the exact input format, and a value that doesn't match raises an error rather than being misread. This example reformats some shipment timestamps to canonical forms, and uses a malformed input to show the fail-fast behaviour.

What it teaches¶

Reformatting with convert_date. Parse a date with one format string and emit it in another.
Strict, fail-fast parsing. A value that doesn't match the declared source_format raises an error rather than being guessed at.
No fuzzy / multi-format parser. Each rule pins exactly one source format; a genuinely mixed column must be split and handled per format.

The data¶

input.csv — shipment timestamps in one consistent source format:

shipment_id	ordered_at	delivered_on
SH1	`2025-01-05 09:30:00`	`2025-01-08`
…	…	…

The harmonization choices (and why)¶

Source	Target	Conversion	Why
`ordered_at`	`order_date`	`%Y-%m-%d %H:%M:%S` → `%Y-%m-%d`	Drop time-of-day to a canonical date.
`ordered_at`	`order_month`	`%Y-%m-%d %H:%M:%S` → `%b %Y`	Month-grain label (`Jan 2025`) for rollups, from the same column.
`delivered_on`	`delivered_us`	`%Y-%m-%d` → `%m/%d/%Y`	US display format; source is date-only, so no time in the source format.

How the format strings work¶

Each convert_date takes two format strings: a source_format that says how to read the incoming value, and a target_format that says how to write it out. Both use Python's strftime/strptime codes, where each %-token stands for one part of a date or time — %Y is a four-digit year, %m a zero-padded month number, %d a zero-padded day, %H:%M:%S the time, and %b an abbreviated month name. So %Y-%m-%d %H:%M:%S reads 2025-01-05 09:30:00, and %b %Y writes the same instant back out as Jan 2025. The literal characters between the tokens — the dashes, slashes, and spaces — are matched and emitted verbatim, which is why %m/%d/%Y produces the slash-separated US form.

Two rules read the same column, ordered_at, with the same source_format but different target_formats, producing both a canonical order_date and a coarser order_month label from one source. The source format is a property of the input; the target format is a separate choice about how you want the result to look.

Fail-fast is a feature¶

convert_date does not try to be clever. Run it against bad_input.csv (a row with 01/05/2025 9:30 AM, which doesn't match %Y-%m-%d %H:%M:%S):

../../../harmonization-framework/venv/bin/python -c "
from harmonization_framework import RuleSet
from harmonization_framework.harmonize import harmonize_file
r=RuleSet(); r.load('rules.json', clean=True)
harmonize_file('bad_input.csv','/tmp/out.csv', r, dataset_name='events')
"

You get:

ValueError: Failed to parse date/time value '01/05/2025 9:30 AM' with source_format='%Y-%m-%d %H:%M:%S'

A silent misparse (guessing month vs. day) would corrupt data quietly; a loud error is the safer default for a meaningful field. If a column genuinely mixes formats, split it (e.g. by a flag column) and apply a convert_date per format — there is no single multi-format rule.

--on-missing is unrelated. That CLI flag governs missing columns, not malformed values. A bad date value always raises an error regardless of --on-missing.

The rules, serialized¶

As in chapter 01, the saved file is the mapping, and the same rule set serializes to JSON (the default) or YAML — the extension decides which, and both load identically. Shown in both formats below.

Worth noticing here is how convert_date captures its input and output formats in the file itself, so the parsing and the target representation are both explicit — there is no hidden assumption about how a date string is read or written.

rules.json:

[
  {
    "sources": [
      "ordered_at"
    ],
    "target": "order_date",
    "operations": [
      {
        "operation": "convert_date",
        "source_format": "%Y-%m-%d %H:%M:%S",
        "target_format": "%Y-%m-%d"
      }
    ],
    "metadata": {
      "rationale": "Reduce a full timestamp to a canonical date-only field; the time component is not needed downstream."
    }
  },
  {
    "sources": [
      "ordered_at"
    ],
    "target": "order_month",
    "operations": [
      {
        "operation": "convert_date",
        "source_format": "%Y-%m-%d %H:%M:%S",
        "target_format": "%b %Y"
      }
    ],
    "metadata": {
      "rationale": "Month-grain label for reporting/rollups, derived from the same timestamp."
    }
  },
  {
    "sources": [
      "delivered_on"
    ],
    "target": "delivered_us",
    "operations": [
      {
        "operation": "convert_date",
        "source_format": "%Y-%m-%d",
        "target_format": "%m/%d/%Y"
      }
    ],
    "metadata": {
      "rationale": "Reformat ISO date to US display format. Source is date-only, so the source_format has no time component."
    }
  }
]

rules.yaml:

- sources: [ordered_at]
  target: order_date
  operations:
  - {operation: convert_date, source_format: '%Y-%m-%d %H:%M:%S', target_format: '%Y-%m-%d'}
  metadata: {rationale: Reduce a full timestamp to a canonical date-only field; the
      time component is not needed downstream.}

- sources: [ordered_at]
  target: order_month
  operations:
  - {operation: convert_date, source_format: '%Y-%m-%d %H:%M:%S', target_format: '%b
      %Y'}
  metadata: {rationale: 'Month-grain label for reporting/rollups, derived from the
      same timestamp.'}

- sources: [delivered_on]
  target: delivered_us
  operations:
  - {operation: convert_date, source_format: '%Y-%m-%d', target_format: '%m/%d/%Y'}
  metadata: {rationale: 'Reformat ISO date to US display format. Source is date-only,
      so the source_format has no time component.'}

Running it¶

../../../harmonization-framework/venv/bin/python build_rules.py
../../../harmonization-framework/venv/bin/python run_python.py
../../../harmonization-framework/venv/bin/python run_yaml.py   # same, from rules.yaml
bash run_cli.sh

Expected output¶

...,order_date,order_month,delivered_us,...
...,2025-01-05,Jan 2025,01/08/2025,...

Data files¶

Rendered from the committed CSV files in this example.

`input.csv`¶

shipment_id	ordered_at	delivered_on
SH1	2025-01-05 09:30:00	2025-01-08
SH2	2025-02-14 17:05:00	2025-02-19
SH3	2025-03-30 23:59:00	2025-04-02
SH4	2025-11-01 06:00:00	2025-11-03

`expected_output.csv`¶

shipment_id	ordered_at	delivered_on	order_date	order_month	delivered_us	source dataset	original_id
SH1	2025-01-05 09:30:00	2025-01-08	2025-01-05	Jan 2025	01/08/2025	events	0
SH2	2025-02-14 17:05:00	2025-02-19	2025-02-14	Feb 2025	02/19/2025	events	1
SH3	2025-03-30 23:59:00	2025-04-02	2025-03-30	Mar 2025	04/02/2025	events	2
SH4	2025-11-01 06:00:00	2025-11-03	2025-11-01	Nov 2025	11/03/2025	events	3