PySpark & Notebook Templates

The notebooks you’d otherwise write from scratch for every lakehouse — idempotent, parameter-driven, and testable. Logic lives in lib/ so the notebook stays thin and the transforms run under pytest locally. Matches the Data Engineering docs.

What’s included

• incremental_load.py — watermark read + idempotent MERGE upsert, returns the new watermark to the pipeline
• scd2_merge.py — Slowly Changing Dimension Type 2 with an attribute hash
• data_cleansing_framework.py — trim / coerce / default / dedupe / validate, splitting clean vs. quarantine rows
• delta_maintenance.py — OPTIMIZE (+ Z-ORDER), V-Order, target file size, VACUUM, driven by a control table
• lib/fabric_utils.py — structured logging, boundary assertions, idempotency markers, notebookutils guards
• lib/transforms.py — pure transformation functions, no I/O
• tests/ — a local SparkSession fixture and pytest coverage for every transform
• config/tables.example.json — one control file for loads, cleansing, and maintenance

File tree

README.md
config/tables.example.json
lib/fabric_utils.py
lib/transforms.py
notebooks/data_cleansing_framework.py
notebooks/delta_maintenance.py
notebooks/incremental_load.py
notebooks/scd2_merge.py
tests/conftest.py
tests/test_transforms.py

Using them in Fabric

1. Import each notebooks/ and lib/ file as a Fabric notebook (or attach lib/ as a notebook resource).
2. Mark the top cell of each notebook as the parameter cell.
3. Orchestrate with a data pipeline — one Notebook activity per table, or a ForEach over config/tables.example.json.
4. Run the tests locally: pip install pyspark pytest && pytest -q.

Updated for new Fabric Runtimes — subscribers hear about changes in the change briefing.