Data Engineering
Production PySpark on Fabric — patterns that survive Runtime upgrades and keep CU consumption predictable.
Fabric notebooks are the workhorse of the platform. The gap between a notebook that "works in the demo" and one that runs unattended for a year is: idempotency, observability, resource sizing, and staying ahead of Runtime deprecations.
Pages
Production PySpark patterns
Idempotent MERGE, schema evolution, parameterized notebooks, structured logging.
notebookutils reference
fs, notebook.run / runMultiple, credentials, lakehouse, session — with recipes.
Native Execution Engine
When NEE accelerates a query, when it falls back to the JVM, and how to cut CU.
Resource profiles
readHeavyForSpark, writeHeavy, and custom profiles for the workload shape.
Runtime upgrades
Surviving Spark / Delta version bumps without a fire drill.
Principles
Every notebook is a function. Inputs come from parameters, not hard-coded paths. Output is deterministic for the same inputs. Re-running it is always safe.
- Idempotent by default — use
MERGEkeyed on a business key, orreplaceWhereon a partition, never blindappend. - Fail loud, fail fast — assert row counts and schema at the boundaries; a job that writes zero rows silently is worse than one that errors.
- One notebook, one responsibility — orchestrate with pipelines, not with 1000-line notebooks.
- Measure CU, not wall-clock — a faster job on a bigger pool can cost more.
Stay ahead of Fabric changes
Fabric runtime changes, API updates, and deprecations. No spam, unsubscribe anytime.