// concepts & guides

Data Engineering Concepts

In-depth guides to every core concept — architecture patterns, modeling techniques, streaming systems, and more.

architecture8 min read

ETL vs ELT

The fundamental architectural choice in modern data pipelines. When to transform before loading vs after.

ETLELTarchitecture
ingestion12 min read

Change Data Capture (CDC)

Capture only the rows that changed in your source database using transaction logs — no full scans.

CDCDebeziumKafka
modeling15 min read

Slowly Changing Dimensions (SCD)

How to handle historical tracking when dimension attributes change over time. Types 1, 2, 3, and 6.

SCDType 2dimensional modeling
architecture10 min read

Data Lakehouse Architecture

Combining data lake flexibility with warehouse ACID guarantees using Delta Lake or Apache Iceberg.

lakehouseDelta LakeIceberg
streaming14 min read

Stream Processing

Real-time data processing with event-time semantics, watermarks, and stateful computations.

KafkaFlinkSpark Streaming
modeling18 min read

Data Modeling for Analytics

Star schema, snowflake schema, Data Vault 2.0, and One Big Table — when to use each approach.

star schemaKimballData Vault
quality11 min read

Data Quality & Observability

Completeness, accuracy, consistency checks. Anomaly detection, schema monitoring, and lineage.

Great Expectationsdbt testsMonte Carlo
architecture16 min read

Data Mesh

Decentralized data architecture where domain teams own their data products end-to-end.

data meshdomain ownershipdata products
orchestration13 min read

Pipeline Orchestration

DAGs, scheduling, dependency management. Airflow vs Prefect vs Dagster — a practical comparison.

AirflowPrefectDagster