// tools & platforms

Data Engineering Tools

Deep dives into every major tool — concepts, architecture, code examples, and interview questions.

🔌 Ingestion

Airbyte
freemium
Open-source ELT platform with 300+ connectors
ELTconnectorsopen-source
Debezium
free
Open-source CDC platform built on Apache Kafka
CDCKafkareal-time

⚙️ Transformation

AWS Glue
paid
Serverless ETL service on Amazon Web Services
serverlessAWSSpark
dbt (data build tool)
freemium
SQL-first data transformation framework
SQLELTtransformation

⚡ Processing

Apache Spark
free
Unified engine for large-scale data processing
big dataPySparkbatch
Databricks
paid
Unified analytics platform built on Apache Spark
SparkDelta LakeML

🎛️ Orchestration

Apache Airflow
free
Platform to programmatically author, schedule and monitor workflows
DAGorchestrationscheduling
Azure Data Factory
paid
Cloud-based data integration service by Microsoft
AzureETLorchestration

🏛️ Data Warehouse

Snowflake
paid
Cloud data warehouse with separated compute and storage
DWHcloudSQL

📡 Streaming

Apache Kafka
free
Distributed event streaming platform
streamingreal-timemessaging

🗄️ Databases

Oracle Database / Oracle Data Integrator
enterprise
Enterprise RDBMS and ETL platform
SQLRDBMSODI
SQL Server / SSIS
paid
Microsoft SQL Server and Integration Services
SQLSSISMicrosoft

🔗 ETL / Integration

IBM DataStage
enterprise
Enterprise-grade ETL and data integration
ETLIBMenterprise
Talend
freemium
Enterprise data integration and ETL platform
ETLGUIenterprise

💻 Programming

Python for Data Engineering
free
The universal language of modern data pipelines
ETLscriptingpandas