Cost & Performance Comparison of Data Orchestration on Azure; Data Analysis of Data!

Question

New Project: Cost & Performance Comparison of Data Orchestration on Azure

I recently completed an experimental study comparing runtime and cost efficiency across three common data engineering orchestration approaches on Azure:

• Databricks Notebook Clusters
• Azure Data Factory → Databricks
• Databricks Workflows (multi-notebook pipelines)

Instead of theoretical claims, I used measured execution times, a fixed cluster configuration (Standard_D4s_v3), and then scaled the results to 1 TB of data and 100 hours of processing using clearly stated assumptions.

Key findings:

✅ For small datasets, orchestration overhead dominates (ADF & Workflows are slower)
✅ At TB-scale, all approaches converge in runtime — architecture matters more than speed
✅ Cost differences are marginal at scale, but orchestration choices matter for long-running workloads
✅ Databricks Workflows are ideal for Spark-heavy pipelines
✅ ADF still makes sense when orchestrating multiple Azure services

This project helped me better understand where architectural decisions actually matter — beyond “ADF vs Databricks” debates.

GitHub repo with experimental results, tables, and screenshots:
https://github.com/usman19zafar/Data-Engineering-Cost-Performance-Comparison

Happy to hear thoughts from other data engineers — especially how you decide orchestration strategy in production systems.