A comprehensive guide to every layer of the modern data stack — ingestion, warehousing, transformation, orchestration, BI, data quality, reverse ETL, and streaming — with real tool recommendations and pricing.
EB
Egor Burlakov
••11 min read
The modern data stack (MDS) has matured from a buzzword into a well-defined architecture that thousands of data teams run in production. But the landscape has shifted significantly — the "classic" MDS of 2021 (Fivetran + Snowflake + dbt + Looker) has evolved into something more nuanced, with new categories emerging and old assumptions being challenged.
This guide maps every layer of the modern data stack as it exists in 2026, with real tool recommendations, pricing context, and honest assessments of what works and what's overhyped.
What Is the Modern Data Stack?
The modern data stack is a collection of cloud-native tools that handle the full data lifecycle: collecting data from sources, storing it in a cloud warehouse, transforming it into useful models, and making it available for analysis and action. The key principles:
Cloud-native: Everything runs in the cloud — no on-premise servers to manage
SQL-centric: SQL is the primary language for transformation and analysis
Modular: Best-of-breed tools at each layer, connected through standard interfaces
ELT over ETL: Load raw data first, transform it in the warehouse (not before loading)
Warehouse as the hub: The cloud data warehouse is the central source of truth
The 8 Layers of the Modern Data Stack
Layer 1: Data Ingestion (ELT)
What it does: Extracts data from source systems (SaaS apps, databases, APIs) and loads it into your warehouse.
The shift in 2026: The ingestion layer has commoditized. Most tools support 300+ connectors and the core functionality is similar. The differentiation is now in pricing, connector quality for niche sources, and CDC (Change Data Capture) capabilities.
Top tools:
Fivetran — The market leader with 500+ connectors. Fully managed, zero maintenance. Pricing starts at $1/credit (~$2,000/month for mid-size). Best for teams that want reliability and don't mind paying for it.
Airbyte — The open-source leader with 350+ connectors. Self-hosted is free; Airbyte Cloud starts at $2.50/credit. Best for cost-conscious teams and those needing custom connectors.
dlt (data load tool) — Python-first ingestion library. Write dlt.pipeline() in Python and load data from any API. Free and open-source. Best for data engineers who prefer code over UI.
Meltano — Open-source ELT platform built on Singer taps. Free, CLI-driven, Git-native. Best for teams that want infrastructure-as-code for their pipelines.
Stitch — Talend's managed ELT service. Simple, affordable ($100/month for 5M rows). Best for small teams with straightforward needs.
Our take: If budget isn't a constraint, Fivetran is the safe choice. If you want to save 50-70%, Airbyte (self-hosted) or dlt give you comparable functionality. See our Fivetran vs Airbyte vs Stitch comparison for a detailed breakdown.
Layer 2: Cloud Data Warehouse
What it does: Stores all your data and provides the compute engine for queries and transformations.
The shift in 2026: The "big three" (Snowflake, BigQuery, Databricks) dominate, but the lines between them have blurred. Snowflake added ML features, Databricks added SQL warehousing, and BigQuery added everything. The choice increasingly depends on your cloud provider and existing ecosystem.
Top tools:
Snowflake — The independent cloud data warehouse. Separation of storage and compute, per-second billing, cross-cloud support. Starts at $2/credit (~$23/hour for a small warehouse). Best for multi-cloud organizations and teams that want the most mature SQL warehouse.
Google BigQuery — Serverless warehouse with no cluster management. $6.25/TB scanned (on-demand) or capacity pricing. Best for Google Cloud-native teams and those who want true serverless.
Databricks — The lakehouse platform combining warehouse and data lake. $0.07–$0.55/DBU. Best for teams that need both SQL analytics and ML/AI workloads on the same platform.
Amazon Redshift — AWS's warehouse, now with serverless option. From $0.36/hour. Best for AWS-native teams, especially those already using the AWS data ecosystem.
ClickHouse — Open-source columnar database for real-time analytics. Free self-hosted; ClickHouse Cloud from $0.30/hour. Best for real-time analytics dashboards with sub-second queries.
What it does: Transforms raw data in the warehouse into clean, modeled tables ready for analysis.
The shift in 2026: dbt remains dominant but faces real competition for the first time. SQLMesh's virtual environments and column-level lineage address pain points that dbt hasn't solved. Dataform is free for BigQuery users.
Top tools:
dbt — The industry standard. SQL + Jinja templating, 4,000+ community packages, massive ecosystem. dbt Core is free; dbt Cloud from $100/developer/month. Best for most teams — the ecosystem and hiring pool are unmatched.
SQLMesh — The challenger with virtual environments, column-level lineage, and incremental-by-default models. Free and open-source. Best for teams with large datasets where full rebuilds are expensive.
Dataform — Google's transformation tool, free with BigQuery. Best for BigQuery-only teams who want zero additional cost.
Our take: dbt is the safe default. SQLMesh is worth evaluating if you're spending a lot on warehouse compute for full table rebuilds. See our dbt vs Dataform vs SQLMesh comparison.
Layer 4: Data Orchestration
What it does: Schedules and coordinates data pipelines — ensuring transformations run in the right order, at the right time, with proper error handling.
The shift in 2026: Airflow is still the most deployed orchestrator, but Dagster and Prefect have captured significant market share with better developer experience. The "orchestrator wars" have settled into three clear tiers.
Top tools:
Apache Airflow — The incumbent with 37,000+ GitHub stars and 2,500+ contributors. Free and open-source. Managed options: Astronomer ($400+/month), MWAA (AWS), Cloud Composer (GCP). Best for teams that want the largest ecosystem and most battle-tested option.
Dagster — Software-defined assets with built-in data lineage and testing. Free open-source; Dagster Cloud from $0. Best for teams that want a modern, asset-centric approach to orchestration.
Prefect — Python-native orchestration with minimal boilerplate. Free open-source; Prefect Cloud from $0. Best for Python-heavy teams that find Airflow's DAG syntax cumbersome.
Kestra — YAML-based orchestration with a visual editor. Free and open-source. Best for teams that want declarative pipeline definitions without writing Python.
Our take: Airflow if you need the ecosystem, Dagster if you're starting fresh and want the best developer experience. See our Airflow vs Dagster vs Prefect comparison.
Layer 5: Business Intelligence
What it does: Visualizes data through dashboards, reports, and self-service exploration for business users.
The shift in 2026: The BI market has split into two camps: enterprise platforms (Looker, Tableau, Power BI) for governed, organization-wide analytics, and lightweight tools (Metabase, Superset, Evidence) for fast, developer-friendly dashboards.
Top tools:
Looker — Google's enterprise BI with LookML semantic layer. Best for organizations that want governed, consistent metrics across teams. Pricing: ~$5,000+/month.
Tableau — The visual analytics leader with the most powerful drag-and-drop interface. Best for analysts who need advanced visualizations. Pricing: $70/user/month (Creator).
Power BI — Microsoft's BI tool with deep Office 365 integration. Best for Microsoft-ecosystem organizations. Pricing: $10/user/month (Pro).
Metabase — Open-source BI that non-technical users can actually use. Best for startups and teams that want self-serve analytics without enterprise complexity. Free self-hosted.
Apache Superset — Open-source BI with SQL-first approach. Best for technical teams that want free, customizable dashboards. Free self-hosted.
What it does: Monitors data pipelines and warehouse tables for anomalies, freshness issues, schema changes, and quality problems.
The shift in 2026: Data observability has matured from "nice to have" to essential. The category has consolidated around a few leaders, and open-source options have become production-ready.
Top tools:
Great Expectations — Open-source data validation framework. Define expectations in Python, run them in pipelines. Free; GX Cloud for managed features. Best for teams that want programmatic data testing.
Monte Carlo — The market leader in data observability. Automated monitoring, lineage, and incident management. Enterprise pricing (~$50K+/year). Best for large data teams that need automated anomaly detection.
Soda — Data quality checks defined in YAML (SodaCL). Free open-source; Soda Cloud from $200/month. Best for teams that want simple, declarative data quality checks.
OpenMetadata — Open-source data catalog with built-in quality monitoring. Free; Collate Cloud for managed hosting. Best for teams that want a unified catalog + quality platform.
Layer 7: Reverse ETL & Data Activation
What it does: Syncs data from the warehouse back to business tools (CRMs, ad platforms, email tools) — the "last mile" of the data stack.
The shift in 2026: Reverse ETL has become a standard layer. The debate is no longer "do we need this?" but "which tool?" CDPs (Segment, RudderStack) are converging with reverse ETL tools (Hightouch, Census).
Top tools:
Hightouch — The reverse ETL leader with 200+ destinations and Customer Studio for audience building. Free tier (1 destination); Pro from $350/month. Best for marketing-heavy teams that need audience activation.
Census — Reverse ETL with strong dbt integration and developer experience. Free tier (1 destination); Core from $300/month. Best for data engineering teams that want SQL-first activation.
Segment — The CDP that also does reverse ETL. 400+ destinations, identity resolution. Free tier (1K visitors); Team from $120/month. Best for teams that need both event collection and data activation.
RudderStack — Open-source CDP with warehouse-first architecture. Free self-hosted; Cloud from $450/month. Best for teams that want Segment's functionality without the cost.
Our take:Hightouch or Census for pure reverse ETL, Segment or RudderStack if you also need event collection. See our tool reviews for detailed comparisons.
Layer 8: Data Streaming (Real-Time)
What it does: Processes data in real time as events happen, enabling sub-second analytics, real-time features, and event-driven architectures.
The shift in 2026: Streaming has moved from "advanced" to "expected" for many use cases. Confluent has made Kafka accessible, and Redpanda has emerged as a simpler alternative.
Top tools:
Apache Kafka — The standard for event streaming. 27,000+ GitHub stars, proven at LinkedIn (7T messages/day). Free and open-source. Best for teams with Kafka expertise.
Confluent — Managed Kafka by its creators. 120+ connectors, Schema Registry, ksqlDB. Cloud from $0 (first $400 free). Best for teams that want Kafka without the operational burden.
Redpanda — Kafka-compatible, written in C++, no JVM. 10x lower latency claimed. Best for teams that find Kafka operationally painful.
Over-tooling early: You don't need 8 layers on day one. Start with warehouse + dbt + a BI tool. Add layers as pain points emerge.
Ignoring data quality: The most common regret. Add Great Expectations or Soda early — fixing data quality retroactively is 10x harder.
Choosing tools for the resume: Pick tools that match your team's skills, not what's trending on Twitter.
Skipping the semantic layer: Without consistent metric definitions (via dbt metrics, Looker LookML, or a dedicated semantic layer), every team calculates "revenue" differently.
Conclusion
The modern data stack in 2026 is mature, well-defined, and more accessible than ever. Open-source tools at every layer mean a startup can build a production data stack for under $1,000/month. The key decisions are: which warehouse (Snowflake vs BigQuery vs Databricks), which transformation tool (dbt vs SQLMesh), and which orchestrator (Airflow vs Dagster vs Prefect). Everything else follows from those choices.
Engineering and Science Leader with experience building scalable data infrastructure, data pipelines and science applications. Sharing insights about data tools, architecture patterns, and best practices.
Explore Further
Dive deeper into the tools and categories mentioned in this article.