Great Expectations: product and architecture
Great Expectations is the open-source framework that brought data quality testing to the data engineering mainstream. If you work with data pipelines and want the confidence that your data meets defined standards before it flows downstream, this great expectations data quality review covers everything you need to evaluate the tool. Great Expectations lets you define expectations for your data in Python, such as row counts, null percentages, value ranges, and distribution checks, and then validates them automatically in your pipeline. With over 11,400 GitHub stars, an Apache 2.0 license, and active development through version 1.16.1, Great Expectations has earned its position as one of the most adopted data quality frameworks in the Python ecosystem. We tested GX across SQL, Pandas, and Spark backends to evaluate its strengths, limitations, and where it fits in the modern data stack.
Overview
Great Expectations (commonly referred to as GX) is an open-source Python framework for data validation and documentation. It was created to solve a fundamental gap in data engineering: most teams test their code but not their data. GX addresses this by providing a declarative system for defining, executing, and documenting expectations about datasets.
The framework has grown from a community project into a full platform offering. GX Core is the open-source library available under the Apache 2.0 license, with over 11,400 stars on GitHub and topics spanning data quality, data profiling, pipeline testing, and MLOps. The latest release is version 1.16.1, published in April 2026, reflecting consistent and active maintenance. GX Cloud is the managed SaaS platform built on top of the open-source core, offering a hosted experience with collaboration tools, observability dashboards, and real-time monitoring.
GX targets data engineers, analytics engineers, and data scientists who need programmatic control over data quality. It fits naturally into Python-based data workflows and integrates with orchestration tools like Airflow, Dagster, and Prefect. The framework has been described as the "pytest of data quality" because it makes testing your data feel as natural as testing your code.
Key Features and Architecture
Expectation Suites are the core building block. An Expectation Suite is a collection of reusable data rules, such as expect_column_values_to_not_be_null, expect_column_mean_to_be_between, or expect_table_row_count_to_be_between. You define these in Python and apply them to any dataset. GX ships with over 300 built-in expectation types, and you can write custom expectations for domain-specific logic.
Multi-backend execution means your expectations run wherever your data lives. GX supports SQL databases (PostgreSQL, MySQL, BigQuery, Snowflake, Redshift, Databricks), Pandas DataFrames, and Apache Spark. You write expectations once and execute them against any supported backend without changing your validation logic.
Data Docs is the auto-generated documentation system. Every validation run produces a browsable HTML report showing which expectations passed, which failed, and the observed values. This documentation becomes a living artifact of your data quality posture, shareable with both technical and business stakeholders.
ExpectAI is the newer AI-powered feature in GX Cloud that auto-generates test expectations based on your data profiles. Instead of manually writing every expectation, ExpectAI analyzes your data and suggests appropriate validations, which you can accept, modify, or reject.
Pipeline integration works through checkpoints that you embed in your orchestration tool. In Airflow, you add a GX checkpoint operator that runs validations as a DAG task. If validations fail, the pipeline halts before bad data propagates. Similar integrations exist for Dagster, Prefect, and custom CI/CD pipelines.
The architecture follows a modular design: Data Sources connect to your data, Expectation Suites define your rules, Checkpoints orchestrate execution, and Data Docs render the results.
Ideal Use Cases
Great Expectations is best for data engineering teams running Python-based ETL/ELT pipelines who want to validate data at every stage. If you use Airflow, Dagster, or Prefect for orchestration and want programmatic data checks that integrate directly into your DAGs, GX is the strongest open-source option available.
It excels for teams building data contracts between producers and consumers. You define expectations that codify what downstream consumers require, and validation runs catch contract violations before they cause dashboard errors or model drift.
MLOps teams benefit from GX for training data validation. Before retraining a model, you validate that the input data meets distribution expectations, feature completeness requirements, and schema constraints.
GX is not suitable for real-time streaming validation. It operates in batch mode, running validations against snapshots of data. Teams needing sub-second validation on streaming data should look elsewhere. It is also not a full observability platform; it validates data quality but does not provide anomaly detection, lineage tracking, or data cataloging without additional tooling.
Strengths & Trade-offs
Pros:
- Fine-grained, explicit data checks with over 300 built-in expectation types covering nulls, ranges, distributions, and custom logic
- Documentation generated as a byproduct of validation, not a separate maintenance burden
- No vendor lock-in with Apache 2.0 licensing and multi-backend support across SQL, Pandas, and Spark
- Strong integration with orchestration tools including Airflow, Dagster, and Prefect
- Active development with 11,400+ GitHub stars and regular releases through version 1.16.1
- ExpectAI auto-generates tests, reducing the manual effort of writing expectations from scratch
Cons:
- Manual definition effort is significant for large datasets; you need to write and maintain expectation suites per table
- Not a full observability platform; you need separate tools for anomaly detection, lineage, and cataloging
- Requires external orchestration to run validations on a schedule; GX does not include a built-in scheduler
- Learning curve for the configuration layer, particularly around Data Contexts, Stores, and Checkpoint configurations