CloudQuery: product and architecture
CloudQuery is an open-source ELT framework that extracts data from cloud APIs, databases, and SaaS applications into data warehouses and data lakes for analysis and governance. In this CloudQuery review, we examine how the platform provides multi-cloud visibility for governance and platform teams, and how it compares to alternatives like Steampipe, Fivetran, and Airbyte.
Overview
CloudQuery (cloudquery.io) is an open-source ELT framework originally focused on cloud infrastructure data but now expanded to cover any API-based data source. The platform extracts data from 70+ cloud and SaaS sources (AWS, GCP, Azure, GitHub, Kubernetes, Cloudflare, Okta, and many more) and loads it into destinations like PostgreSQL, BigQuery, Snowflake, S3, and ClickHouse.
The primary use case is cloud governance and visibility: platform teams extract their cloud resource inventory into a data warehouse, then use SQL to build compliance dashboards, cost analysis reports, and security audits. CloudQuery positions itself as "multi-cloud visibility and automation for governance and platform teams."
CloudQuery is built in Go with a plugin-based architecture — source plugins extract data, destination plugins load it, and the framework handles scheduling, incremental syncs, and schema management. The project has 5,800+ GitHub stars and an active community.
Key Features and Architecture
Plugin-Based Architecture
CloudQuery uses a source/destination plugin model: source plugins extract data from APIs (AWS, GCP, Azure, GitHub, Kubernetes, etc.), and destination plugins load data into warehouses (PostgreSQL, BigQuery, Snowflake, ClickHouse, S3). Plugins are independent binaries, so adding a new source doesn't require modifying the core framework.
70+ Source Plugins
Sources cover cloud providers (AWS with 200+ tables, GCP, Azure, Oracle Cloud), developer tools (GitHub, GitLab, Terraform, Cloudflare), identity (Okta, Auth0), monitoring (Datadog, PagerDuty), and many more. Each source plugin maps API resources to relational tables with consistent schemas.
Incremental Syncs
CloudQuery supports incremental extraction — only fetching resources that changed since the last sync. This reduces API calls, sync time, and costs for large cloud environments with thousands of resources.
SQL-Based Policy and Compliance
Once cloud data is in a warehouse, teams write SQL queries to check compliance policies: "find all S3 buckets without encryption," "list IAM users without MFA," or "show EC2 instances running for more than 90 days." CloudQuery provides pre-built policy packs for CIS benchmarks, SOC 2, and other frameworks.
Transformation with dbt
CloudQuery integrates with dbt for transforming raw cloud data into analytics-ready models. Pre-built dbt packages provide common transformations like cost allocation, security posture scoring, and resource tagging compliance.
Multi-Cloud Normalization
For organizations running across AWS, GCP, and Azure, CloudQuery normalizes resource data into consistent schemas, enabling cross-cloud queries like "show me all compute instances across all clouds sorted by cost."
Ideal Use Cases
Cloud Security and Compliance
Security teams extract cloud resource inventories into a warehouse and run SQL-based compliance checks against CIS benchmarks, SOC 2 controls, or custom policies. This provides continuous compliance monitoring without relying on cloud-provider-specific tools.
Cloud Cost Optimization
FinOps teams extract billing and resource data from multiple cloud providers, combine it in a warehouse, and build cost allocation dashboards, idle resource reports, and rightsizing recommendations using SQL and dbt.
Platform Engineering Visibility
Platform teams managing Kubernetes clusters, Terraform state, and cloud resources across multiple accounts use CloudQuery to build a unified inventory. This enables questions like "which teams are running the most expensive resources?" or "how many clusters are running outdated Kubernetes versions?"
Multi-Cloud Governance
Organizations operating across AWS, GCP, and Azure use CloudQuery's normalized schemas to apply consistent governance policies across all clouds from a single SQL interface.
Strengths & Trade-offs
Pros
- 70+ cloud and SaaS sources — covers AWS (200+ tables), GCP, Azure, GitHub, Kubernetes, Cloudflare, Okta, and many more
- Open-source core — free to use with full source code access; no vendor lock-in on the extraction framework
- SQL-based governance — extract cloud data into warehouses and use SQL for compliance, cost, and security analysis
- dbt integration — pre-built dbt packages for common cloud governance transformations
- Lightweight deployment — single binary, minimal infrastructure; simpler than running Airbyte or Fivetran self-hosted
- Multi-cloud normalization — consistent schemas across AWS, GCP, and Azure for cross-cloud queries
Cons
- Niche focus — primarily designed for cloud infrastructure and API data; not a general-purpose ELT tool for application databases
- Plugin quality varies — plugin maturity and completeness vary across sources; AWS is comprehensive, small sources may have gaps
- No built-in transformation — relies on dbt or SQL for transformations; no visual transformation builder
- Limited community — 5,800+ GitHub stars is healthy but trails Airbyte (15,000+) or dbt (9,000+)
- MPL 2.0 license — has stricter terms than Apache 2.0; some organizations have concerns about copyleft provisions
Getting Started
Getting started with CloudQuery is straightforward. Visit the official website to create a free account or download the application. The onboarding process typically takes under 5 minutes, and most users can be productive within their first session. For teams evaluating CloudQuery against alternatives, we recommend a 2-week trial period to assess whether the feature set and user experience align with your specific workflow requirements. Documentation and community resources are available to help with initial setup and configuration.