CloudQuery

CloudQuery is an open-source ELT framework that extracts data from cloud APIs, databases, and SaaS applications into data warehouses and data lakes for analysis and governance. In this CloudQuery review, we examine how the platform provides multi-cloud visibility for governance and platform teams, and how it compares to alternatives like Steampipe, Fivetran, and Airbyte.

Overview

CloudQuery (cloudquery.io) is an open-source ELT framework originally focused on cloud infrastructure data but now expanded to cover any API-based data source. The platform extracts data from 100+ sources (AWS, GCP, Azure, GitHub, Kubernetes, Cloudflare, Okta, and many more) and loads it into destinations like PostgreSQL, BigQuery, Snowflake, S3, and ClickHouse.

The primary use case is cloud governance and visibility: platform teams extract their cloud resource inventory into a data warehouse, then use SQL to build compliance dashboards, cost analysis reports, and security audits. CloudQuery positions itself as "multi-cloud visibility and automation for governance and platform teams."

CloudQuery is built in Go with a plugin-based architecture — source plugins extract data, destination plugins load it, and the framework handles scheduling, incremental syncs, and schema management. The project has 5,800+ GitHub stars and an active community.

Key Features and Architecture

Plugin-Based Architecture

CloudQuery uses a source/destination plugin model: source plugins extract data from APIs (AWS, GCP, Azure, GitHub, Kubernetes, etc.), and destination plugins load data into warehouses (PostgreSQL, BigQuery, Snowflake, ClickHouse, S3). Plugins are independent binaries, so adding a new source doesn't require modifying the core framework.

100+ Source Plugins

Sources cover cloud providers (AWS with 200+ tables, GCP, Azure, Oracle Cloud), developer tools (GitHub, GitLab, Terraform, Cloudflare), identity (Okta, Auth0), monitoring (Datadog, PagerDuty), and many more. Each source plugin maps API resources to relational tables with consistent schemas.

Incremental Syncs

CloudQuery supports incremental extraction — only fetching resources that changed since the last sync. This reduces API calls, sync time, and costs for large cloud environments with thousands of resources.

SQL-Based Policy and Compliance

Once cloud data is in a warehouse, teams write SQL queries to check compliance policies: "find all S3 buckets without encryption," "list IAM users without MFA," or "show EC2 instances running for more than 90 days." CloudQuery provides pre-built policy packs for CIS benchmarks, SOC 2, and other frameworks.

Transformation with dbt

CloudQuery integrates with dbt for transforming raw cloud data into analytics-ready models. Pre-built dbt packages provide common transformations like cost allocation, security posture scoring, and resource tagging compliance.

Multi-Cloud Normalization

For organizations running across AWS, GCP, and Azure, CloudQuery normalizes resource data into consistent schemas, enabling cross-cloud queries like "show me all compute instances across all clouds sorted by cost."

Ideal Use Cases

Cloud Security and Compliance

Security teams extract cloud resource inventories into a warehouse and run SQL-based compliance checks against CIS benchmarks, SOC 2 controls, or custom policies. This provides continuous compliance monitoring without relying on cloud-provider-specific tools.

Cloud Cost Optimization

FinOps teams extract billing and resource data from multiple cloud providers, combine it in a warehouse, and build cost allocation dashboards, idle resource reports, and rightsizing recommendations using SQL and dbt.

Platform Engineering Visibility

Platform teams managing Kubernetes clusters, Terraform state, and cloud resources across multiple accounts use CloudQuery to build a unified inventory. This enables questions like "which teams are running the most expensive resources?" or "how many clusters are running outdated Kubernetes versions?"

Multi-Cloud Governance

Organizations operating across AWS, GCP, and Azure use CloudQuery's normalized schemas to apply consistent governance policies across all clouds from a single SQL interface.

Pricing and Licensing

CloudQuery open-source is free under the Mozilla Public License 2.0. CloudQuery Cloud offers managed capabilities:

Self-hosted CloudQuery runs as a single binary with minimal infrastructure requirements — just the CloudQuery CLI and a destination database. A typical setup costs $50–$200/month for the destination warehouse. For comparison, Steampipe is free (open-source), Fivetran starts at $1/credit (~$1,000+/month for moderate usage), and Airbyte Cloud starts at $0 with usage-based pricing.

Pros and Cons

Pros

• 100+ source plugins — covers AWS (200+ tables), GCP, Azure, GitHub, Kubernetes, Cloudflare, Okta, and many more
• Open-source core — free to use with full source code access; no vendor lock-in on the extraction framework
• SQL-based governance — extract cloud data into warehouses and use SQL for compliance, cost, and security analysis
• dbt integration — pre-built dbt packages for common cloud governance transformations
• Lightweight deployment — single binary, minimal infrastructure; simpler than running Airbyte or Fivetran self-hosted
• Multi-cloud normalization — consistent schemas across AWS, GCP, and Azure for cross-cloud queries

Cons

• Niche focus — primarily designed for cloud infrastructure and API data; not a general-purpose ELT tool for application databases
• Plugin quality varies — some source plugins are more mature and complete than others; AWS is comprehensive, smaller sources may have gaps
• No built-in transformation — relies on dbt or SQL for transformations; no visual transformation builder
• Smaller community — 5,800+ GitHub stars is healthy but significantly smaller than Airbyte (15,000+) or dbt (9,000+)
• MPL 2.0 license — more restrictive than Apache 2.0; some organizations have concerns about copyleft provisions

Getting Started

Getting started with CloudQuery is straightforward. Visit the official website to create a free account or download the application. The onboarding process typically takes under 5 minutes, and most users can be productive within their first session. For teams evaluating CloudQuery against alternatives, we recommend a 2-week trial period to assess whether the feature set and user experience align with your specific workflow requirements. Documentation and community resources are available to help with initial setup and configuration.

Alternatives and How It Compares

Steampipe

Steampipe is the closest alternative — an open-source tool that lets you query cloud APIs using SQL directly (without loading into a warehouse). Steampipe is better for interactive queries and real-time checks; CloudQuery is better for loading data into a warehouse for historical analysis, dashboards, and dbt transformations.

Fivetran

Fivetran ($1/credit) is a managed ELT platform with 500+ connectors focused on application data (Salesforce, Stripe, HubSpot, databases). Fivetran is better for traditional SaaS and database replication; CloudQuery is better for cloud infrastructure and API data that Fivetran doesn't cover.

Airbyte

Airbyte (open-source, Cloud with usage-based pricing) provides 350+ connectors for data integration. Airbyte covers some cloud sources but is primarily designed for application data. CloudQuery's cloud-native focus provides deeper AWS/GCP/Azure coverage with more tables and better schema normalization.

AWS Config / GCP Asset Inventory

Cloud-provider-native tools provide resource inventory within a single cloud. They're free but cloud-specific — no cross-cloud queries. CloudQuery provides a unified view across all clouds in your own warehouse.