Ray: product and architecture
Overview
Ray was created at UC Berkeley's RISELab in 2017 by Robert Nishihara and Philipp Moritz, with Ion Stoica (co-creator of Apache Spark) as advisor. The project has 35K+ GitHub stars and is one of the fastest-growing open-source AI infrastructure projects. Anyscale, the company behind Ray, has raised $260M+ in funding. Ray is used by OpenAI (for ChatGPT training infrastructure), Uber, Spotify, Instacart, Shopify, Netflix, and Ant Group. The framework processes over 1 exabyte of data monthly across its user base. Ray's core innovation is a universal distributed computing API that works for any Python workload — not just ML — making it the foundation for distributed AI applications. The 2.0+ releases added Ray AIR (AI Runtime) which unifies the ML-specific libraries under a consistent API.
Key Features and Architecture
Ray Core (Distributed Computing)
The foundation layer provides distributed task execution and actor-based programming. Any Python function can become a distributed task with @ray.remote, and any Python class can become a distributed actor. Ray's scheduler handles task placement, fault tolerance, and resource management across the cluster. The object store enables zero-copy data sharing between tasks on the same node.
Ray Train (Distributed Training)
Distributed model training with support for PyTorch DDP, DeepSpeed, Hugging Face Accelerate, TensorFlow, and XGBoost. Ray Train handles data parallelism, model parallelism, and pipeline parallelism across multi-GPU and multi-node setups. It provides automatic checkpointing, fault tolerance, and elastic training (adding/removing workers during training). OpenAI uses Ray Train for training large language models.
Ray Serve (Model Serving)
A scalable model serving framework that supports complex inference graphs with multiple models, business logic, and data processing steps. Ray Serve provides dynamic batching, model composition, and fractional GPU allocation. It can serve any Python-based model — not just ML frameworks — making it suitable for complex inference pipelines.
Ray Tune (Hyperparameter Optimization)
Distributed hyperparameter tuning with support for Bayesian optimization (via Optuna, HyperOpt), population-based training, and ASHA early stopping. Ray Tune scales trials across the cluster automatically and integrates with all major ML frameworks. It supports multi-objective optimization and can run thousands of trials in parallel.
Ray Data (Distributed Data Processing)
A distributed data processing library for ML workloads that handles data loading, preprocessing, and augmentation at scale. Ray Data provides streaming execution for datasets that don't fit in memory and integrates with Ray Train for end-to-end ML pipelines.
Ideal Use Cases
Large-Scale Model Training
Organizations training large models (LLMs, diffusion models, large vision models) that need distributed training across multiple GPUs and nodes. Ray Train's integration with DeepSpeed and FSDP makes it the go-to framework for distributed training. OpenAI, Cohere, and Anyscale use Ray for training models with billions of parameters across hundreds of GPUs.
Complex Inference Pipelines
Applications that need multi-model inference with business logic — for example, an e-commerce recommendation system that chains embedding generation, candidate retrieval, ranking, and filtering. Ray Serve handles these multi-step inference graphs with independent scaling per component.
Batch Inference at Scale
Processing millions of items through ML models — image classification, text embedding generation, or feature extraction. Ray Data + Ray Serve handle distributed batch inference with automatic scaling and fault tolerance. Spotify uses Ray for processing billions of audio tracks.
Hyperparameter Optimization
Teams running large-scale hyperparameter searches that need to parallelize hundreds of trials across a cluster. Ray Tune distributes trials automatically and supports advanced algorithms like population-based training.
Pricing and Licensing
Ray is distributed under an open source license, with no cost for the core software. This model aligns with industry benchmarks for distributed computing frameworks, where open source tools often provide free access to core functionality while reserving enterprise features, support, or specialized integrations for paid tiers. For data engineers and analytics leaders evaluating Ray, the absence of per-seat or usage-based licensing eliminates direct cost barriers, though total cost of ownership (TCO) depends on deployment complexity, infrastructure requirements, and the need for enterprise-grade support or compliance certifications. Open source tools in this category typically require investment in training, maintenance, or third-party services for production environments. Users should verify whether Ray’s open source license includes restrictions on commercial use or requires attribution, and consult the official website for details on enterprise offerings, compliance, or support options. The open source model emphasizes flexibility but demands rigorous evaluation of long-term operational costs and alignment with organizational needs.
Strengths & Trade-offs
Pros
- Universal distributed computing — works for any Python workload, not just ML;
@ray.remotemakes distribution trivial - Proven at massive scale — used by OpenAI for LLM training, processes 1+ exabyte/month across users
- Complete ML ecosystem — Train, Serve, Tune, and Data cover the full ML lifecycle
- 35K+ GitHub stars — one of the most active open-source AI projects with strong community
- Elastic scaling — add/remove workers during training without restarting jobs
- Framework-agnostic — works with PyTorch, TensorFlow, JAX, XGBoost, and any Python code
Cons
- Complexity for simple workloads — distributed overhead isn't worth it if your data fits on one machine
- Cluster management — self-managed Ray clusters require DevOps expertise; Anyscale solves this but adds cost
- Debugging distributed code — distributed failures are harder to diagnose than single-machine errors
- Memory management — Ray's object store can cause OOM issues with large objects if not managed carefully
- Learning curve — understanding Ray's task/actor model and resource management takes time
