# Databricks Boosts Cluster Creation Limits to Enhance Scalability and Reliability

> Discover how Databricks transformed its cluster management by implementing adaptive concurrency limits, achieving a 7.5× increase in request-rate limits and enhancing operational efficiency.

**Source**: databricks.com | **Published**: 2026-11-18 | **Type**: article

## Key Facts

- Databricks increased cluster-creation limits by 7.5×, enhancing customer scalability and satisfaction.
- Over 99.99% reduction in rate-limit errors indicates improved system reliability and customer experience.
- Removal of 100+ customer-specific overrides reveals streamlined operations and reduced manual intervention.
- Adaptive limits optimize resource use, showcasing a strategic shift towards dynamic capacity management.
- Fixed API limits still protect against traffic spikes, balancing customer needs with system stability.

## Summary

Databricks has significantly enhanced its operational efficiency by increasing its default cluster-creation request-rate limits by 7.5 times. This strategic move, driven by the implementation of adaptive concurrency limits, addresses the limitations of fixed request-rate limits in managing long-running operations. The adjustment not only improves customer experience by reducing wait times but also optimizes resource utilization across cloud platforms, including AWS, Azure, and Google Cloud.

Historically, Databricks relied on fixed request-rate limits to manage cluster creation workflows. However, these limits often resulted in underutilization of available capacity during peak times and excessive rejections during system slowdowns. The challenge lay in the unpredictable nature of cloud provisioning, where the duration of operations could vary significantly based on external factors, such as the health of cloud providers. This inconsistency made it difficult to maintain effective limits, leading to a cumbersome manual adjustment process during incidents.

The introduction of adaptive concurrency limits allows Databricks to dynamically adjust the number of active workflows based on real-time system health and workload demands. This approach enables the system to accept more requests when capacity is available while scaling back during periods of degradation. The result is a more resilient service that can handle bursts of demand without overwhelming the system. The fixed API request-rate limit remains as a safeguard against exceptionally high incoming traffic, ensuring a balanced approach to workload management.

The implications of this shift are profound for Databricks and its competitors. By removing over a hundred customer-specific rate-limit overrides, Databricks has streamlined its operations and reduced complexity, which can lead to cost savings and improved scalability. The decrease in cluster-creation rate-limit errors by more than 99.99% indicates a significant enhancement in service reliability. This improvement not only strengthens customer trust but also positions Databricks favorably against competitors who may still rely on more rigid rate-limiting strategies.

As the cloud computing landscape continues to evolve, the adoption of adaptive concurrency limits signals a broader trend towards more intelligent and responsive resource management. Competitors will need to consider similar strategies to remain competitive, particularly as customer expectations for performance and reliability increase. The ability to efficiently manage workloads in real-time will become a key differentiator in the market.

Looking ahead, Databricks is poised to explore further applications of adaptive concurrency limits across various long-running workflows. This could involve developing weighted slots for operations with differing resource costs or refining control mechanisms to prevent slow dependencies from stalling unrelated tasks. As cloud environments become increasingly complex, the ability to adaptively manage resources will be essential for maintaining high service levels and customer satisfaction. This strategic focus on flexibility and responsiveness will likely define the next phase of growth for Databricks and set new standards within the industry.

## Entities

- **Companies**: Databricks
- **Technologies**: AWS, Azure, Google Cloud
- **People**: Heqing Ya, Tan Zhang, Ankit Batra, Abhishek Rai

## Key Concepts

adaptive concurrency limits, fixed request-rate limits, cluster creation, cloud infrastructure, API request-rate limiter, workflows, system health, load management

## Definitions

- **Rate limiting**: A mechanism to cap the number of requests accepted per unit time to prevent abuse and manage resource usage.
- **Concurrency limit**: A cap on the number of operations that can be active at once, ensuring that resources are not overwhelmed.
- **Backlog**: Accepted work that is waiting to start, which has not yet acquired an active slot.
- **Adaptive control**: A dynamic adjustment of limits based on current system conditions to optimize resource usage.
- **Cluster creation**: The process of provisioning virtual machines and resources to create a computing cluster.

## Use Cases

- Improving cluster creation efficiency
- Managing API request rates
- Optimizing resource allocation in cloud environments
- Reducing customer-specific rate-limit overrides
- Enhancing system health monitoring
- Implementing fair scheduling for workloads

## Frequently Asked Questions

**What is the difference between request-rate and concurrency limits?**

A request-rate limit caps how many requests are accepted per unit time; a concurrency limit caps how many operations are active at once. Each active operation holds a slot until it finishes, including time spent waiting on dependencies.

**Why must a concurrency slot cover the whole cluster-creation workflow?**

A CreateCluster workflow holds its slot while provisioning VMs, including time spent waiting on dependencies after the API returns. When provisioning finishes, the slot is released even though the cluster may continue running customer workloads.

**How do adaptive concurrency limits respond to system degradation?**

When a cloud provider slows down, provisioning workflows hold their slots longer and accepted requests wait longer to launch. The controller also reduces the slot limit when congestion signals indicate pressure, cautiously allowing more concurrent work as conditions recover.

**Why does Databricks still keep a fixed API request-rate limit?**

The fixed API request-rate limit protects against exceptionally high incoming traffic and provides a request-rate boundary we can communicate to customers. Adaptive concurrency limits work inside the service to manage capacity pressure.

**What results did Databricks report after adopting adaptive concurrency limits?**

The initial rollout analysis reported 7.5× higher default cluster-creation request-rate limits, more than 99.99% fewer cluster-creation rate-limit errors, and the removal of over a hundred customer-specific rate-limit overrides.

## Links

- [Read on Welcome.AI](https://welcome.ai/content/databricks-boosts-cluster-creation-limits-to-enhance-scalability-and-reliability)
- [Original source](https://www.databricks.com/blog/how-we-raised-databricks-cluster-rate-limits-75)
- [Databricks](https://welcome.ai/company/databricks): Featured company

---

Source: Welcome.AI | https://welcome.ai/content/databricks-boosts-cluster-creation-limits-to-enhance-scalability-and-reliability