Databricks has increased the cluster rate limit by 7.5 times
The story so far
Databricks has increased the cluster rate limit by 7.5 times, adopting adaptive concurrency limits instead of fixed request rate limits to address the volatility brought by cloud resource dependency, allowing the system to accelerate processing when resources are available and slow down when services are degraded, reducing manual tuning. This move aims to address the shortcomings of fixed request rate limits when handling long-running operations. Databricks retains API request rate limits as a last line of defense against extremely high inflow traffic, while adaptive concurrency limits are managed internally by the service to determine the timing of accepting work. Databricks emphasizes that successful API responses do not guarantee immediate cluster startup or completion within a fixed time.
AI-written from the reports · Updated 19 hours ago
Coverage timeline
Follow the coverage to see every side of the story.
- Databricks BlogHow we raised Databricks cluster rate limits by 7.5×
Databricks has increased the cluster rate limit by 7.5 times, adopting adaptive concurrency limits instead of fixed request rate limits to address the volatility brought by cloud resource dependency, allowing the system to accelerate processing when resources are available and slow down when services are degraded, reducing manual tuning.
Heat over time
- Now
- 6
- Peak
- 1018 Nov, 11:00
- 24h change
- –
Hover or tap the chart to see each hour; use the arrow keys on a keyboard.