GPU Autoscaling

GPU autoscaling dynamically adjusts the number of active GPU instances allocated to a workload based on incoming traffic, queue depth, or other signals. When demand increases, additional GPU instances are provisioned and added to the serving pool. When demand decreases, excess instances are drained and deallocated, reducing costs. The most aggressive form of autoscaling, scale-to-zero, removes all instances when there is no traffic at all.

Autoscaling for GPU workloads is significantly more challenging than autoscaling for CPU-based web applications. GPU provisioning takes longer due to hardware allocation, driver initialization, and model loading. Cold start latency means that reactive scaling alone may not be fast enough for traffic spikes. Effective GPU autoscaling requires a combination of reactive scaling based on current load and predictive scaling based on anticipated traffic patterns.

Key metrics used to drive GPU autoscaling decisions include request queue depth, GPU utilization percentage, inference latency percentiles, and requests per second. Each metric has tradeoffs. Queue-depth-based scaling responds quickly to demand changes but can oscillate if not dampened. Utilization-based scaling provides stable behavior but may not react fast enough to sudden traffic spikes. Production systems typically use a combination of metrics with configurable thresholds.

The scaling speed — how quickly new instances become ready to serve traffic — is a critical differentiator among serverless GPU platforms. A platform with 60-second cold starts needs to scale proactively with a longer lead time, often requiring persistent warm capacity. A platform like Cumulus with 12.5-second cold starts can scale reactively to traffic changes while maintaining acceptable latency, enabling more aggressive scale-down policies and greater cost savings.

Advanced autoscaling strategies include scheduled scaling for predictable traffic patterns, burst scaling with cloud spillover for on-premises clusters, and per-model scaling policies that account for different models having different resource requirements and latency sensitivities. The goal is to maintain the target latency SLO while minimizing the total GPU-seconds consumed.