Cloud GPU Compute: Cost Models and Elastic Architecture

Note: This is a review article (no first-hand experimental data), based on public pricing and common architecture practices.

Experiment Objective

Self-built clusters (chapters 1-14) are the "heavy-asset, slow-scaling" route; cloud GPUs are the "light-asset, fast-elasticity" route. The goal is not to argue which is better, but to answer when to use the cloud and how to use it cheapest: the cost models and time value of three billing modes (on-demand/spot/monthly), the elasticity differences between training and inference, and a "data on-prem, compute in cloud" sourcing architecture. After reading, you can answer for a concrete job: should this job go to the cloud? Which billing mode? How to set elasticity?

Code Walkthrough

Part 1: Cost accounting script

def gpu_cost_scenarios(hourly, spot_discount=0.7, monthly_ratio=0.55, util=0.75):
    """Three billing modes × utilization → effective per-card-hour cost"""
    per_spot  = hourly * (1 - spot_discount)     # spot ≈ 30% of list
    per_month = hourly * 24 * 30 * monthly_ratio / (24*30)  # monthly conversion
    def eff(h, u):  return h / u                  # spread over effective hours
    return {
        'on-demand-low-util(20%)': eff(hourly, 0.2),
        'on-demand-high-util(75%)': eff(hourly, util),
        'spot-elastic-jobs':       eff(per_spot, 0.9),
        'monthly-continuous':      eff(per_month, 0.9),
    }

# Example: a GPU at ~20 RMB/card-hour on-demand
for k, v in gpu_cost_scenarios(20).items():
    print(f"{k}: {v:.1f} RMB/effective card-hour")

Key points:

1. Count "effective card-hours," not list price: at low utilization, an on-demand task's effective cost is 3-5× the list price — the machine waits for you while the meter runs. Accounting must use list price ÷ utilization.
2. Spot suits interruptible jobs: spot instances run around 30% of list but can be reclaimed at any time. They fit frequent-checkpoint, restartable batch jobs; not long training (interruption loses progress).
3. Monthly suits continuously running inference/training: monthly discounts convert to ~55% of list per card-hour, but you also pay for unused time — below 50% utilization, monthly may cost more.

Part 2: Autoscaling config (K8s + Cluster Autoscaler example)

# cluster-autoscaler snippet: GPU node-pool elasticity
nodeGroups:
  - name: gpu-on-demand
    instanceType: gpu-24g
    minSize: 0
    maxSize: 32
    labels: { role: inference }
    taints: [{ key: gpu, value: "true", effect: NoSchedule }]
    scaleDownUnneededTime: 10m
  - name: gpu-spot
    instanceType: gpu-24g-spot
    minSize: 0
    maxSize: 64
    labels: { role: training }
    spotMaxPrice: 6.0

Key points:

• Separate on-demand and spot pools: inference (needs stability) on on-demand; training (interruptible) on spot; set elastic ceilings separately so a spot reclaim does not disturb inference.
• scaleDownUnneededTime: 10m: idle GPUs are reclaimed after 10 minutes — GPUs bill per minute; idle is a loss.
• spotMaxPrice is a cost guardrail: when spot prices spike past the psychological ceiling, stop adding machines; queuing is fine.

Principles

Cloud vs self-built decision framework

Dimension	Self-built (earlier chapters)	Cloud
Scaling speed	weeks-months (order/arrival/install)	minutes
Unit cost (high utilization)	low (amortized)	medium-high
Unit cost (low utilization)	high (idle waste)	medium (elastic)
Burst ability	procurement-cycle bound	within elastic ceiling
Data sovereignty	local	data egress considerations

Decision rule: stable high-utilization continuous training/inference → self-built or monthly; bursts, fluctuating load, short projects → cloud elasticity; mixing → "self-built baseline + cloud elastic overflow" is the most common production shape.

Training vs inference elasticity

• Training: long, checkpointable, controllable interruption cost → spot + frequent checkpoints (per chapter 04's design). Elasticity granularity is "job-level" — queue for compute.
• Inference: online, latency-sensitive, uninterruptible → on-demand primary + predictive scaling (ramp nodes ahead of request QPS; cold start takes 5-15 minutes, so you must be early). Elasticity granularity is "instance-level" — second-scale replica scaling.

Data sourcing architecture

When "data on-prem, compute in cloud," the worst move is copying the whole training dataset to the cloud. Standard practice:

On-prem storage ──(async sync of hot data)──> cloud object store (hot set/increments only)
Cloud training nodes ──> mount object store (read-only pull) / cache nodes for repeated reads

Key point: sync only the hot data the job needs, not a full mirror; training access to object storage must go through a cache layer (JuiceFS/cloud-disk cache), or GET latency and cost will drag training down.

Results

(Estimated; prices move with the market, shown only to demonstrate the method)

Scenario	Effective card-hour cost	Conclusion
On-demand 20% utilization	~100 RMB/card-hour	most expensive; emergencies only
On-demand 75% utilization	~27 RMB/card-hour	middle of the road
Spot (30%)	~7 RMB/card-hour	first choice for interruptible jobs
Monthly (55%, 90% utilized)	~12 RMB/card-hour	best for continuous training

Engineering Takeaways

1. Choose the mode by utilization first: at the same 20 RMB/card-hour list price, utilization decides whether the card is really "100 RMB" or "27 RMB" — account uniformly in effective card-hours.
2. Training in the cloud requires Checkpoint: without chapter 04's two-tier checkpoint, one spot reclaim pays back all the savings.
3. Inference elasticity needs lead time: GPU nodes cold-start in 5-15 minutes; scaling by current QPS alone always jitters. Use predictive scaling plus a buffer watermark.
4. Multi-cloud does not solve cost; it solves single-point dependency: cost optimization comes from matching utilization and billing mode; multi-cloud is insurance, not savings.

Next article preview: from public cloud back to the heterogeneous compute site — "PyTorch Adaptation and Migration for Domestic Accelerators: From Operator Differences to Performance Alignment."