The economics of AI · GKE Standard

When does self-hosting an LLM pay off?

Compare GKE fleet costs, API pricing, and capacity for your workload.

USD planning estimates · Source snapshot 2026-09-05 · No GPU benchmark is implied.

1 · Start with the work you need done

Presets change demand and keep your fleet settings.

Requests per active day

Include every model call in agent and tool loops.

Active days per month

1 for a one-off job; 22 for workdays; 30 for daily traffic.

Input tokens per request

Include instructions, retrieved context, history, and tool results.

Output tokens per request

Include billed reasoning tokens; use observed usage.

2 · Choose a model and precision

Open-weight model

Choose the smallest model that meets your quality bar.

Weight precision

Start with 8-bit; test 4-bit for memory savings or 16-bit as a precision reference. See the trade-offs for this model.

Runtime memory reserve (%)

Reserve above weight memory; long context and concurrency need extra KV cache.

3 · Price the capacity you keep

GPU node and billing plan

Whole-machine pricing. On-demand keeps a pilot flexible.

Serving replicas

Complete model copies; validate failover capacity separately.

Fleet running time (%)

100% = 730 h/month. Stopping on-demand nodes saves compute; commitments stay billed.

Additional hosting operations ($/month)

Hosting labor, software, and support beyond costs common to both options.

Capacity assumptions

100 tokens/s is illustrative. Re-measure after changing the model, GPU, precision, or workload.

Assumed output tokens/s per replica

Aggregate output across concurrent requests, measured end to end.

Usable share of measured throughput (%)

Planning margin for bursts and overhead; actual demand is separate.

API delivery and reuse API pricing mode

Batch: 50% off, with up to 24-hour completion. Caching: reuse an exact prefix. Discounts are not stacked.

Shared prefix tokens (cached mode)

Minimum: 4,096 for Gemini; 1,024 for Claude and Astra. Ineligible requests use standard rates.

Calls per cache group (cached mode)

Calls sharing an exact prefix within five minutes, including the first. Each group pays write or storage fees.

Assumes no misses within each group: Gemini five-minute storage, Claude five-minute writes, and Astra 30-minute retention.

Reset assumptions

Share the URL to save this scenario.

Your scenario · Small team · 5–10 people

What your demand actually costs

Change assumptions ↑

3,520 requests/month · 7.04M input tokens · 2.11M output tokens.

Self-hosting / month

$4,773.22

$3,700.22 compute + $1,000 operations + $73 GKE fee, including idle time.

Fleet footprint

1 node · 1 GPU

Qwen3.8 27B (xhigh) · 31.0 GB memory floor per replica · 1 node per replica.

Share of modeled capacity needed

1.2%

Demand / usable output capacity of 170.82M tokens/month. Over 100% means this fleet is too small.

Start with an API on cost grounds

Gemini 3.8 Flash is $13.20/month for this token mix, versus $4,773.22 for your fleet. Self-hosting needs enough value from control, residency, customization, or avoided quality failures to cover the difference. Compare task success before choosing.

A2 Ultra · on-demand: Best used for models that fit on one node; inter-node serving needs separate validation.

Three managed API baselines

Same request count and token mix, using your selected API mode. Compare quality separately.

Gemini 3.8 Flash

$13.20/ month

Per 1,000 requests: $3.75
Self-hosting break-even: 1,272,860 requests/month

Beyond this fleet's modeled capacity.

Claude Sonnet 5

$35.20/ month

Per 1,000 requests: $10.00
Self-hosting break-even: 477,323 requests/month

Beyond this fleet's modeled capacity.

GPT-6 Astra

$176.00/ month

Per 1,000 requests: $50.00
Self-hosting break-even: 95,465 requests/month

Within modeled capacity; quality still unproved.

Monthly costs and break-even at the selected input and output token mix
Option	Standard input / output USD per 1M tokens	Your monthly bill	Per 1,000 requests	Self-hosting break-even
Your GKE fleetA2 Ultra · on-demand	Fixed fleet cost	$4,773.22	$1,356.03	Reference fleet1.2% of usable capacity needed
Gemini 3.8 Flash	$0.75 / $3.75	$13.20	$3.75	1.27M requests/month Beyond this fleet's modeled capacity
Claude Sonnet 5	$2.00 / $10.00	$35.20	$10.00	477.3k requests/month Beyond this fleet's modeled capacity
GPT-6 Astra	$10.00 / $50.00	$176.00	$50.00	95.5k requests/month Within modeled capacity; quality still unproved

Price notes and comparison boundaries · 2026-09-05

Gemini 3.8 Flash: Introductory rates through Dec 31, 2026; input, output, cache reads, and storage double on Jan 1, 2027.
Standard uncached text rates. Immediate requests; no batch or cache discount assumed.
Verified 2026-09-05 · review by 2026-10-05
Claude Sonnet 5: $2 / $10 is standard pricing; the previously announced September increase was canceled. No announced end date.
Standard uncached text rates. Immediate requests; no batch or cache discount assumed.
Verified 2026-09-05 · review by 2026-10-05
GPT-6 Astra: OpenAI's published Astra name. Above 272,000 input tokens, full-request rates rise to $20 input / $75 output per million. No announced end date.
Standard uncached text rates. Immediate requests; no batch or cache discount assumed.
Verified 2026-09-05 · review by 2026-10-05
API estimates include input and output tokens. The selected mode includes either batch discounts or eligible cache reads, writes, and storage. Tools, media, taxes, regional premiums, and negotiated rates are excluded. API quotas and context limits still apply.
Include billed reasoning in output tokens and all repeated tool calls in demand. These inputs cannot predict each model's actual verbosity, retries, or task success.
For a one-off job, the API bill covers just that job. The fleet bill still covers your selected monthly running time. Reduce running time only if you can actually stop on-demand nodes, allowing for startup and processing.

Watch demand change the cost case

Explore the API cost curves against your fleet bill and capacity.

━━ GKE fleet
━ ━ Gemini 3.8 Flash
– – Claude Sonnet 5
··· GPT-6 Astra

Logarithmic axes (cost + $1). Circles: break-even. Dashed line: capacity of 284,700 requests/month. Shading exceeds fleet capacity.

3,520 requests/month · within modeled capacity. GKE fleet $4,773.22 · Gemini 3.8 Flash $13.20 · Claude Sonnet 5 $35.20 · GPT-6 Astra $176.00 per month. Apply this demand

What if demand or throughput is wrong?

Try ½× or 2× demand and throughput with the same fleet. Select a scenario to apply it.

Lower demand · ½×

1.8k requests/mo · 50.0 tokens/s

API costs less

1.2% of capacity needed

Lowest API: $6.60/mo

Try this scenario →

1.8k requests/mo · 100.0 tokens/s

API costs less

0.6% of capacity needed

Lowest API: $6.60/mo

Try this scenario →

1.8k requests/mo · 200.0 tokens/s

API costs less

0.3% of capacity needed

Lowest API: $6.60/mo

Try this scenario →

Your demand · 1×

3.5k requests/mo · 50.0 tokens/s

API costs less

2.5% of capacity needed

Lowest API: $13.20/mo

Try this scenario →

3.5k requests/mo · 100.0 tokens/s

API costs less

1.2% of capacity needed

Lowest API: $13.20/mo

Try this scenario →

3.5k requests/mo · 200.0 tokens/s

API costs less

0.6% of capacity needed

Lowest API: $13.20/mo

Try this scenario →

Higher demand · 2×

7.0k requests/mo · 50.0 tokens/s

API costs less

4.9% of capacity needed

Lowest API: $26.40/mo

Try this scenario →

7.0k requests/mo · 100.0 tokens/s

API costs less

2.5% of capacity needed

Lowest API: $26.40/mo

Try this scenario →

7.0k requests/mo · 200.0 tokens/s

API costs less

1.2% of capacity needed

Lowest API: $26.40/mo

Try this scenario →

Each column shows lower, current, then higher throughput. These are stress scenarios, not confidence intervals.

Enough capacity. Fast enough for people?

Monthly capacity

1.2% needed

A total-volume check. A month's spare capacity cannot absorb a burst that arrives in a few seconds.

Peak responsiveness

Responsiveness is still unproved

Record p95 first-token and completion times at your target concurrency using this configuration.

Enter latency targets and pilot measurements (optional)

Use p95 measurements from the same configuration at peak concurrency. Zero means unmeasured.

Target simultaneous requests

Peak in-flight requests, not total team members.

Target p95 first token (seconds)

An editable user-experience target, not a measured benchmark.

Target p95 completion (seconds)

Choose a target suitable for the task and answer length.

Pilot simultaneous requests

Record the concurrency actually sustained during the pilot.

Measured p95 first token (seconds)

Use the full workload, including prompt processing and network delay.

Measured p95 completion (seconds)

Measure the whole response; aggregate tokens/s cannot predict this.

4, 8, or 16-bit: what are you trading?

Memory and fleet cost for Qwen3.8 27B (xhigh), with the same fleet settings.

4-bit · prioritize memory savings

13.5 GBweights only, per replica

1 node · $4,773.22/month

¼ of 16-bit weight memory. Validate reasoning, tool use, and accuracy before accepting the quality trade-off.

8-bit · balanced starting experiment

27.0 GBweights only, per replica

1 node · $4,773.22/month

½ of 16-bit weight memory. A practical first compression test with a supported checkpoint and runtime.

16-bit · precision reference

54.0 GBweights only, per replica

1 node · $4,773.22/month

Full weight memory as a precision reference. Expanding quantized weights cannot recover lost quality.

Compression cuts the bill only when it reduces whole-node requirements or improves measured throughput. Check KV-cache memory and runtime and GPU format support.

Frontier comparison

Top 10 open-weight models

Artificial Analysis Intelligence Index v4.2, ranked by evaluated variant. Costs reuse your fleet settings; speed and quantized quality require measurement.

Top ten open-weight models and modeled GKE footprint and cost
Rank and model	AA index	License	Memory	Fleet	Monthly	Decision
#1 · Kimi K3 (max)Moonshot AI · 2800B parameters	50.2	Kimi K3 LicenseCommercial license	3220 GB	41 nodes / 41 GPUs	$152,782	Review commercial terms
#2 · GLM-5.3 (max)Z AI · 753B parameters	48.6	GLM-5.3 LicenseCommercial license	866 GB	11 nodes / 11 GPUs	$41,775	Review commercial terms
#3 · Qwen3.8 2.4T A95BAlibaba · 2400B parameters	46.7	Qwen3.8-Max LicenseCommercial license	2760 GB	35 nodes / 35 GPUs	$130,581	Review commercial terms
#4 · GLM-5.3-FlashZ AI · 320B parameters	46.2	MITPermissive	368 GB	5 nodes / 5 GPUs	$19,574	Validate multi-node serving
#5 · DeepSeek V4 Pro 0813 (max)DeepSeek · 1600B parameters	42.1	MITPermissive	1840 GB	23 nodes / 23 GPUs	$86,178	Validate multi-node serving
#6 · Qwen3.8 27B (xhigh)Alibaba · 27B parameters	41.6	Apache 2.0Permissive	31 GB	1 node / 1 GPU	$4,773	Pilot on this topology
#7 · K2 Horizon 375B A23BMBZUAI · 375B parameters	37.8	Apache 2.0Permissive	431 GB	6 nodes / 6 GPUs	$23,274	Validate multi-node serving
#8 · MiniMax-M3MiniMax · 428B parameters	35.7	MiniMax Community LicenseCommercial license	492 GB	7 nodes / 7 GPUs	$26,975	Review commercial terms
#9 · Inkling (xhigh)Thinking Machines · 975B parameters	32.2	Apache 2.0Permissive	1121 GB	15 nodes / 15 GPUs	$56,576	Validate multi-node serving
#10 · Muse Glimmer (high)Meta · 30B parameters	24.4	Apache 2.0Permissive	34 GB	1 node / 1 GPU	$4,773	Pilot on this topology

Compare cost per accepted task

Optional quality and human-review experiment

Compare the same tasks and rubric. Starting values are illustrative; replace them with measurements. Calls include retries, and review covers all attempted tasks.

Enable the accepted-task comparison

Attempted tasks per month

A separate task workload for this experiment. Required calls = tasks × each candidate's calls per task.

Human review cost ($/hour)

Use a loaded labor cost. Do not count hosting operations labor again here.

Your GKE fleet

Accepted tasks (%)

Final acceptance after all retries and review.

Average calls per task

Includes every retry and model call in a tool loop.

Review minutes per task

Average across all attempted tasks, including rejected ones.

Gemini 3.8 Flash

Accepted tasks (%)

Final acceptance after all retries and review.

Average calls per task

Includes every retry and model call in a tool loop.

Review minutes per task

Average across all attempted tasks, including rejected ones.

Claude Sonnet 5

Accepted tasks (%)

Final acceptance after all retries and review.

Average calls per task

Includes every retry and model call in a tool loop.

Review minutes per task

Average across all attempted tasks, including rejected ones.

GPT-6 Astra

Accepted tasks (%)

Final acceptance after all retries and review.

Average calls per task

Includes every retry and model call in a tool loop.

Review minutes per task

Average across all attempted tasks, including rejected ones.

Go deeper on the trade-offs

The Affordable AI Agents →

As AI agents transition from isolated developer experiments into production-scale workflows, engineering organizations are waking up to a…

CAG vs. RAG: Choosing the Right Strategy for Your AI Application →

CAG vs RAG for Generative AI: Compare latency, cost & complexity. Choose the best LLM context strategy for your app.

Method and limitations

Whole GPU nodes, total model weights, and an editable runtime-memory reserve.
Fleet cost includes compute, the GKE cluster fee, and your operations budget. Commitments remain billed while idle.
Capacity uses output throughput × replicas × usable share × running hours. Validate peak traffic separately.
API bills include input, output, and the selected batch or cache mode. Tools, media, taxes, regional premiums, and negotiated rates are excluded.
Validate quality, KV-cache memory, multi-node efficiency, regional availability, and model terms in a pilot.

Sources and freshness · 2026-09-05

Snapshot checked 2026-09-05. Prices and rankings move; follow the linked authority before committing spend.

Artificial Analysis model leaderboardIndependent Intelligence Index v4.2, open-weight status, model parameters, context, and license links.
Google Kubernetes Engine pricing$0.10 per cluster-hour management fee and GKE billing boundaries.
Google Cloud accelerator-optimized VM pricingA4, A3, and A2 machine composition and list-price options used here.
Google Cloud accelerator-optimized machine specificationsGPU count, GPU memory, host memory, networking, and provisioning constraints.
Google Cloud committed-use billingCommitted resources are billed throughout the term, including unused hours.
Gemini 3.8 Flash pricingIntroductory rates through Dec 31, 2026; input, output, cache reads, and storage double on Jan 1, 2027.Verified 2026-09-05 · review by 2026-10-05Cache fees and requirements
Claude Sonnet 5 pricing$2 / $10 is standard pricing; the previously announced September increase was canceled. No announced end date.Verified 2026-09-05 · review by 2026-10-05Cache fees and requirements
GPT-6 Astra pricingOpenAI's published Astra name. Above 272,000 input tokens, full-request rates rise to $20 input / $75 output per million. No announced end date.Verified 2026-09-05 · review by 2026-10-05Cache fees and requirements
vLLM quantization documentationMemory trade-offs and format support by GPU and runtime implementation.