# Serverless GPUs on Cloud Run: paying for inference only when you need it
Table of Contents
In June 2025 Google brought GPU support on Cloud Run, its “serverless” service, to general availability. In plain terms: you can serve an artificial-intelligence model on a graphics card that turns on only when a request arrives and shuts off when there’s nothing to do, paying only for the seconds actually used. It sounds like a technical detail, but it touches a cost that weighs heavily today. It’s worth explaining, without needless jargon, what changes and when it’s genuinely worth it.
A word on the terms: “GPU”, “inference”, “serverless”
A GPU is a graphics card: a kind of processor born for video games, able to do a great many calculations in parallel. That’s exactly why it became the standard tool for artificial intelligence, which is largely made of repeated calculations over large amounts of numbers.
Inference means using a model that has already been trained: you’re not teaching it, you’re just asking it something and waiting for the answer. When you write to an assistant and it replies, that’s inference at work. It needs a GPU to happen quickly.
Serverless is the rental model behind all this. It doesn’t mean “no servers”: the servers are there, but you don’t rent them for a fixed amount of time. You hand your program to the provider, which starts it when a request arrives and stops it when traffic drops, billing only the compute time consumed. Cloud Run is Google’s serverless service: until now it worked with ordinary processors, and now it can also use GPUs, an evolution not unlike the one that brought containers alongside Cloudflare’s Workers for heavier workloads.
Why it matters: the cost of an always-on GPU
The problem this announcement addresses is concrete, and it concerns anyone who wants to offer an AI-based feature: a chat, an image generator, a system that analyses text or photos, perhaps connected to external tools through standards like MCP.
The traditional way is to rent a machine with a GPU and keep it running. The catch is that you pay even while no request is coming in. A data-centre graphics card costs a fair bit per hour, and the traffic of almost any service isn’t constant: there are peaks and there are dead hours, at night, on holidays. Keeping the “engine” running around the clock to serve requests concentrated in a few hours means paying mostly for the time the GPU sits idle, waiting.
The alternative was to switch it off and on by hand, or to run your own system that adds and removes machines depending on load: doable, but it’s work to maintain. The idea of serverless with GPUs is to take all of that off your plate: if there are no requests, nothing is running and you pay nothing.
What Google announced
GPU support on Cloud Run has moved from preview to general availability, meaning it’s considered ready for production use and covered by the product’s service commitments. The main points:
- The GPU on offer is the NVIDIA L4, with 24 GB of memory dedicated to the card. It’s a model designed for inference, not for training the largest models.
- The serverless principle holds: scale-to-zero, so when no requests arrive the number of instances drops to zero and the GPU cost goes to zero too; and per-second billing, you pay only for the GPU actually consumed.
- You no longer need to file a quota request: the L4 GPU is available to every account, you just turn it on.
- Google states that an instance with the GPU and drivers already installed starts in about 5 seconds. That’s the time to make the card ready; on top of it you then add loading the model.
To give a sense of the total time, Google cites an example with a small language model in which the first response arrives in about 19 seconds starting from zero instances: within those seconds are the startup, loading the model and generating the reply. It’s a vendor example figure, tied to that specific model, not a guarantee that holds for every case.
How you use it, briefly
Anyone already familiar with Cloud Run finds the GPU as one more option. Turning it on via the console is a checkbox; from the command line it’s a flag. Here’s an example of deploying a model with the Ollama tool:
gcloud run deploy my-service \ --image ollama/ollama --port 11434 \ --gpu 1 --cpu 4 --memory 16GiFor the L4 GPU a minimum of 4 CPUs and 16 GiB of memory per instance is required. The service supports streaming the response over both HTTP and WebSocket: that means text can reach the user as it’s being generated, as in chats, instead of appearing all at once at the end.
Where it’s available and on what terms
At launch, GPUs on Cloud Run are active in five regions: Iowa (United States), Belgium and the Netherlands (Europe), Singapore and Mumbai (Asia). Coverage is therefore still partial, as is normal for a feature that has just reached general availability; Google says more regions are on the way.
By default, GPU instances are spread across several zones of the same region, to withstand the failure of a single zone. You can give up this redundancy to spend less, accepting a “best-effort” recovery - that is, not guaranteed - in the event of a zone problem.
Beyond services that answer requests in real time, Google is also extending GPUs to Cloud Run jobs, the batch tasks that start, do a piece of work and finish: fine-tuning a model on your own data, running inference over large amounts of images or text, transcoding video. At the time of the announcement this part is in private preview, so not yet open to everyone.
What to weigh before adopting it
Serverless with GPUs solves a specific problem, but it isn’t the answer to everything. A few points to keep in mind.
Cold starts have a cost in time. If the service has scaled to zero, the first request after a pause has to wait for the instance to power on and the model to load. For interactive use, a few seconds of initial wait can be annoying. Anyone who needs always-instant answers can keep one minimal instance always on, but then they partly give up the saving of scale-to-zero. It’s a trade-off to tune to your own traffic.
The L4 is an inference card. It has 24 GB of memory: fine for small and mid-sized models and for serving requests. It isn’t meant to train the largest models from scratch, which need different cards and far more memory.
Do the maths for your own case. Serverless pays off when traffic is uneven, with peaks and long pauses. If instead the GPU is used almost continuously, a machine rented for a fixed time may cost less. The right question isn’t “how much does a second of GPU cost”, but “how many real hours a day would I actually use it”.
Mind the tie to the provider. Packaging your service for Cloud Run is simple, but the way it scales, bills and is configured is specific to Google. Moving everything elsewhere later takes work. It’s not a flaw, it’s something to factor in, the same kind of trade-off worth weighing when you compare cloud providers in general.
In short, the underlying idea is reasonable and addresses a real cost: paying for a GPU only while it works, instead of keeping it running for nothing. For AI services with uneven traffic it can cut the cost noticeably and remove the chore of managing when machines turn on. For steady loads or very large models, traditional solutions remain a better fit. As always, the right choice is the one measured against your own case, not against the example figure in the press release.
