Stop Rate-Limiting Requests. Start Scheduling Tokens: Introducing DataRobot TokenGrid

Authors: Sudeeptha Jothiprakash, Venkat Bala, Tushar Pandey, Romi Datta

The real bottleneck in the modern AI stack

Enterprise IT has a strange problem: token spend and third-party model subscription costs keep climbing, while the GPU clusters running these workloads sit at just 20% utilization. That gap comes down to one thing: the tools managing access were never built to see the workload itself.

Traditional application orchestrators, Kubernetes frameworks, and API gateways like Tyk or Envoy manage static hardware: CPU, GPU count, memory, network traffic. They have no visibility into token counts, prompt length, or the KV cache pressure that actually drives LLM inference cost, and that blind spot hits both sides of the AI stack. It can’t optimize self-hosted GPU infrastructure, and it can’t govern quota, rate limits, or cost across third-party model APIs either.

To manage access today, platform teams lean on stateless proxies like LiteLLM: static configuration maps, hardcoded bandwidth rules, and count-based rate limits, say, a flat ceiling of 100 requests per minute. That works fine for simple traffic. It breaks down fast in multi-turn generative AI and agentic systems. A token-blind gateway sees two requests as identical if their metadata matches, even if one is a 10-token query and the other passes a 200K-token context file. The actual cost inside the cluster is orders of magnitude apart. Without a way to isolate these workloads, that 200K-token request locks up the model server’s KV cache, drives up P95 tail latency for every other tenant on that hardware, and can cascade into failures across the cluster, triggering a wave of 429 rejections across unrelated, mission-critical applications. The gateway admits it either way: a single rogue user or an unmanaged agent loop can trigger an uncontrolled surge that saturates model bandwidth for everyone else, because the gateway was never built to look for it.

Platform teams run into this friction in three places.

Model proliferation and silos. Token management is scattered across local infrastructure and third-party cloud APIs like AWS Bedrock, Azure OpenAI, and Anthropic. Business units spin up model endpoints independently or plug into third-party APIs on their own, so admins have no single place to track, audit, or enforce quota rules across the organization.

Bandwidth hijacking. Static quotas only check compliance at admission. A multi-turn prompt or agent request can look fine going in, then expand its context size and execution intensity well past that checkpoint mid-run. A single unmanaged agent loop can monopolize shared cluster bandwidth, lock up memory, starve neighboring applications, and blow through production SLAs, all without ever tripping the admission check that was supposed to catch it.

Runtime demand volatility. Enterprise AI architectures now mix third-party cloud APIs with self-hosted open-source models. Hosting those OSS models yourself means managing token-level resource surges directly, and static hardware management can’t keep up. Legacy infrastructure can’t align container scaling with actual request behavior, so platform teams end up choosing between slow response times during traffic spikes or expensive over-provisioning that leaves millions of dollars of enterprise compute sitting idle.

DataRobot TokenGrid works alongside your existing gateways and proxies rather than replacing them. It’s a stateful, token-aware capacity scheduler that sits directly in the admission path, bridging the gap between what the gateway lets in and what the underlying compute can actually handle.

DataRobot TokenGrid: scheduling by token physics, not hardware count

TokenGrid replaces static hardware partitions with a shared, work-conserving capacity pool. It’s a software scheduling layer that sits between your LLM gateways and wherever your models run: self-hosted clusters, hyperscaler-hosted models, or third-party models accessed directly from vendors like Anthropic and Google. Three capabilities make that possible.

1. Dynamic rate limiting

Instead of a fixed requests-per-minute ceiling, TokenGrid enforces unified, multi-dimensional quotas that map directly to physical cluster and API constraints. By tracking live metrics, tokens per second, monthly token budgets (TPM), and container-level memory, it turns raw telemetry into policy-based quotas that platform teams can apply consistently across business units, without piling heavy reasoning workloads onto already-saturated nodes.

2. The fair-sharing engine

TokenGrid runs every inbound request through a multi-stage fairness check that weighs workload shape, a tenant’s quota history, and current hardware constraints before a prompt ever reaches a model.

• Stage 1, workload classification. TokenGrid estimates prefill demand at the gateway layer, separating heavy context requests from lightweight queries and routing large contexts to dedicated or chunked prefill paths instead of letting them block the queue.
• Stage 2, quota enforcement. TokenGrid keeps a running ledger of token consumption across self-hosted clusters and third-party APIs. When a tenant exceeds its fair share, the scheduler lowers its priority. On self-hosted clusters, that means deprioritizing the request and pulling its prefix-cache affinity bonus, so compliant tenants keep access to fast KV cache nodes. On third-party APIs, it means applying backpressure and pacing so one high-volume tenant doesn’t exhaust the shared TPM/RPM ceiling or trigger provider rate limits (429s) for everyone else.
• Stage 3, memory-based backpressure. For self-hosted and managed deployments where container telemetry is visible, since third-party APIs don’t expose KV cache state, TokenGrid watches GPU memory and cache occupancy directly. When a model server pool crosses a configurable KV cache threshold, TokenGrid applies backpressure (HTTP 429) to low-priority background threads, shifts interactive streams to available nodes, and reorders the execution queue from first-come-first-served to shortest-job-first.

3. Automated capacity reconciliation

TokenGrid feeds container performance signals into a centralized Token Admin Service, then cross-references that data against real-time traffic patterns on a regular interval. That closes the loop between what’s happening at the infrastructure layer and what’s happening at the traffic layer, so the system can recommend and recalibrate execution slots, TPM thresholds, and RPM caps across your model fleet automatically. Platform teams no longer need to over-provision for spiky traffic. TokenGrid learns from real usage patterns and resizes allocations accordingly, freeing GPU capacity for other models or training workloads.

Built for on-prem and hybrid, built for what’s next

We hear this problem most from platform teams running predominantly on-premises. These organizations have invested millions of dollars in dedicated, bare-metal GPU clusters, where every percentage point of wasted capacity is a real cost. On-prem admins are the ones managing rigid compute boundaries with no native way to isolate a sudden developer burst from a core business application.

TokenGrid gives these environments an immediate answer to traffic surges that static proxy configs can’t handle. As a lightweight, container-aware scheduling layer, it lets platform teams reclaim GPU margin, enforce multi-dimensional quotas, and keep one noisy tenant from taking down the rest, directly in the admission path.

That’s the immediate value. The longer-term shift matters more: moving from fixed hardware placement to an ongoing, automated capacity-scheduling loop opens up new ways to think about enterprise inference orchestration altogether, not just how to contain a runaway agent loop today.

As enterprise AI moves from simple chat interfaces to multi-turn agentic systems, the teams that win won’t be the ones over-provisioning GPUs or bolting static rate limits onto their API proxies. They’ll be the ones who understand how tokens actually move through their infrastructure. TokenGrid turns unmanaged model consumption from a source of chaos into something predictable and efficient.

DataRobot TokenGrid is currently in Private Preview for select co-engineering and enterprise accounts. Reach out to our product team to explore the architecture and request early access.