Load Management for AI Models - Managing OpenAI Rate Limits with Request Prioritization- Harjot Gill

### Overview In this presentation, Harjot Gill, co-founder and CEO of Flux Ninja, addresses the operational challenges of productizing generative AI applications. He contrasts traditional web APIs with large language model (LLM) APIs, categorizing them using a "mice vs. elephant" analogy. While traditional transactional endpoints respond in 100–500 milliseconds and OLAP queries take up to 1.5 seconds, LLMs such as GPT-4 can take several seconds to several minutes per call. Driven by compute constraints, GPU shortages, and high infrastructure costs, AI providers like OpenAI enforce strict rate limits based on requests per minute (RPM) and tokens per minute (TPM). Gill explains how consumers of these APIs can manage these constraints on the client side using Aperture, an open-source load management platform.

### Key Topics - **LLM Latency and Rate Limiting Constraints**: High computational requirements and fan-out architectures (such as mixture of experts and RAG pipelines) lead to aggressive provider rate limits. On GPT-4 Tier 2, for example, limits may restrict users to 5,000 RPM but only 40,000 TPM. - **Limitations of Ad-Hoc Solutions**: Common workarounds such as opening multiple accounts across cloud regions, setting container concurrency caps, or relying on HTTP `Retry-After` headers suffer from stale header metrics, lack of cross-cluster coordination, and an inability to prioritize important requests. - **Case Study: Code Rabbit**: An automated code review platform using GPT-3.5 Turbo and GPT-4 experienced severe peak-hour failures and serverless timeouts across different customer tiers (paid, trial, free/open-source) and workload types (interactive chat versus background code reviews). - **Client-Side Schedulers & Prioritization**: By tracking external provider limits on the client side, applications can smooth outbound request rates and schedule requests with fair queuing.

### Technical Details - **Aperture Architecture**: Designed for cloud-native environments, Aperture deploys an agent as a daemonset/sidecar on Kubernetes worker nodes, communicating with a control plane backed by etcd and Prometheus. Application services interface via Python or TypeScript SDKs, or via proxies/service meshes like Envoy and Istio. - **Distributed Token Bucket & Leaky Bucket**: Token buckets are sharded across agents in the cluster rather than using a centralized Redis instance, avoiding Redis write saturation. The system implements a leaky bucket algorithm to smooth out bursty traffic. - **Weighted Fair Queuing (WFQ)**: Requests are queued and scheduled based on request priority (e.g., product tier) and request cost (estimated prompt tokens). In Code Rabbit's implementation, interactive chat requests receive top priority with near-zero queue latency, paid users receive priority over trial users, and background open-source reviews are scheduled with lower priority during peak times. - **Integration Pattern**: Integrating Aperture requires three steps: defining business labels (such as hashed API key, model variant, product tier, and estimated tokens), wrapping outbound API calls in `start_flow` and `end_flow` boundaries, and configuring policies via YAML or a UI decoupled from the application code. - **Cost & Abuse Protection**: Client-side controls can also throttle inbound abuse (e.g., limiting open-source users to specific commit rates or files reviewed per hour) and leverage caching across incremental pull request commits to skip redundant reviews.

### Takeaways - Managing elephant-sized LLM workloads requires shifting load management thinking from simple reactive retries to proactive client-side scheduling. - Client-side token tracking allows systems to smooth outbound requests to match provider limits (e.g., maintaining an even outbound flow matching 40,000 TPM) without triggering provider rejections. - In-cluster distributed rate-limiting agents introduce minimal overhead (approximately 5 ms in-cluster, or 20 ms via cloud) while preventing serverless timeouts and reducing third-party API spend.