AI-901 · Unit 05 · Quotas: TPM and RPM

Why max_output_tokens = 4,000 brings 429s

The rate limiter does not wait to see how long the answer is. When a request arrives it adds prompt + max_output_tokens to this minute's running count, and once that count reaches the deployment's TPM, every further request in the minute gets HTTP 429. Turn the knob and watch the column.

50K TPM
Chip column: what is left of this minute's 50,000 TPMDose: one request's reservation, prompt + max_output; the bright bottom part is what the answer really uses429 lamp: lit once the minute's count reaches the limitMonday 09:00 on a 50K-TPM deployment: 40 requests a minute, each with a 450-token prompt and a 150-token answer. The back tray holds the minute's requests still to arrive; each one ends in the passed tray or the 429 tray. The timer turns once per simulated minute (shown 2.5× faster); the count resets on each turn.

Try it

  1. Watch one minute at 4,000. Note when the lamp lights and how full the passed tray is by then.
  2. Before you turn the knob, predict how many of the 40 requests pass at 1,000. Then set it and check the readout at 0:59.
  3. Set 300. All 40 pass, yet the answers are exactly as long as before: only the reservation changed.

What the bench leaves out

The real estimate is approximate (it is based partly on character count) and also includes best_of. RPM is checked separately over short windows of 1 or 10 seconds; at 40 requests a minute it is not the limit here. Billing counts the tokens really processed, not the estimate.

Source: Microsoft Learn, Azure OpenAI quota management: Understanding rate limits.