Message Batches API: Cost Discount Without Latency Improvement

What the Message Batches API Does

The **Message Batches API processes a large set of requests asynchronously, completing within a 24-hour window, in exchange for a roughly 50% per-token discount** versus standard synchronous pricing. It fits latency-tolerant, high-volume jobs -- bulk classification, large-scale content generation, offline evaluation runs -- where no individual request needs an immediate response.

The Exact Limits Per Batch Call

A single batch call accepts up to 100,000 requests OR 256 MB of total request size, whichever limit is hit first. A batch of 100,000 tiny requests can hit the count limit well before the size limit; a batch of a few thousand requests each carrying a large document can hit the 256 MB size limit long before 100,000 requests. Either ceiling caps the batch -- you submit multiple batch calls if your workload exceeds both.

The Mechanics: Submit, Poll, Retrieve

Submitting a batch returns a batch_id immediately -- the batch itself runs asynchronously in the background. Your code then polls for completion: repeatedly checking the batch's status on some interval until the API reports the batch has finished (or, at the latest, until the 24-hour window elapses). Once complete, you retrieve the results.

Results Return in Arbitrary Order -- Use `custom_id` to Match Them Back

Here's the detail that catches people in practice: the order results come back has nothing to do with the order the requests went in. Submit 500 requests and you can't just assume result #1 in the response maps to request #1 you sent -- position tells you nothing. What does work is setting a custom_id field on each request before submission; that same value comes back attached to its corresponding result, so matching is a lookup by custom_id, never by index.

Cost Lever, Not a Latency Lever

The defining exam point: batching trades speed for price. It is slower by design (results arrive asynchronously, potentially up to 24 hours later), so it reduces cost, not latency. This puts it in a different category from caching and right-sized model selection:

Batching and prompt caching are not mutually exclusive -- they compound. A scheduled, non-urgent job that reuses the same long system prompt across many requests inside one batch benefits from the batch discount on every request AND from caching's discount on the repeated prefix inside each one.

When to Reach for Batching

Batching is the right lever specifically when a workload is latency-tolerant -- the results don't need to come back synchronously -- and high enough volume that the ~50% discount meaningfully affects total spend. Looping over the synchronous API one request at a time is NOT batching, even if it processes a large list: it makes the same number of individual API calls, runs into the same rate limits, and gets none of the discount. The Message Batches API is a genuinely different submission model (one call, up to 100,000 requests/256 MB, a batch_id, async completion), not just a smaller version of calling the synchronous endpoint in a loop.

Common exam traps

• Assuming batching improves latency because it processes "in bulk" -- it is explicitly slower, trading time for a lower per-token price.
• Reaching for batching on a latency-sensitive workflow step, when caching or a smaller model would be the correct lever there instead.
• Assuming batch results preserve submission order -- they don't, and custom_id (set on the request, echoed back on the result) is the only dependable way to pair a result with its original input.
• Treating a for-loop over the synchronous API as "batching" -- it isn't; it hits the same per-request rate limits and pays full synchronous price, with none of the ~50% discount.