The Batches API: A Distinct Cost Lever With a Hard Latency Tradeoff

Model tier selection and extended-thinking discipline both reduce cost by changing *how much reasoning* a request pays for. The Batches API reduces cost through an entirely different mechanism: it changes *when and how* a request is processed, trading immediacy for a substantial price reduction on the same underlying work. An architect needs both levers in the toolkit, and needs to be precise about which problem each one actually solves, because they are not interchangeable and the exam (and production incidents) both punish conflating them.

What the Batches API Actually Is

The Batches API accepts a large collection of independent requests submitted together, processes them asynchronously (not necessarily immediately, and not necessarily in submission order), and returns results once the batch completes -- typically well after the submission call itself has returned. In exchange for accepting that asynchrony, batched requests are billed at roughly half the price of the equivalent standard API calls. This makes the Batches API the right tool specifically for large volumes of work where nobody is sitting and waiting for any single result: reprocessing a document archive overnight, re-scoring a historical dataset against an updated eval, generating summaries for ten thousand support tickets as a nightly job.

Batch Size: Order-of-Magnitude Guidance, Not a Hard Number to Memorize

A batch can hold a substantial number of requests -- illustratively, on the order of tens of thousands of requests per batch submission -- rather than being limited to a handful. The architect-relevant point is the order of magnitude (this is designed for genuinely bulk workloads, not for submitting three requests at a time to save a few cents), not a specific ceiling number, because exact platform limits are exactly the kind of detail that changes over time and should be confirmed against current API documentation before a design depends on it.

The Trap: Batch API Reduces Cost, NOT Per-Request Latency

This is the single most important distinction in this concept, and it is a distinction the exam leans on directly: the Batches API is a cost lever, not a latency lever. It does not make any individual request faster. It makes a large collection of requests cheaper to process in aggregate, at the cost of not knowing exactly when any single result will be ready. A request submitted through the Batches API does not stream back a response the way a synchronous call does -- there is no user on the other end of that call waiting in real time, and there structurally cannot be, because the whole mechanism depends on the caller being willing to submit now and collect results later.

This means a scenario describing a user waiting in real time for a response is *never* a Batches API scenario, no matter how much cheaper batching is per token, and no matter how tempting "but it saves 50%" sounds as an argument. Batching solves a throughput-and-cost problem for bulk, asynchronous work; it does not solve -- and actively cannot solve -- a latency problem for a single interactive request. Reaching for the Batches API to speed up a chat widget's response time is applying a throughput optimization to a latency problem, the exact mismatch pattern this course flags repeatedly across different levers (it's the same shape of mistake as reaching for the biggest model when the actual ask was cost control).

Batches as a Fifth, Distinct Lever

If you inventory the cost-optimization levers available to an architect, batching sits alongside -- but is mechanically distinct from -- prompt caching, model right-sizing, context trimming, and output-length capping. Each of those four levers reduces the *amount of work* a request does (fewer tokens reprocessed, a cheaper model doing the same job, less input volume, less output volume). Batching does not touch the amount of work at all -- the exact same request, with the exact same token count, costs roughly half as much purely because it was submitted asynchronously in bulk rather than synchronously one at a time. That's a genuinely different mechanism, which is exactly why it needs to be evaluated on its own terms rather than folded into "just another way to save tokens."

Worked Decision Rule