How We Cut Our AI API Costs Substantially Without Changing Models

Our AI API bill grew steadily for months before anyone looked closely at what was actually driving it, and the answer wasn't "we need a cheaper model" — it was that a meaningful share of our spend was going toward requests that didn't need to happen at all, or that were carrying far more context than the task actually required. Here are the four changes that brought the bill down substantially, in the order we found and fixed them, with the actual before-and-after shape of each one.

1. Caching identical requests instead of re-asking every time

The biggest single source of waste: our support-ticket classification step (similar in shape to the triage system covered in building an AI support ticket triage workflow) was re-classifying the same auto-generated system messages every time they appeared, because dozens of near-identical "your invoice is ready" notifications were passing through the exact same classification prompt independently, every single time, with an identical answer coming back every single time.

function classifyTicket(string $ticketText, CacheInterface $cache, OpenAIClient $client): array
{
    $cacheKey = 'classify:' . md5($ticketText);
    $cached = $cache->get($cacheKey);

    if ($cached !== null) {
        return $cached; // identical input, skip the API call entirely
    }

    $result = $client->classify($ticketText);
    $cache->set($cacheKey, $result, ttlSeconds: 86400);

    return $result;
}

Hashing the exact input text as a cache key is a blunt instrument — it only catches byte-for-byte identical inputs, not semantically similar ones — but for auto-generated system messages that really are identical text every time, it eliminated a large share of classification calls outright. This is the same lazy-expiration caching pattern walked through in more depth in building a simple in-memory cache with expiry, applied here to API calls instead of a database lookup — the mechanism doesn't care what's expensive on the other end of the cache, only that repeating an identical request is wasted work.

2. Trimming context that wasn't actually being used

Our document-summarization prompts were including an entire document's metadata block — author, revision history, internal tags — on every single call, because the code that assembled the prompt just concatenated "everything about this document" without anyone auditing what the model actually needed to do its job. None of that metadata affected the summary. We measured it directly: running a sample document through a plain word counter before and after stripping the unused metadata block showed the prompt shrinking by roughly a third, for a task where none of the removed text ever influenced the output.

// Before: dumped the entire document object into the prompt
$prompt = "Summarize this document:\n\n" . json_encode($document);

// After: only the fields the summarization task actually uses
$prompt = "Summarize this document:\n\n" . $document->body;

This sounds almost too simple to be worth mentioning, and that's exactly why it's worth mentioning — nobody had deliberately decided to include the metadata, it accumulated because the "give the model everything, just in case" instinct is an easy default that nobody revisits until someone actually measures what's being sent.

3. Batching requests that were firing one at a time

A background job that tagged newly imported products ran one API call per product, sequentially, even on imports of several hundred items at once. Several providers' embedding and completion endpoints support submitting multiple inputs in a single request, which reduces per-request overhead and, depending on the provider's pricing structure, can reduce total cost as well as latency:

// Before: one request per product
foreach ($products as $product) {
    $tags[$product->id] = $client->generateTags($product->description);
}

// After: batched into groups the API endpoint actually supports
foreach (array_chunk($products, 20) as $batch) {
    $descriptions = array_map(fn ($p) => $p->description, $batch);
    $results = $client->generateTagsBatch($descriptions);

    foreach ($batch as $i => $product) {
        $tags[$product->id] = $results[$i];
    }
}

The overhead reduction here isn't about token count — it's the fixed per-request cost (and, for rate-limited endpoints, the time spent waiting on sequential round trips) that batching eliminates. For a job processing hundreds of items nightly, the cumulative effect over a month was one of the more noticeable line items in the before-and-after comparison, even though no single request got meaningfully cheaper. It also had a side benefit nobody had specifically asked for: the nightly job finished substantially faster once it stopped waiting on hundreds of sequential round trips, which shrank the window during which a mid-run failure could leave the import in a half-tagged state.

4. Moving simple classification off the expensive model tier

The most involved change: auditing which tasks actually needed a top-tier model's reasoning ability versus which ones were simple enough for a smaller, cheaper model in the same provider's lineup. Ticket urgency classification — pick one of four labels based on fairly explicit rules, the same task described in the triage workflow post — turned out to perform comparably on a smaller model once the prompt was tightened with clear examples, while genuinely nuanced tasks (drafting a customer-facing reply that needed to read as natural and appropriately toned) stayed on the larger model, where the quality difference was actually noticeable in the output.

• Good fit for a cheaper/smaller model: classification with a small fixed label set, structured data extraction, simple format conversion.
• Worth keeping on the larger model: anything requiring nuanced judgment, natural-sounding generated prose meant for a customer to read, or multi-step reasoning where a smaller model's answers were measurably worse on a side-by-side spot check.
• The only way to know which bucket a task falls into is testing both on the same real examples and comparing actual output quality — assuming a cheaper model is "good enough" without checking is how you trade cost for silently worse results.

Measuring which change actually mattered

Shipping all four at once would have told us costs went down without telling us why, which matters if a future change needs to trade one of them off against something else. We rolled them out roughly a week apart specifically so each one's effect was visible in isolation against the previous week's baseline, rather than as one large bundled drop that's hard to attribute afterward. Caching had the single largest individual effect, because our workload happened to have a lot of exact-duplicate requests — a different team with less repetitive traffic might find context trimming or tiered model selection matters more for their specific usage pattern, which is exactly why measuring your own actual request logs before assuming a general "cost optimization checklist" applies uniformly is worth the hour it takes.

A mistake worth naming: optimizing before measuring

The first instinct on seeing a rising bill was to assume the fix was switching to a cheaper model across the board, and we very nearly did that before actually looking at where the spend was concentrated. Pulling a breakdown by endpoint first showed that classification and the duplicate-heavy notification pipeline accounted for a disproportionate share of total calls, while the actually-valuable, harder-to-replace summarization and reply-drafting calls were a comparatively small fraction of total spend. Switching everything to a cheaper model first would have degraded the quality of the calls that mattered most while barely touching the actual source of the bill's growth. Looking at where the money was actually going, before deciding what to change, is the step that's easy to skip when a rising bill creates pressure to act quickly.

A trade-off that cost us briefly: caching that was too aggressive

The caching change wasn't free of its own bug. The classification cache's exact-text-match key initially had a 24-hour TTL applied uniformly to every cached result, including classifications for tickets containing time-sensitive language like "urgent — need this today." A near-duplicate ticket submitted the next day with only the date changed hit the cache and returned the previous day's classification unchanged, because the cache key was based on exact text and the two tickets' bodies were similar enough in structure that a templated notification system produced byte-identical text apart from one date field — which meant they hashed to different cache keys after all in this specific case, but it exposed how fragile an exact-match cache key is in general the moment inputs are almost-but-not-quite identical. We shortened the TTL specifically for the classification cache to a few hours rather than a full day, trading away some of the cost savings for less risk of a stale classification surviving into the next business day.

The broader lesson: every cost-saving change here also has a quality or freshness cost somewhere, and treating cost reduction as free money rather than a trade-off is how a caching change meant to save money quietly starts costing correctness instead. Reviewing each optimization for what it might be trading away, not just what it saves, is worth doing before it ships, not after a near-miss like this one prompts the review after the fact.

What we didn't do

We didn't fine-tune a custom model, and we didn't switch providers chasing a lower headline price — both were on the table early on and both would have added real engineering and evaluation overhead for a payoff that was genuinely uncertain compared to the four changes above, which were each verifiable in isolation before and after shipping. Caching, trimming, batching, and tiered model selection are also the kind of changes you can measure independently, one at a time, and roll back individually if one of them turns out to hurt quality — which matters more than it sounds like it should, because "we made four changes and costs went down" tells you far less than knowing which specific change actually mattered, or which one might need to be partially undone the next time a trade-off like the caching TTL issue above shows up somewhere else in the system. Cheap, reversible, individually measurable changes beat one large bet almost every time we've compared the two approaches directly.