Hey folks! Just wrapped up a pretty intense month of API usage at my SaaS and thought I’d share some key learnings that helped us optimize our LLM costs by 40%!

January spent of tokens:

https://preview.redd.it/lymlzhln8gpg1.png?width=2122&format=png&auto=webp&s=6cfae12f09de49ae1c814ae1fdd4d567bb3956b1

1. Choosing the right model is CRUCIAL. Choose the cheapest model, which does the job. There is a huge difference between the cost of the models (could be 20x the price). Choose wisely!

https://developers.openai.com/api/docs/pricing

2. Use prompt caching. This was a pleasant surprise - OpenAI automatically routes identical prompts to servers that recently processed them, making subsequent calls both cheaper and faster. We’re talking up to 80% lower latency and 50% cost reduction for long prompts. Just make sure that you put dynamic part of the prompt at the end of the prompt. No other configuration needed.

3. SET UP BILLING ALERTS! Seriously. We learned this the hard way when we hit our monthly budget in just 17 days.

4. Structure your prompts to minimize output tokens. Output tokens are 4x the price! Instead of having the model return full text responses, we switched to returning just position numbers and categories, then did the mapping in our code. This simple change cut our output tokens (and costs) by roughly 70% and reduced latency by a lot.

5. Consolidate your requests. We used to make separate API calls for each step in our pipeline. Now we batch related tasks into a single prompt. Instead of:

```

Request 1: “Analyze the sentiment”

Request 2: “Extract keywords”

Request 3: “Categorize”

```

We do:

```

Request 1:
“1. Analyze sentiment

1. Extract keywords

2. Categorize”

```

6. Finally, for non-urgent tasks, the Batch API is a godsend. We moved all our overnight processing to it and got 50% lower costs. They have 24-hour turnaround time but it is totally worth it for non-real-time stuff.

Hope this helps to at least someone! If I missed sth, let me know!

Cheers,

Tilen from blg