Latency in LLM Apps: The Silent Killer of User Experience
9/28/2025 •Engineering •7 min read
Latency Is a Product Problem
Slow models feel fine during prototyping.
But once real users show up, latency becomes the No.1 reason they abandon the flow.
Even a product with perfect accuracy feels broken if it feels slow.
Latency isn’t “an engineering detail.”
It’s a **user experience problem**.
---
Where Latency Actually Comes From
🔹 Model startup
Cold starts on GPU/CPU inference.
🔹 Token generation speed
Some models generate 4 tok/s.
Some generate 120 tok/s.
🔹 Retrieval
Slow vector DB queries or over-chunked documents.
🔹 Network distance
Running inference in a region far from the user.
🔹 Bad architecture
One giant monolithic call instead of streamed step-by-step responses.
---
Strategies That Actually Work
1. Stream Everything
Even if the backend is slow, the *perceived* speed improves massively.
2. Cache Both Directions
Cache:
• embedding → vector DB
• prompt → LLM output
• tool-call results
• RAG context
Caching sometimes gives you **10× faster responses** for free.
---
3. Use Small Models First
Pattern:
1. small model → quick answer
2. large model → refine only when needed
You halve costs and speed up UX.
---
4. Pre-warm Models
Schedule background job to pre-load models into memory every 5 minutes.
No more cold starts.
---
5. Async Retrieval
Parallel fetching:
• embeddings
• raw docs
• metadata
• filters
• agent state
Don’t chain what can be parallel.
---
Key Takeaway
The fastest apps aren’t fast because the model is fast.
They’re fast because the **architecture** is fast.
Latency is solved at the system level — not at the model level.
