Building Production-Ready LLM Applications with LangChain

A LangChain prototype takes an afternoon. A LangChain application that survives real users, real documents and a real invoice takes considerably longer, and the gap between the two is where most LLM projects stall.

This walks through what changes between the two, in the order the problems actually arrive.

Start by deciding whether you need the framework

LangChain is genuinely useful for what it is good at: swapping providers without rewriting, composing multi-step chains, and the enormous library of integrations that means you are not writing a PDF loader yourself.

It is less useful when your application is one prompt against one provider with one retrieval step. At that point the abstraction is costing you debuggability and adding a dependency that moves quickly, in return for saving perhaps forty lines of code.

A reasonable rule: use the framework where you benefit from the integrations and the composition, and drop to the provider SDK for the hot path once the design has settled. Mixing the two is normal and not a failure of architecture.

Retrieval is the whole game

In almost every production RAG system we have worked on, answer quality was limited by retrieval rather than by the model. The model was perfectly capable of answering correctly from the right passage; it was not given the right passage.

If the answer is not in the retrieved context, no amount of prompt engineering will produce it. It will produce something confident instead.

The default recursive splitter at 1000 characters is a starting point, not an answer. Two failure modes to design against:

• Chunks too small: the passage retrieved is missing the context that makes it meaningful. A clause that says "this does not apply to enterprise customers" is useless three chunks away from the rule it modifies.
• Chunks too large: the embedding averages several topics and matches nothing well. Precision collapses and the model receives a lot of near-relevant text.

What consistently helps more than tuning the size: chunk along the document's own structure. Split on headings for documentation, on articles or clauses for contracts, on question boundaries for FAQ material. Then prepend the section path to each chunk so the embedding carries where it came from.

# The chunk that gets embedded should carry its own context.
# Without the heading path, a chunk about 'the limit' matches nothing useful.

chunk_text = f"""{doc_title} > {section_path}

{raw_chunk}"""

# Store the clean text separately from the embedded text - the model should
# read the passage, not the breadcrumb you added to help retrieval.
records.append({
    "embed": chunk_text,
    "content": raw_chunk,
    "source": doc_url,
    "section": section_path,
})

Vector search finds semantic similarity and is reliably weak at exact terms - product codes, error numbers, proper nouns, version strings. A user searching for error E4021 wants the passage containing E4021, and an embedding model considers it broadly similar to E4022.

Combining BM25 keyword scoring with vector similarity, then fusing the two result sets, is the single largest quality improvement available to most RAG systems. It is also unglamorous, which is probably why it gets skipped.

Retrieve twenty candidates, rerank with a cross-encoder, pass the top four to the model. Rerankers read the query and the passage together rather than comparing two independently-computed vectors, so they are substantially more accurate at ordering.

The cost is one extra call in the tens-of-milliseconds range. Against the cost of passing twenty passages into a generation call, reranking usually saves money as well as improving answers.

This decision is made early, is easy to make casually, and is expensive to revisit - changing it means re-embedding the entire corpus. Three things actually matter:

• Domain fit. A general-purpose model handles general prose well and can be noticeably weaker on dense technical or legal material. Test on your own documents before committing; the public leaderboards are not measuring your corpus.
• Dimensions. Higher is not automatically better, and it is directly a memory cost. Several current models support truncating the vector at modest accuracy loss, which is worth measuring rather than assuming.
• Where it runs. A hosted embedding API is simplest and means every document you index leaves your network. If that is a problem for your data, decide it now rather than after the first ingestion run.

Record the model and its version alongside every stored vector. When you eventually migrate, you will need to know exactly what produced what, and reconstructing it later from deployment history is miserable.

Prompt structure that survives contact with users

Three things matter more than clever wording.

1. Make refusal explicit and acceptable. State that if the context does not contain the answer, the correct response is to say so. Without this the model will bridge the gap, plausibly and wrongly.
2. Require citations. Ask for the source identifier alongside each claim. This is the mechanism that makes the system auditable, and it visibly reduces invention because the model has to point at something.
3. Separate instructions from retrieved content structurally. Delimit the context clearly. Retrieved documents can contain text that reads like an instruction, and a system that cannot tell the difference is one hostile document away from a problem.

Version your prompts in git alongside the code, and treat a prompt change as a deployment. A prompt edited directly in a dashboard is an untracked production change, and it will be the thing you cannot explain when quality moves.

As soon as another system consumes the model's output, you need it in a fixed shape. Modern providers support constrained decoding against a schema, which is far more reliable than asking for JSON in the prompt and hoping.

Even so, validate on arrival. Schema-constrained output guarantees the shape, not the sense - a required field will be present and can still be an invented value. Parse into a typed model, reject what fails, and decide deliberately whether a failure retries or escalates. Silently accepting a malformed response is how bad data reaches your database with no error anywhere.

Evaluation, before you need it

The prototype-to-production transition fails most often here. Without evaluation, every change is a guess: someone tweaks a prompt, the three examples they check look better, and nobody knows what happened to the other four hundred cases.

Fifty to a hundred real questions with known-correct answers, drawn from actual user queries or from the people who currently answer them. This is manual work and there is no shortcut, and it is the highest-value day of work in the project.

Include the hard cases deliberately: questions with no answer in the corpus, questions with contradictory sources, questions that are ambiguous. A golden set of only answerable questions tests the easy half of the system.

This is the diagnostic that saves the most time. For each question, first ask whether the correct passage was retrieved at all. Then ask whether the answer was correct given what was retrieved.

Teams without this split spend weeks refining prompts against a retrieval problem. It is the most common wasted effort in RAG work.

Cost, latency and the things that fail

A generated answer takes seconds. A user watching a spinner for four seconds assumes the system is broken; a user watching text appear after four hundred milliseconds waits happily. Nothing else in the stack buys that much perceived performance for so little work.

• Embedding cache. Embeddings are deterministic per model - never compute one twice for the same text. This alone removes a large share of a document-heavy workload's cost.
• Retrieval cache, keyed on the normalised query. Real traffic is far more repetitive than teams expect.
• Response cache, only where answers are stable and the corpus is not changing hourly.

Model APIs rate-limit, time out and have incidents. In production that needs handling explicitly: retry with backoff on the transient classes only, a hard timeout well below your user-facing budget, and a fallback path - a second provider, a smaller model, or an honest message. Silently hanging is the worst of the available options.

Log the model version with every response. When answers change quality overnight and nothing in your code shipped, this is how you find out that the provider updated the model.

Security worth thinking about early

Two concerns specific to retrieval systems, both easier to design in than to retrofit.

The first is access control. If your corpus contains documents not everyone should see, permissions have to be applied at retrieval time - filtering the vector search by what this user may read - rather than by asking the model not to mention them. Post-hoc filtering of a generated answer is not a security control.

The second is indirect prompt injection. A document in your corpus can contain text crafted to redirect the model. Where the corpus includes anything user-submitted or externally sourced, treat retrieved content as untrusted input: delimit it clearly, never let it grant capability, and be conservative about what tools the model can reach in the same request.

The realistic order of work

1. Ingestion and chunking that respects document structure.
2. Hybrid retrieval, then a reranker.
3. A golden set and the retrieval-versus-generation split.
4. Prompt with citations and explicit refusal.
5. Streaming, caching and failure handling.
6. Logging of query, retrieved passages, model version and answer - the trace you will need for every quality complaint.
7. Access control at retrieval time, if the corpus needs it.

Steps one to three are where the quality is. Steps five and six are where the reliability is. Neither is glamorous, and skipping either is how a promising demo turns into a system nobody trusts.