The Evolution of Production Generative AI: RAG

While Large Language Models (LLMs) possess vast generalized linguistic knowledge, they suffer from two fundamental weaknesses in enterprise software: 1. Hallucinations: Inventing plausible-sounding but factually incorrect assertions when lacking exact data. 2. Knowledge Cutoffs: Lacking access to private corporate databases, student records, or recent documentation.

Retrieval-Augmented Generation (RAG) solves both problems by dynamically fetching verified text passages from a vector database and injecting them into the model's context window before generation.

Here is the step-by-step engineering architecture of a production RAG pipeline.


1. Document Ingestion & Chunking Strategies

Feeding raw 50-page PDF documents into an embedding model degrades semantic fidelity. You must segment text into coherent chunks:

  • •Chunk Size & Overlap: Typical chunk sizes range from 300 to 500 tokens with a 10-15% sliding overlap to preserve sentence context across boundaries.
  • •Recursive Splitting: Splitting on markdown headings (##), paragraphs (\n\n), and punctuation rather than arbitrary character offsets.
def chunk_markdown_document(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    """Splits text into sliding overlapping chunks preserving paragraph continuity."""
    words = text.split()
    chunks = []
    i = 0
    while i < len(words):
        chunk = " ".join(words[i:i + chunk_size])
        chunks.append(chunk)
        i += chunk_size - overlap
    return chunks

2. Generating High-Dimensional Vector Embeddings

Embedding models transform textual semantic meaning into floating-point vectors (e.g. 1536 dimensions):

import numpy as np

# Conceptual vector embedding representation
def get_mock_embedding(text: str) -> np.ndarray:
    # In production: invoke OpenAI text-embedding-3-small or Gemini text-embedding-004
    np.random.seed(abs(hash(text)) % (2**32))
    vec = np.random.randn(768)
    return vec / np.linalg.norm(vec)  # Normalize to unit length

3. Approximate Nearest Neighbor (ANN) Vector Search

When comparing a user query vector against millions of document vectors, calculating brute-force cosine similarity across every row is too slow. Vector indexes (such as HNSW — Hierarchical Navigable Small World) navigate vector graphs in logarithmic time:

def find_most_relevant_chunks(
    query_vector: np.ndarray,
    document_vectors: list[dict],
    top_k: int = 3
) -> list[dict]:
    scored_chunks = []
    for doc in document_vectors:
        # Dot product of normalized vectors equals cosine similarity
        similarity = float(np.dot(query_vector, doc["vector"]))
        scored_chunks.append({"chunk": doc["text"], "score": similarity})

    # Sort descending by semantic similarity
    scored_chunks.sort(key=lambda x: x["score"], reverse=True)
    return scored_chunks[:top_k]

4. Context Assembly & Grounded Prompt Injection

Once the most relevant text chunks are retrieved, assemble the augmented prompt:

def build_grounded_rag_prompt(user_query: str, retrieved_chunks: list[dict]) -> str:
    context_body = "

".join([f"--- Context Block {idx+1} ---
{c['chunk']}" for idx, c in enumerate(retrieved_chunks)])

    return f"""You are the PRAYXIS Academic Advisory Assistant.
Answer the student's question strictly and accurately based ONLY on the verified context provided below.
If the context does not contain sufficient details to answer, reply honestly that information is not available in the curriculum.

VERIFIED CURRICULUM CONTEXT:
{context_body}

STUDENT QUESTION:
{user_query}

ACCURATE ANSWER:"""

5. Evaluating RAG Quality (RAG Triad)

Before deploying a RAG assistant to production, evaluate performance against three metrics: 1. Context Relevance: Did the retriever find chunks that actually relate to the user's query? 2. Groundedness / Faithfulness: Is every claim in the generated answer supported by the retrieved context (zero hallucinations)? 3. Answer Relevance: Does the answer directly address the student's question?

Learn applied AI engineering and vector search systems in the PRAYXIS AI & Machine Learning Track.