The Evolution of Production Generative AI: RAG
While Large Language Models (LLMs) possess vast generalized linguistic knowledge, they suffer from two fundamental weaknesses in enterprise software: 1. Hallucinations: Inventing plausible-sounding but factually incorrect assertions when lacking exact data. 2. Knowledge Cutoffs: Lacking access to private corporate databases, student records, or recent documentation.
Retrieval-Augmented Generation (RAG) solves both problems by dynamically fetching verified text passages from a vector database and injecting them into the model's context window before generation.
Here is the step-by-step engineering architecture of a production RAG pipeline.
1. Document Ingestion & Chunking Strategies
Feeding raw 50-page PDF documents into an embedding model degrades semantic fidelity. You must segment text into coherent chunks:
- •Chunk Size & Overlap: Typical chunk sizes range from 300 to 500 tokens with a 10-15% sliding overlap to preserve sentence context across boundaries.
- •Recursive Splitting: Splitting on markdown headings (
##), paragraphs (\n\n), and punctuation rather than arbitrary character offsets.
def chunk_markdown_document(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
"""Splits text into sliding overlapping chunks preserving paragraph continuity."""
words = text.split()
chunks = []
i = 0
while i < len(words):
chunk = " ".join(words[i:i + chunk_size])
chunks.append(chunk)
i += chunk_size - overlap
return chunks2. Generating High-Dimensional Vector Embeddings
Embedding models transform textual semantic meaning into floating-point vectors (e.g. 1536 dimensions):
import numpy as np
# Conceptual vector embedding representation
def get_mock_embedding(text: str) -> np.ndarray:
# In production: invoke OpenAI text-embedding-3-small or Gemini text-embedding-004
np.random.seed(abs(hash(text)) % (2**32))
vec = np.random.randn(768)
return vec / np.linalg.norm(vec) # Normalize to unit length3. Approximate Nearest Neighbor (ANN) Vector Search
When comparing a user query vector against millions of document vectors, calculating brute-force cosine similarity across every row is too slow. Vector indexes (such as HNSW — Hierarchical Navigable Small World) navigate vector graphs in logarithmic time:
def find_most_relevant_chunks(
query_vector: np.ndarray,
document_vectors: list[dict],
top_k: int = 3
) -> list[dict]:
scored_chunks = []
for doc in document_vectors:
# Dot product of normalized vectors equals cosine similarity
similarity = float(np.dot(query_vector, doc["vector"]))
scored_chunks.append({"chunk": doc["text"], "score": similarity})
# Sort descending by semantic similarity
scored_chunks.sort(key=lambda x: x["score"], reverse=True)
return scored_chunks[:top_k]4. Context Assembly & Grounded Prompt Injection
Once the most relevant text chunks are retrieved, assemble the augmented prompt:
def build_grounded_rag_prompt(user_query: str, retrieved_chunks: list[dict]) -> str:
context_body = "
".join([f"--- Context Block {idx+1} ---
{c['chunk']}" for idx, c in enumerate(retrieved_chunks)])
return f"""You are the PRAYXIS Academic Advisory Assistant.
Answer the student's question strictly and accurately based ONLY on the verified context provided below.
If the context does not contain sufficient details to answer, reply honestly that information is not available in the curriculum.
VERIFIED CURRICULUM CONTEXT:
{context_body}
STUDENT QUESTION:
{user_query}
ACCURATE ANSWER:"""5. Evaluating RAG Quality (RAG Triad)
Before deploying a RAG assistant to production, evaluate performance against three metrics: 1. Context Relevance: Did the retriever find chunks that actually relate to the user's query? 2. Groundedness / Faithfulness: Is every claim in the generated answer supported by the retrieved context (zero hallucinations)? 3. Answer Relevance: Does the answer directly address the student's question?
Learn applied AI engineering and vector search systems in the PRAYXIS AI & Machine Learning Track.