A real RAG system also needs to decide which information should actually be sent to the LLM. This is where RAG becomes an engineering problem rather than simply an LLM problem.
From Retrieval to Ranking
A vector search may return several potentially relevant chunks. Not all of them are equally useful. So the system may perform an additional ranking step to determine which chunks are most relevant. The best results are selected and combined into a context. This context is then added to the prompt.
Building the Augmented Prompt
The final prompt may contain several parts, including instructions, retrieved information and original question. This is where the word augmented comes from. We are not changing the LLM itself. Instead, we are augmenting the
prompt with information retrieved from an external knowledge source. The LLM uses both the user's question and the retrieved context to generate its answer.
RAG Is More Than a Vector Database
When people first learn RAG, it can look as simple as "putting documents into a vector database and searching them." But a production RAG system involves many design decisions, such as "How should we clean the documents?", "How large should each chunk be?", "Should chunks overlap?", "Which embedding model should we use?" and so on.Each decision can affect the final answer. A powerful LLM cannot completely compensate for poor retrieval. If the system retrieves the wrong information, the model may receive the wrong context. In other words, better RAG is not only about using a better LLM. It is about building a better pipeline around the LLM.
The Bigger Picture
This is why RAG is fundamentally an engineering problem. The LLM is only one component of the system. The quality of the final answer also depends on document processing, chunking, embeddings, retrieval, ranking, context selection, and prompt design. Once we understand this pipeline, the next question is no longer simply which LLM I should use, but also how I should design the entire RAG system so the LLM receives the right information at the right time.
