Building a small RAG system: what you actually need

A RAG system sounds like heavy infrastructure, like server farms and data engineering teams. For a small, personal or internal company project, none of that is needed. At its core, four building blocks are enough: a handful of documents, a tool that translates text into meaning, a place to store that meaning, and a language model that formulates the final answer. This article walks step by step through what you actually need, without assuming any programming experience.

Building block 1: The documents the system draws on

The starting point is the question of what the system should actually answer about. That can be a folder of PDFs, a collection of Word files, exported notes, or plain text files. The only requirement: the content should be in a format that text can be extracted from — PDF, Word, Markdown, or plain text all work well.

A few dozen pages are already enough for a small system to meaningfully test whether the answers turn out accurate. You don't need to digitize an entire library just to get started.

Building block 2: Splitting documents into bite-sized pieces

A language model can't search through an entire 200-page manual at once. That's why every document is first broken down into smaller sections, usually a few hundred words each, with some overlap between sections so an important sentence doesn't get cut right at the seam.

This step is called chunking, and it determines more of the eventual answer quality than you might initially assume. Sections that are too small lose context; sections that are too large later dilute the search with too much irrelevant material. A simple rule of thumb works for getting started: sections about half a paragraph to a full paragraph in size, aligned with natural break points like headings or paragraph ends rather than a rigid character count.

Building block 3: An embedding model that translates text into numbers

For a computer to recognize "similar meaning," text first has to be converted into a series of numbers, called an embedding. Two passages that mean something similar in content, even with completely different words, end up close together in that numeric space.

Google offers a model for exactly this conversion through the Gemini Embedding API (currently gemini-embedding-001 for plain text, with a newer variant that can also process images and audio files). The entry point is deliberately kept simple: you sign up in Google AI Studio, get a free API key there, and can send text for conversion directly, without any server infrastructure of your own.

Every text section from building block 2 passes through this model once and gets its own numeric fingerprint.

Building block 4: A place to store and retrieve these numbers

The resulting number sequences need to be stored somewhere, in a way that lets you instantly find the most similar ones for a new question later. That's what vector databases are for.

For a small system, an elaborate hosted solution isn't necessary. A popular entry point is ChromaDB: a free, open-source vector database that runs without its own server and can be installed directly on your own machine. It stores the embeddings together with the corresponding original text and handles the similarity search in the background. That's entirely sufficient to start with; it's only worth switching to a larger, hosted vector database once a system grows to serve very many users or very large amounts of data.

Building block 5: The language model that answers at the end

The last building block is the actual language model that turns the question and the retrieved text sections into an understandable answer. A model from the same family is a natural fit here, for example Gemini via the Gemini API, since the embedding model and the language model then come from a single source and can be used with the same API key.

The process at query time is always the same: the user's question is likewise converted into an embedding, the vector database retrieves the most similar stored sections, and these sections are sent to the language model together with the original question, which then formulates the final answer, instructed to rely only on the supplied passages.

How the building blocks work together

In summary, a small RAG system needs five things, assembled once during setup and then run through again with every query:

1. Documents that serve as the knowledge source.
2. Chunking to break these documents into manageable sections.
3. An embedding model (for example Gemini Embedding) that turns each section into a number sequence.
4. A vector database (for example ChromaDB) that stores these number sequences and makes them searchable.
5. A language model (for example Gemini) that writes the answer from the question and the retrieved sections.

The setup itself happens once upfront: reading in documents, splitting them, converting them into embeddings, storing them in the database. Actual use then happens anew with every new question: embed the question, search for similar sections, have the answer generated.

What to realistically expect

A small RAG system of this kind can be built with manageable effort, but it doesn't replace careful upkeep. If the underlying documents aren't updated, the system stays on an outdated footing too. If the chunking is poorly chosen, the search later finds only incomplete information, and the language model sometimes fills the gaps with its own guesses. Anyone who starts with a few documents, spot-checks the answers, and adjusts chunk size as needed will still end up with a working, personal reference system for manageable effort.