Building a Simple RAG Pipeline Over Your Own Docs (No Vector Database Required at First)

Every RAG (retrieval-augmented generation) tutorial I read before building my own assumed a vector database from line one — Pinecone, Weaviate, pgvector — before explaining what the pipeline is actually doing underneath. For a first version over a few hundred internal docs, a vector database is more infrastructure than the problem needs. This is a walkthrough of the minimal version: a flat file of embeddings, a cosine similarity function you can read in ten seconds, and the specific chunking mistake that made the first real version give confident, plausible, wrong answers.

What RAG is actually doing, stripped down

Retrieval-augmented generation is two steps, not one: retrieve the most relevant pieces of your own documents for a given question, then hand those pieces to an LLM as context and ask it to answer using only that context. The "generation" part is just a normal prompt. The part worth understanding deeply is retrieval, because that's where the real pipeline lives and where most of the failures actually happen.

Step 1: chunking your documents

You can't embed and search an entire document as one unit — it's too large, and a good match on one paragraph gets diluted by irrelevant text in the rest of the file. The naive approach is to split every document into fixed-size chunks:

function chunkText(string $text, int $chunkSize = 500): array
{
    $words = explode(' ', $text);
    $chunks = [];

    for ($i = 0; $i < count($words); $i += $chunkSize) {
        $chunks[] = implode(' ', array_slice($words, $i, $chunkSize));
    }

    return $chunks;
}

This is exactly where the first real version of this pipeline went wrong, and it's worth walking through why before fixing it.

The mistake: chunking mid-thought, not mid-document

Splitting purely by word count with no regard for structure means a fixed 500-word boundary lands wherever it lands — frequently in the middle of a sentence, or worse, between a heading and the paragraph that explains it. On a real internal policy document, a chunk boundary landed exactly between the sentence "Refunds are approved automatically" and the next sentence, "except for orders placed more than 90 days ago" — which was in the next chunk. When a user asked whether a 120-day-old order qualified for an automatic refund, the retrieval step correctly found the first chunk (high similarity to "refund" and "automatically"), the LLM answered confidently using only that chunk's context, and the answer was wrong — not because the LLM hallucinated, but because the actual caveat had been chunked into a different piece of text that retrieval didn't happen to also pull in for that query.

This is the single most common RAG failure mode in practice, and it's not a model problem — the model answered correctly and confidently given the context it was handed. The context was the bug.

The fix: chunk by structure, not by fixed word count

function chunkByHeadings(string $markdown): array
{
    // Split on markdown headings, keeping each heading with its
    // own content instead of chunking blindly across sections
    $sections = preg_split('/\n(?=#{1,3}\s)/', $markdown);
    $chunks = [];

    foreach ($sections as $section) {
        $section = trim($section);

        if ($section === '') {
            continue;
        }

        // Still cap oversized sections, but split on paragraph
        // boundaries rather than mid-sentence
        if (str_word_count($section) > 600) {
            $chunks = array_merge($chunks, explode("\n\n", $section));
        } else {
            $chunks[] = $section;
        }
    }

    return array_values(array_filter($chunks));
}

Chunking along real structural boundaries — headings, then paragraphs as a fallback for oversized sections — keeps a caveat and the rule it modifies in the same chunk far more often than a blind fixed-word-count split does. It's not a perfect fix (a caveat can still legitimately live in the paragraph after the one that got retrieved), but it fixed the specific refund-policy failure and every similar case we found afterward in a spot-check of twenty real queries.

Step 2: embedding each chunk

function embedChunk(string $text, OpenAIClient $client): array
{
    $response = $client->embeddings()->create([
        'model' => 'text-embedding-3-small',
        'input' => $text,
    ]);

    return $response->embeddings[0]->embedding; // array of floats
}

$chunks = chunkByHeadings(file_get_contents('refund-policy.md'));
$index = [];

foreach ($chunks as $chunk) {
    $index[] = [
        'text' => $chunk,
        'embedding' => embedChunk($chunk, $client),
    ];
}

file_put_contents('doc-index.json', json_encode($index));

Each chunk becomes a vector of a few hundred to a few thousand floating-point numbers representing its meaning in a way that lets semantically similar text end up numerically close together. This whole index — text plus embedding, for every chunk across every document — is what gets saved to a flat JSON file. For a few hundred chunks, that file is a few megabytes, easily small enough to load entirely into memory on every request without needing a dedicated database at all.

Step 3: cosine similarity search, no database required

function cosineSimilarity(array $a, array $b): float
{
    $dotProduct = 0.0;
    $normA = 0.0;
    $normB = 0.0;

    foreach ($a as $i => $value) {
        $dotProduct += $value * $b[$i];
        $normA += $value ** 2;
        $normB += $b[$i] ** 2;
    }

    return $dotProduct / (sqrt($normA) * sqrt($normB));
}

function searchIndex(string $query, array $index, OpenAIClient $client, int $topK = 3): array
{
    $queryEmbedding = embedChunk($query, $client);

    $scored = array_map(function ($entry) use ($queryEmbedding) {
        return [
            'text' => $entry['text'],
            'score' => cosineSimilarity($queryEmbedding, $entry['embedding']),
        ];
    }, $index);

    usort($scored, fn ($a, $b) => $b['score'] <=> $a['score']);

    return array_slice($scored, 0, $topK);
}

Cosine similarity measures the angle between two vectors rather than their raw distance, which is what makes it a good fit for embeddings — it cares about direction (meaning) more than magnitude. Looping over every chunk in the index and scoring it against the query is O(n), which sounds naive compared to a dedicated vector database's indexed search, but for a few hundred to a few thousand chunks it runs in well under a second — genuinely fast enough that reaching for infrastructure to solve a performance problem you don't have yet is premature.

Step 4: assembling the final prompt

$topChunks = searchIndex('Can I get an automatic refund on a 120-day-old order?', $index, $client);

$context = implode("\n\n---\n\n", array_column($topChunks, 'text'));

$instructions = "Answer the question using ONLY the context below. If the context "
    . "doesn't contain enough information to answer confidently, say so "
    . "explicitly instead of guessing.";

$prompt = "{$instructions}\n\nContext:\n{$context}\n\n"
    . "Question: Can I get an automatic refund on a 120-day-old order?";

The explicit "say so instead of guessing" instruction matters more than it looks like it should — without it, a model handed incomplete context will often still produce a fluent, confident-sounding answer rather than admitting the retrieved chunks don't fully cover the question, which defeats a large part of the point of grounding the answer in real documents in the first place.

A second failure: retrieving the wrong chunk entirely

Fixing chunking solved the "caveat split across chunks" problem, but it exposed a second, different failure a couple of weeks later: a question about a specific pricing tier retrieved a chunk about a completely different, older pricing tier that happened to score highly on cosine similarity because it used a lot of the same vocabulary — "plan," "monthly," "included" — without actually being the relevant section. The embedding model was doing its job correctly; the two chunks genuinely were semantically close in vocabulary, even though only one of them was factually correct for the question being asked.

The fix here wasn't a code change so much as a content change: renaming section headings to be more specific ("Pricing (2026 tiers)" instead of just "Pricing") measurably improved retrieval precision, because the heading text itself becomes part of what gets embedded and searched. Retrieval quality depends as much on how your source documents are written as it does on the search algorithm — a genuinely underrated point that most RAG tutorials skip entirely because they're demoing on clean, well-structured sample data rather than the kind of inconsistently-titled internal docs a real team actually accumulates over time.

Re-ranking: a cheap second pass that improved precision further

Even with better chunking and clearer headings, the top-3 similarity results sometimes included a chunk that was topically related but not the best answer to the specific question asked. Adding a lightweight re-ranking step — asking the LLM itself to look at the top 5 retrieved chunks and pick which ones are actually relevant to the specific question, discarding the rest before generating the final answer — caught cases pure vector similarity missed:

function rerankChunks(string $query, array $topChunks, OpenAIClient $client): array
{
    $numbered = collect($topChunks)->map(
        fn ($c, $i) => ($i + 1) . ". " . $c['text']
    )->implode("\n\n");

    $prompt = "Question: {$query}\n\nChunks:\n{$numbered}\n\n"
        . "Which chunk numbers actually answer this question? "
        . "Reply with just the numbers, comma-separated, or 'none'.";

    $response = $client->complete($prompt);

    return array_filter($topChunks, fn ($_, $i) =>
        str_contains($response, (string) ($i + 1)), ARRAY_FILTER_USE_BOTH);
}

This adds one extra API call per query, which is a real cost trade-off worth being deliberate about — it's the kind of thing to skip on a low-stakes internal tool and worth adding on anything where a wrong answer has a real cost, the same cost-versus-accuracy judgment call covered in more general terms in cutting AI API costs without changing models.

When to actually add a vector database

• Document count in the low thousands or below: a flat file and linear scan is genuinely fine, and simpler to debug because you can just open the JSON file and read it.
• Growing past that, or needing sub-100ms search at scale: a dedicated vector database's indexed approximate-nearest-neighbor search stops being optional and starts being the only practical option.
• Needing metadata filtering alongside similarity (search only within a specific document category, for instance) is also where purpose-built vector stores start earning their complexity over a hand-rolled flat file.

Chunk size and quality matter more to answer accuracy than which vector store you use — a well-chunked flat file beats a poorly-chunked vector database every time, which is exactly why this walkthrough spent most of its time on chunking rather than on infrastructure. If you're applying the same "state assumptions explicitly, don't let the model guess silently" instinct from the prompt above to other parts of your workflow, it's the same core idea covered more generally in five prompt engineering patterns that actually improve output quality.

Before you settle on a chunk size, it's worth actually measuring what you're working with rather than guessing — running a sample document through a plain word counter to see its real length and paragraph rhythm takes thirty seconds and tells you far more about a sensible chunk boundary than picking a round number like 500 and hoping it fits your specific content.