The math looks tempting. Everything A selection of what I have published over the last two years adds up to 127,068 tokens. That is twelve percent of the window. So I could simply dump all of it into the prompt instead of building chunks, computing embeddings and worrying about retrieval quality.

But “it fits” and “it works better” are two different things.

That is exactly what I wanted to measure in a small experiment. In this article I show you what happens when the same twelve questions are answered once through a RAG setup and once through the full corpus in the context window (I call it the long_context setup here). I also show you what went wrong along the way, which turned out to be almost more instructive than the experiment itself.

This is the setup I used:

  • The same prompts: Both paths get exactly the same system instruction. The only thing that differs is what comes before it. For the RAG setup that is five selected text passages, for the long_context setup it is all 32 articles.
  • The same model: Both paths run on Kimi K3 with the same generation settings. Temperature is the only setting I could not choose freely, so it is fixed at temperature = 1.
  • A blind evaluation at the end: I shuffled the two answers to each question and put them in front of me in an Excel file as “X” and “Y”, with no hint which path produced which. The key sits in a separate file that I only opened after I was done grading.

That way I could make sure the conditions were the same for RAG and for the full corpus (the long_context setup), and that the blind grading kept me from influencing myself.

→ 🤓 Find the full code in the GitHub Repo 🤓 ←

Table of Contents

1 – Why the question looks different now

2 – The setup: One corpus, two paths

3 – Twelve questions in three difficulty levels

4 – Evaluation: Why I graded blind

5 – Three things that went wrong

6 – Results

7 – When I would use which

Final Thoughts

Where to Continue Learning?

1 — Why the question looks different now

RAG (Retrieval Augmented Generation) came about as an answer to a technical limitation: a model could only see a few thousand tokens at a time. If you had more material than that, you had to pick out the relevant passages first and pass only those on. That works well, but it adds another component to our system that we have to maintain, tune and debug.

This is where Kimi K3 comes in. The model by Moonshot AI offers a window of one million tokens. There are also other models with similarly large context windows. This removes one of the main reasons why many people have used RAG. Other advantages still cited in favour of RAG include:

  • It’s cheaper.
  • It’s faster. Latency is lower.
  • It gives you traceable sources.

You can read more about this topic in the paper by Google DeepMind. Those three are what I wanted to test, on a corpus where I can judge every answer myself, because I wrote it.

Hint for Newbies: A context window is the amount of text a model can read at once in a single request. Anything that does not fit has to be selected beforehand. That selection is exactly what RAG does.

2 — The setup: One corpus, two paths

The corpus consists of 32 files holding 33 of my own articles from Medium and Towards Data Science. That makes a total of 127,068 tokens. I counted with tiktoken and cl100k_base, so not with Kimi’s own tokenizer. What Moonshot billed later was 127,346 input tokens per request, and that also includes the system instruction and the question. So the estimate was close enough to plan with.

Three topics appear twice, once as the Medium version and once as the TDS version. At the time, Towards Data Science was still a publication on Medium and not the separate platform it is today, which is how the same topic ended up in two versions. That is not an oversight but on purpose, since one of the questions targets exactly that duplication.

I did not notice that it was 33 articles and not 32 myself. It came up while I was grading the answers: two articles had ended up in the same file, and both models pointed it out independently.

```

One shared instruction for both paths. If the prompts differed, any quality

gap could come from the wording instead of from the retrieval strategy.

SYSTEM_PROMPT = (
"You answer questions about a collection of articles written by one author. "
"Use only the provided article text. If the text does not contain the answer, "
"say so plainly instead of guessing. When you state a fact, name the article "
"title it comes from."
)
`` The RAG path splits the articles into 788 chunks of 900 characters with 150 characters of overlap, embeds them withall-MiniLM-L6-v2` and sends the five most similar chunks to the model together with the question. That comes to roughly 1,200 tokens per request.

The long_context path sends all 32 articles along with every single question. That is 127,346 tokens per request, so about a hundred times as much.

One detail decides the cost: the corpus always comes before the question in the prompt. Prefix caching only works as long as the beginning of the message stays identical character for character. If the question came first, every single call would be a cache miss and the whole experiment several times more expensive.

  • Hint for Newbies: You can picture prefix caching like a colleague you hand the same thick folder to every time, with a different question each time. The first time he has to read all of it, after that he still knows what is in there. That is what the provider does in the background automatically. The important word is prefix: only the beginning is stored, and only as far as it stays the same character for character. If anything is changed at the beginning, everything that follows is worthless. That is why the corpus comes first and the question last.* In terms of cost, this results in a

difference between $0.30 and $3.00 per million input tokens.

```

The context always comes first and the question last. That order matters:

prefix caching only works while the beginning of the message stays identical

across calls. Putting the question first would make every call a cache miss.

messages = [
{"role": "system", "content": config.SYSTEM_PROMPT},
{"role": "user", "content": f"{context}\n\n---\n\nQuestion: {question}"},
]
```

3 — 12 questions in three difficulty levels

Before the first call went out, I defined twelve questions. They are split into three groups of four. Those groups are the actual point of the experiment:

Group A, single-fact:

The answer sits in exactly one place in exactly one article. For example: which embedding model I used in the chunk size experiment, and why. This is what retrieval is built for.

Group B, cross-article:

The answer needs two or three sources at the same time. For example: when I recommend RAG and when fine-tuning, including the places where I contradict myself.

Group C, corpus-wide:

The answer requires that the LLM has seen everything. One example is the question of how many of my articles link to a GitHub repo, and which ones.

For group C the outcome is fairly predictable: five chunks cannot possibly answer a question that covers all 32 articles. The group still belongs in there, because what is interesting is not whether RAG loses here, but how it loses. So, the question here is more: Does the model say honestly that the information is missing? Or does it guess and sound convincing while doing it?

4 — Evaluation: Why I graded blind

Instead of grading myself, I could have used a second model as a judge. I deliberately did not, because an LLM judge would only have added another source of error. My assumption is that with a corpus of my own texts I am the more accurate instance, because I know what is in there.

To keep myself from influencing the outcome, a small script (make_grading_sheet.py) turns the run into a grading sheet: the two answers to each question are shuffled and written out as A1-X and A1-Y, with no indication which path produced them. The key ends up in a separate file that stays closed until the end.

However, the assessment wasn’t entirely blind. Some of the responses gave themselves away with phrases such as ‘based on the text chunks I have’. That part in an answer can only come from a system that works with text snippets. So at those points, I knew I was looking at a RAG response. Next time, I would remove such self-revealing information before assessing the responses.

I graded along the following three criteria, with 0 to 2 points each:

The difference between the first and the third criterion is the most important one in the whole setup. While ‘correctness’ asks whether something is true, ‘grounded’ asks where it comes from. An answer can be factually correct and still come from the model’s world knowledge instead of from my articles. It looks good then, but it proves not what I wanted to test.

5 — Three things that went wrong

This part turned out to be more interesting than the results.

1) Half of my answers were empty, and I did not notice

The first complete run looked flawless in the terminal. Every line showed a plausible latency, plausible cost, a plausible cache rate. There was no error and no abort.

I only noticed when I opened the grading sheet in Excel and saw empty answer fields for several questions. In the end there were twelve out of 24 answers that were completely empty, plus three more that broke off mid-word.

The cause: K3 is a reasoning model, and the thinking tokens apparently count against the same limit as the visible answer. I did not find anything about this in the documentation at the time of writing, but I concluded it from the measurements: for every affected call the output tokens sat exactly on the configured limit, finish_reason reported length, and the content was empty. In the first run I had set max_completion_tokens to 800, which would have been generous for the answer length alone. On the harder questions the model used up all 800 tokens thinking, and nothing was left for the answer.

Fix: I solved it so that every affected call now prints << EMPTY OR CUT OFF in the terminal, and the end of the run lists which questions need to be repeated.

@property def truncated(self) -> bool: """The model ran out of completion budget before finishing its answer.""" return self.finish_reason == "length" or not self.answer.strip()
Lesson Learned: In a reasoning model, max_completion_tokens does not represent a limit on the length of the response, but rather a thinking budget. And a call that fails at this limit appears to be a success by every other metric. If I do not log finish_reason, I do not notice this limitation.

  • Hint for Newbies: What exactly counts as a reasoning model is not defined consistently. In this article I mean that the model produces a series of internal thinking steps before the actual answer. For a question like “What is the capital of Switzerland?” a model would not need any thinking steps. A reasoning model, on the other hand, thinks first: before the actual answer it generates a series of internal steps, sorts, discards, works things out. You normally do not get to see these steps, but you pay for them anyway, since they count as output tokens (*OpenAI-Documentation

;Kimi-Documentation

;Sebastian Raschka Blog

).

With Kimi K3 the thinking cannot be turned off, only turned down through the reasoning_effort parameter, with the values low, high and max. The default is max, and that is exactly what my experiment ran on.

2) The same call fails once and succeeds once

After raising the limit to 4,000 tokens almost everything went through. One question did not: C2, which asks about the tools mentioned most often across all articles. The model thought for 3,997 tokens and again delivered nothing.

I repeated exactly that one question, with identical context and identical wording. The second time the model thought for 884 tokens and gave a complete answer.

So the abort does not seem to have been caused by the question being too hard, it was probably pure chance. K3 only allows temperature=1, which makes deterministic runs impossible. For my experiment that means every single number in my results is a sample size of one.

The cost difference surprised me: the failed attempt cost $0.0635, the successful one $0.0187. That means, what we pay for here is the thinking. On the first attempt the model thought more than four times as long (3,997 tokens against 884). So the higher cost and the abort share the same cause: the model thinks until the budget is used up, and then there is nothing left for the answer. Why it took so much longer on the first attempt, I cannot say.

Lesson Learned: Since the model does not allow a fixed temperature, a single run is not a measurement, it is an observation. The numbers in this article have to be read and interpreted that way.

3) One complete pass does not fit into a single day

The long_context path needs 127,346 input tokens per question. For twelve questions that comes to 1,528,152 tokens spent on the input alone. The daily budget on Moonshot’s entry tier for Kimi K3 is 1,500,000.

So the long_context path alone breaks the daily quota on principle, regardless of money. On top of that, max_completion_tokens counts against the quota the moment the request is sent, not the tokens actually generated. A higher limit costs us quota even when it is never used up.

It also seems that the prefix cache, which is supposed to rescue the cost, does not reliably kick in:

  • When it does, Moonshot AI (the provider running the infrastructure that receives the requests) recognizes that the same corpus sits at the start of the prompt as in the last call, and does not compute it again.
  • When it does not, the entire corpus is processed again and charged at the full price.

In my run with the full corpus the cache kicked in on three out of eight calls. I could not see a pattern to it: the first two calls went at full price, the third came from the cache, then two more at full price, then two from the cache, then one full again. I changed nothing between the calls, the corpus the same.

That shows up clearly in the price. A call from the cache cost $0.0466, one without $0.3916. So roughly eight times as much.

It became even clearer when I added a single question five days later. The output reported cached 0/127,353, so zero out of 127,353 input tokens came from the cache. After a longer break it had expired completely, even though nothing about the corpus had changed.

Lesson Learned: With long_context the real hurdle (at least in the entry tier) is not the price per token, it is the daily quota and the question of whether the cache kicks in. Both of them are worth knowing about, and neither shows up in any pricing table.

How far off the planning can be shows up in the comparison with my own estimate. Before the run, a dry run calculates what the pass will cost, once assuming the cache always kicks in after the first call, and once without any cache at all.

The real value sits between the two estimates and much closer to the bad case. That fits the measured cache rate of 33 percent. So anyone planning with the optimistic number is off by about 30 percent.

6 – Results

After the grading I opened the key. The script report.py maps the scores onto the two paths and adds up what the run cost.

The first line answers two of the three arguments for RAG from chapter 1. Long_context cost sixteen times as much and needed about three times as long per question than RAG. Those are not narrow differences that would be worth a discussion.

The third argument, traceable sources, is not decided by cost and latency. It shows up in the grading, and that is where the picture gets less obvious:

Long_context answered all twelve questions completely

That was the result I had expected the least. Twelve out of twelve answers got the full score from me, across all three question groups and all three criteria. I have double-checked the result and it does indeed appear to be the case.

With 127,068 tokens the corpus is only twelve percent of the window, and at that size the model does not have to select anything and does not have to forget anything.

Whether the result would still look like this at 800,000 tokens, my experiment does not say. What I tested here is a realistic knowledge base, not an extreme case.

RAG loses on almost only one criterion

The breakdown per criterion is the most telling number in the whole experiment. On grounded, RAG is exactly level with long_context at 2.00. On correctness it sits just below, at 1.92. The entire gap comes from completeness, where RAG reaches 0.83 out of 2.

In plain words: in this experiment RAG made nothing up and said almost nothing wrong. But, the answers were often incomplete or not precise enough for what I would expect from a usable answer.

That is a different type of error than the one I had expected. My assumption before the run was that RAG would start guessing on the corpus-wide questions, because five chunks cannot carry a question about 32 articles. That is exactly what did not happen.

How RAG failed

For question C1 I wanted to know how many of my articles link to a GitHub repository, and which ones. I counted the reference by hand, it is 13.

RAG answered:

“Based on the provided article text, none of the excerpts contain a link to a GitHub repository. […] One caveat: these are retrieved chunks rather than complete articles, so it’s possible a GitHub link exists elsewhere in the full text of one of these articles.”

Long_context answered:

“13 articles link to a GitHub repository with the article’s code”

and delivered the list with one supporting quote per article.

It gets even clearer on question C2, about the tools mentioned most often:

“I can’t answer that definitively. I only have partial retrieved chunks from five articles (002, 010, 012, 016, 027), not the full text of your articles, and your collection clearly contains more than these five. Counting mentions across incomplete chunks of a subset wouldn’t tell you what’s most frequent across all your articles, so I won’t guess at a top five.”

That is not a good result, but it is at least a clean one. The model knows it cannot answer the question, and it says so. That is why grounded sits at 2.00 and completeness at 0. A system that had invented a plausible number at this point would have been more dangerous than one that delivers nothing.

That it turned out this way is not down to the retrieval, though, it comes from the system instruction in chapter 2: “If the text does not contain the answer, say so plainly instead of guessing.” Without that sentence the result would probably have looked different.

Which tells us that when we put RAG into production, this system instruction is something we really have to plan for.

The latency finding

You often hear the argument “RAG is faster”, and going by the results of this experiment it only seems to be half true:

On the single facts (A1-A4) both paths are close together. RAG is somewhat faster on A1 to A3, and on A4 long_context is even slightly faster.

On the cross-article questions (B1-B4) it tips clearly: there, long_context needs two to three times as long on most questions.

On the corpus-wide questions (C1-C4) the gap gets the largest. At 273.7 against 46.3 seconds, C3 is the slowest question for long_context in the whole experiment.

The reason sits in the last two columns. On these questions long_context thinks two to ten times as many tokens as RAG. So the time does not go into reading the context, it goes into thinking about it. C2 shows this most clearly: it is the only question where the prefix cache fully kicked in, 127,232 out of 127,342 input tokens came from the cache. It is still the second slowest at 208.3 seconds. The context was already processed, and the time went anyway.

C4 does not quite fit the picture: 106.9 seconds with only 2,229 thinking tokens, while C1 reaches 133.9 seconds with a similar number of thinking tokens. With twelve questions and temperature=1 I cannot read too much into that.

But the direction seems clear: more context creates more thinking, and with a reasoning model the thinking is the expensive part.

7 – When I would use which

For a corpus of this size, so a few dozen documents and well below a fifth of the context window, I would take long_context. The setup was a handful of lines of code, there was no chunk size to tune, no embedding model to pick, no vector index to maintain, and the answers were complete in all three question groups. The RAG path was the more laborious part of the project and it scored worse.

That only holds as long as the knowledge base is queried rarely, though. At twelve questions, a difference between $3.82 and $0.23 is a rounding difference.

If we scaled that up to twelve thousand questions, it would be roughly $3,800 against $230, which changes the discussion again. In that case a company should rather go for the RAG setup.

So the cost question does not hang on the price per token, it hangs on the number of queries per knowledge base. That is the number I want to know at the start of a project.

Three more points I take away from the experiment:

  • Response time: if answers have to come back in seconds, long_context is out. 111 seconds on average (almost 2 minutes) and more than 170 seconds (almost 3 minutes) on the hard questions are not usable for an interactive application.
  • Size of the knowledge base: when the knowledge base grows, the math changes. At ten times the size my corpus would no longer fit into a single prompt. And even now it fails on the daily budget, as we saw above.
  • System prompt: if RAG is used, the instruction to answer honestly when information is missing is not a detail, it is the reason the errors in this experiment stayed harmless.

On my Substack Data Science Espresso, I share practical guides and bite-sized updates from the world of Data Science, Python, AI, Machine Learning, and Tech — made for curious minds like yours.

Have a look and subscribe on Medium or on Substack if you want to stay in the loop.

Final Thoughts

One of the most valuable parts of this experiment, I find, is not the result but the mistakes I made along the way. The column with the thinking tokens only exists because twelve answers came back empty and I had to find out why. Without that detour I would never have seen the latency finding from chapter 6 and would still believe that long_context is barely slower.

For a next experiment I would take the following into account:

  • Log finish_reasonfrom the start instead of only after the first broken run. The empty answers looked like a model problem in the terminal, but they were a logging problem.
  • Ask every question several times instead of once, so that an observation turns into a mean.
  • Grow the corpus far enough that it really uses up the window. With 127,068 tokens I tested twelve percent, so the comfortable case. It gets interesting at sixty or eighty percent, where the model has to start selecting. That needs a higher tier, though, since even my corpus breaks the daily quota at twelve questions.
  • Run the whole thing with a model that allows temperature = 0. That Kimi K3 only permitstemperature = 1is the biggest limitation of this experiment, because no run can be repeated exactly. How much that weighs showed up in the failed C2 attempt: the same question with the same context thought 3,997 tokens once and delivered nothing, and on the second attempt 884 tokens and a complete answer. So what you read here are 24 observations, not a proof.

Even so, the difference between the RAG path and the long_context path was large enough to answer my starting question. For a corpus of this size that is not queried constantly, I can indeed skip RAG. What I cannot skip is a look at what comes back next to the answer: finish_reason, the number of thinking tokens and the share of the input that came from the cache. Without those three values I would have taken twelve empty answers for a model problem, would never have seen the latency finding, and would have planned with a cost estimate that was off by thirty percent.

Where To Continue Learning?

  • Kimi API Platform — Official documentation for Kimi K3
  • GitHub Repository — Full code for this experiment
  • TDS Article — Chunk Size as an Experimental Variable in RAG Systems
  • TDS Article — I Built the Same B2B Document Extractor Twice
  • OpenAI — Reasoning Best Practices
  • Sebastian Raschka — Understanding Reasoning LLMs
  • IBM Blog — What is a reasoning model?
  • DataCamp — Kimi K3 tutorial
  • Sentence-Transformers Hugging Face — all-MiniLM-L6-v2