Large language models are already very good at producing plausible academic prose. In research writing, that creates a slightly strange problem: the sentence can look finished before the reasoning is.
I ran into this while looking at AI continuations for a chemistry manuscript on metal-halide perovskites.
One paragraph was discussing Mn²⁺ incorporation and photoluminescence. The manuscript was gradually narrowing toward defect-related non-radiative recombination, especially halide-vacancy-related states and local structural changes.
The continuation went somewhere else. It brought in cavity engineering, general defect passivation, and compositional optimisation.
None of those topics was absurd. In another paragraph, they might have worked perfectly well. But this paragraph was trying to do something much narrower.
The model had picked up the subject. It had not picked up the reason this paragraph existed.
That failure interests me more than obvious hallucination, because it is easy to miss. If the continuation had used the wrong material or invented a mechanism, I would reject it immediately. Instead, it sounded like normal academic writing.
That is exactly why it was dangerous.
A relevant passage can still be the wrong passage
Suppose a system is trying to help with that same paragraph and retrieves three pieces of literature:
- a paper on cavity engineering in perovskite lasers;
- a paper on general defect-passivation strategies;
- a passage reporting Mn²⁺ incorporation together with reduced vacancy-related defect states.
All three are related to the topic. Only the third is really useful for the paragraph as it currently stands.
This is where I think a lot of AI writing systems still flatten the problem too much. They are very good at asking whether two pieces of text are about similar things.
A researcher is often asking something narrower:
Why am I talking about this here?
The difference becomes obvious inside a manuscript. A paragraph about Mn²⁺ could be explaining a mechanism, comparing previous studies, introducing a limitation, or preparing a transition into stability. Those are different jobs even if the vocabulary is almost identical.
A retrieval result can therefore be semantically close and still push the argument sideways.
I have started paying much more attention to this than to raw fluency.
The polished sentence creates its own bias
There is another problem once the model has already written something convincing.
Take a sentence like:
Mn²⁺ incorporation improves photoluminescence by suppressing defect-mediated non-radiative recombination.
It sounds normal. If I saw it in a Discussion section, nothing about the English would make me stop.
Then I look at the source.
Perhaps the paper shows reduced vacancy-related defect states after Mn²⁺ incorporation and also reports improved photoluminescence. The authors suggest that the two may be related, but the experiment does not establish the full causal chain implied by the sentence.
What I can safely write is closer to:
The improved photoluminescence may be associated with the reduction of vacancy-related defect states following Mn²⁺ incorporation.
The wording change is small. The scientific commitment is not.
What bothers me is how easy it is to reverse the reasoning once the polished sentence already exists. Instead of reading the evidence and asking what it supports, I am now asking whether I can defend wording the model has already handed me.
That is a subtle shift, but I think it matters.
A large library does not behave like working memory
This becomes more obvious when the researcher already has hundreds or thousands of papers.
The information may be somewhere in the library. That does not mean it is available in the moment.
I have seen this pattern often: someone remembers that a paper discussed a similar mechanism, or that there was a stability result under continuous excitation, but cannot remember the title, author, or exact terminology. They try a few searches and open the wrong papers.
Then one short passage appears and they recognise it immediately.
That is a different experience from ordinary search. The person did not reconstruct the paper from metadata. The passage acted as a memory cue.
This is also why I am not convinced that the answer is simply to put more material into a model’s context window. A thousand PDFs contain far more information than a paragraph needs. The useful question is which tiny part of that library deserves attention right now.
Sometimes that piece of evidence is not even the one the researcher thought they were looking for.
While writing about photoluminescence, for example, an old passage about stability under continuous excitation may reappear. Suddenly the paragraph starts to look different. Perhaps stability is the more interesting comparison. Perhaps the original explanation was too narrow.
The literature has not helped finish the sentence. It has interrupted it.
I think that is a useful outcome.
The manuscript already knows more than the search box does
Most literature workflows still begin after the writer decides to search.
You stop writing, invent a query, open a few papers, find the passage, and then return to the manuscript.
That is fine when you know what you need.
The awkward cases are the ones where you do not.
A paragraph already contains a surprising amount of information about the current task. It tells us what concept is active, what was just established, what kind of claim is being developed, and roughly what sort of evidence might matter next.
So I keep coming back to a different workflow:
manuscript context → a few relevant passages → source checking → judgment → writing
The important part, for me, is that generation comes later.
Not because generation is bad. It is extremely useful once the direction is clear.
But if the evidence arrives only after the model has produced a finished claim, the researcher is already reacting to the model’s framing.
I would rather see the source first in cases where the claim itself is still unsettled.
Why only a few passages?
There is an obvious temptation here to retrieve everything that might be relevant.
That gets annoying very quickly.
If every paragraph produces twenty papers, the system has just moved the search problem into the writing interface.
Two or three passages are often enough to be useful.
One may be irrelevant.
One may confirm what you already thought.
The third may make you stop.
That is the one I care about.
This is also where paper-level recommendations feel too coarse. If I vaguely remember a paper, showing me its title may not help much. Showing me the four lines around the result I once cared about can.
The source still needs to be easy to open. I would never want a retrieved snippet to become a substitute for reading the original context. But the snippet can tell me where to look and, just as importantly, why I might want to look there now.
“No continuation” should be a valid result
Most AI writing interfaces quietly assume that more text is useful.
Put the cursor at the end of a paragraph and the obvious action is to continue it.
That assumption is wrong surprisingly often in research writing.
Sometimes the evidence is weak. Sometimes two experiments are not comparable. Sometimes the source only supports a correlation. Sometimes the paragraph is already finished and adding another sentence makes it worse.
In those cases, the useful output is not a better continuation.
It is:
Don’t write this sentence.
I think builders of writing agents should take that seriously.
A system that always rewards itself for producing text will naturally become very good at producing text. That tells us little about whether it is helping the user reason.
The same problem appears in other agent workflows. An agent can take a perfectly reasonable next action and still be moving in the wrong direction because it misunderstood the current stage of the task.
Research writing simply makes the failure easier to see.
What I would look for in a better workflow
I would care less about how many pages the system can generate and more about a few simpler questions.
- Can it tell the difference between material that matches the topic and material that actually helps with the current paragraph?
- Can it bring back something useful from an old library before the researcher remembers enough to search for it?
- When it surfaces evidence, can the researcher get back to the exact source without detective work?
- And when the evidence weakens the current direction, can the system tolerate that instead of immediately producing another polished sentence?
These are not exotic capabilities. None of them requires the model to sound more intelligent.
They mostly require the workflow to give the model less freedom at the wrong moment.
That may matter more as generation itself becomes cheap.
I no longer find “the AI wrote a good paragraph” particularly impressive. Strong models can already do that.
What I want to know is what happened immediately before the paragraph appeared.
What did the system choose to bring into view? What did it leave out? Did the researcher see the evidence first? Was there a chance to reject the direction before fluent language made it feel inevitable?
If a system keeps helping me produce the next sentence when the evidence is telling me to stop, I’m not sure I’d call that assistance.