In June 2026, some Internet users received a surprising piece of information from an artificial intelligence assistant:
Donald Trump had died from rabies after being infected by J.D. Vance.
The information was obviously false.
It came from an experiment conducted by members of the r/poisonAI
subreddit. Their goal was to publish enough absurd content around the
story to see whether it would eventually be picked up by artificial
intelligence systems.
The experiment worked.
DuckDuckGo Search Assist presented the story as a fact, relying on Reddit posts, a fake local news website and unrelated content.
Behind this deliberately ridiculous anecdote lies a much more serious question:
What happens when an attacker no longer tries only to deceive Internet users, but attempts to manipulate the systems they rely on to find and summarize information?
An AI does not verify truth
A large language model, or LLM, is neither a reliable knowledge database nor a judge of truth.
Its role is to generate a plausible answer based on the information available to it:
- training data
- user instructions
- conversation context
- documents provided by users
- web pages retrieved by search engines
- data obtained from external tools or services
The model does not naturally try to determine what is true.
It tries to generate a coherent response based on the information it has been given.
If several pages tell the same story, this repetition may be interpreted as a credibility signal.
However, ten pages repeating the same lie do not represent ten independent sources.
They may all originate from the same initial publication, copied, rewritten or automatically generated.
For a user, this looks like a coordinated campaign.
For a model, it may simply look like a recurring pattern.
Hallucination or poisoning?
It would be tempting to describe this incident as a simple hallucination.
However, the term is not completely accurate.
A hallucination generally refers to information invented by a model without sufficient support from available data or sources.
In this case, the system did not necessarily invent the story. It retrieved deliberately falsified content and summarized it without properly assessing its reliability.
This is not necessarily a permanent poisoning of the model itself either.
There is no evidence that the internal parameters of the model were modified or that it permanently learned the false claim during training.
Instead, the contamination happened in the information supply chain.
This can be described as:
- source poisoning
- search engine poisoning
- retrieval poisoning
- RAG poisoning
The model may be working correctly.
The information provided to it has been manipulated.
From SEO for humans to SEO for machines
Disinformation did not start with artificial intelligence.
For years, attackers have used:
- fake articles
- bot networks
- fake social media accounts
- coordinated campaigns
- websites pretending to be legitimate media
- search engine manipulation techniques
These operations mainly targeted humans.
AI assistants introduce a new target: machines that search, filter and synthesize information on our behalf.
It is now possible to create content specifically designed to be:
- published and indexed by search engines
- considered relevant by semantic search systems
- retrieved by AI assistants or RAG systems
- presented to users as a clear and synthetic answer
We already know SEO, which optimizes content to improve its ranking in search results.
We can now imagine offensive SEO designed for AI systems.
The goal is no longer only to attract readers.
It is to directly influence the answer produced by a machine.
The number of sources does not guarantee independence
AI systems often highlight multiple references to reinforce credibility.
But showing multiple links is not enough.
Three articles may cite the same Reddit post. Ten websites may copy the same press release. Multiple pages may have been automatically generated from a single source.
The model can then appear to have cross-checked multiple sources while actually following a chain of repetitions.
The distinction between:
- number of documents found
- number of different domains
- number of primary sources
- number of truly independent confirmations
is critical.
AI as an information laundering mechanism
A false claim posted on a forum usually looks like a false claim posted on a forum.
It contains warning signs: anonymous accounts, contradictions, poor writing or suspicious context.
An AI-generated answer has a very different appearance.
It is often:
- well written
- structured
- concise
- contextualized
- written in a neutral tone
- accompanied by references
AI can transform an unreliable rumor into a convincing explanation.
It does not only repeat the mistake.
It reformulates it, simplifies it and gives it an appearance of authority.
The assistant becomes a form of information laundering mechanism.
A doubtful claim enters through an unreliable source and comes out seconds later as a professional-looking and apparently objective answer.
A new attack surface
In cybersecurity, we have long considered external data potentially hostile.
We validate user input.
We control software dependencies.
We monitor exchanges with third-party services.
We verify the origin of files and executables.
With artificial intelligence, we must apply the same reasoning to information itself.
A web page, PDF document, knowledge base entry or search result should not automatically be considered trustworthy simply because it contains text.
Information itself becomes an attack surface.
The same principle applies to RAG and AI agents
Poisoning can happen at multiple levels:
- pretraining datasets
- fine-tuning datasets
- internal knowledge bases
- vector databases
- web search results
- external tools and agents
As systems gain permissions to send emails, modify documents, execute code or trigger workflows, poisoned information becomes much more dangerous.
The risk is no longer only that a model produces a wrong answer.
The risk is that we trust that answer enough to let it act.
Reducing the risk
There is no single solution.
Protection must cover the entire chain:
Prefer primary sources
Important claims should be verified through official and independent sources.
Preserve information provenance
RAG systems should track:
- origin
- author
- creation date
- import date
- confidence level
- modification history
Control document ingestion
Users should not be able to freely add documents that will later be treated as trusted knowledge.
Separate trust levels
Internal approved documentation, partner documents and public web pages should not receive the same level of confidence.
Detect coordinated campaigns
A sudden appearance of many similar documents should be considered a potential manipulation signal.
Test AI systems offensively
Applications using LLMs should be tested against:
- fake documents
- conflicting sources
- malicious content
- hidden instructions
- retrieval manipulation attempts
Keep humans involved
The more important the decision, the less it should rely exclusively on an AI-generated answer.
AI is an interface, not a source
The real danger is not discovering that AI systems can make mistakes.
We already know that.
The danger appears when we stop verifying their answers because they are faster, more accessible and better written than the original sources.
AI can be an excellent tool to:
- search
- summarize
- compare
- translate
- explain
- explore a topic
It should not become an oracle.
A fundamental cybersecurity rule remains valid:
Any external data should be considered potentially hostile.
With artificial intelligence, even knowledge becomes user input.
