A chatbot that reads a company's internal PDFs and answers questions about them sounds harmless. You point it at the employee handbook, ask about the leave policy, and you get an answer. Simple. But since August 2026, using one of these in the EU without the right disclosures can break two laws at once. The tool itself works the same either way. What changes is who gets blamed when something goes wrong, and that comes down to one detail most vendors don't mention.
So what does "private" actually mean here?
Most "chat with your documents" tools use something called RAG, short for retrieval-augmented generation. Instead of training a model on a company's files, which takes a lot of time and money, RAG searches the documents the moment someone asks a question. It pulls out the useful parts and hands them to an AI model, which then writes an answer using that information. This way, the answer is based on real content instead of the model just guessing.
But the word "private" means two different things here, and vendors rarely say which one they mean. Sometimes it just means the chatbot only searches a company's own files, nothing from outside mixes in. Other times it means something stronger: nothing, not the files, not the questions, ever leaves the building at all. Only the second one needs the AI model to run on the company's own hardware. So a tool can honestly call itself "private" in the first sense while still sending every question you type to a cloud server somewhere.
Why the model you pick suddenly matters
A few open AI models released or updated this year have made the fully local option easier to actually use. Alibaba's Qwen3 has an Apache 2.0 licence, which means no restrictions on using it commercially, and its smaller versions (8B and 14B) run fine on normal consumer hardware. Meta's Llama 4 Scout is rated by several reviewers as the best all-round local model right now, though its licence only allows free commercial use for companies under 700 million monthly users, a detail that tends to get left out of the pitch.
It helps to be honest about what these rankings really are: opinions from independent reviewers, not hard facts. Results change from model to model, and a lot depends on what hardware and task each one was tested on.
Why regulators are suddenly interested in a document chatbot
The EU AI Act's transparency rules kicked in for chatbots in August 2026. Under Article 50, any AI system that talks to people directly has to make it clear it's a machine, not a human, right from the first message, unless that's already obvious. On top of that, GDPR says a privacy notice has to explain what personal data the chatbot collects, why, how long it keeps it, and whether that data is used to train the AI further.
Here's the part that catches people out: using a third-party AI vendor doesn't get a company off the hook. The company is still responsible for GDPR compliance, even if someone else built and hosts the chatbot. Compliance lawyers and regulators have said this again and again through 2026, as more businesses add AI tools to their products without bothering to update their privacy policies.
The actual trade-off, once you strip out the marketing
Running a model completely offline, say, Ollama paired with an open model and a local vector database like Chroma, avoids the vendor and data-transfer questions completely, because nothing ever leaves the device. But the moment a business sends document data through a hosted API instead, it picks up a whole set of legal and contractual obligations that a fully local tool never has. If you're comparing "private AI chatbot" options, this is the distinction that actually matters, local versus cloud-connected, far more than anything in the marketing copy.