Your Company's Internal AI Assistant Is a Treasure Chest for Hackers

Here's a familiar story: employees ask for an AI tool to help with their work. Some are already pasting company data into public chatbots without asking. Security and legal raise concerns, so management agrees to build an in-house AI assistant.

It's an internal chatbot connected either to a privately hosted model or to a provider's model through an enterprise API. The contract says company data won't be used for training.

Staff start using it, and the use of public chatbots drops.

Within a few weeks, every department is using it. HR drafts performance reviews, finance summarises board papers, sales uploads pipeline reports, legal reviews acquisition documents, and engineers paste in error logs that still contain passwords.

Each of these uses makes sense on its own. But put them together and the company has built something it never meant to build: one place that holds all of its sensitive data. For a hacker, that's not a productivity tool. It's a treasure chest.

What's inside an internal AI assistant

To the user, the assistant is a text box. Behind it, there are several parts, and each one handles your data in some way.

The interface: the web page or app staff type into. It shows chat history and accepts file uploads.
The backend: the part that receives each message, applies any rules, decides what to send to the model and returns the answer. It's often called a gateway or orchestration layer.
The model: either hosted on the company's own servers or reached through a provider's API.
Document search: when staff upload files, or when the assistant is connected to SharePoint or Confluence, the text is broken into pieces and stored in a search index, often a vector database, so the assistant can find and quote it later.
Storage and logs: databases for conversation history, storage for uploaded files, and logs for troubleshooting and monitoring.

Zero data retention isn't the full picture

Many companies ask their model provider for zero data retention. It's an important control. It means the provider doesn't keep your prompts and responses after processing them.

But it only covers the provider's side. It says nothing about what your own assistant keeps. On the company's side, data can sit in:

conversation history, often kept indefinitely so users can scroll back
uploaded files, plus the text extracted from them
the document search index, which holds pieces of every file it has processed
application and gateway logs, which may record full prompts and responses for debugging
monitoring and error-tracking tools, which are sometimes run by yet another vendor
caches, and backups of all of the above

So a company can have zero data retention with its provider and still keep every prompt ever written, in several places, for years.

Simple on the surface, complicated underneath

This is why the assistant is harder to secure than it looks. It isn't one system. It's several systems joined together, often built by different teams and sometimes by different vendors.

Each part has its own storage, its own access rules and its own retention settings. The interface team decides how long chat history stays. The platform team sets up the logging. Someone else connects the document search to SharePoint. Backups follow whatever the infrastructure default is.

Ask a simple question, like "when an employee pastes customer details into the assistant, how many places does that data get copied to?", and you often won't get a clear answer. Usually no single person knows, because no single person built the whole thing.

What concentration changes

The scale of this risk is easy to underestimate. Recent industry research found that 26.4% of all files uploaded to GenAI tools contained sensitive data, up from 22% the quarter before. The sensitive material included source code, access credentials, proprietary algorithms, M&A documents, customer and employee records, and internal financial data. Those figures cover AI tools in general, but there is no reason people behave differently in a sanctioned one. If anything, they share more, because they've been told it's safe.

Hackers have always had to work for this kind of access: break into one system, find where the valuable data lives, move to the next, and collect it piece by piece. An internal AI assistant does that collecting for them.

So where would a hacker start? Look at it layer by layer.

The infrastructure layer: The assistant is one platform serving every department, so it usually runs on a handful of service accounts with broad access to storage, logs and model endpoints. For a hacker, that's the shortcut. Compromise one of those accounts, or one over-privileged admin, and you don't get one team's data. You get everyone's. A single misconfigured storage bucket or exposed admin panel now carries the impact of a company-wide breach.

The data layer: This is where the treasure actually sits. Every prompt, every uploaded file and every response gets stored somewhere: chat history, application logs, caches, debugging traces and the search index the assistant uses to "remember" documents. Retention is often set to forever by default, because nobody decided otherwise. Classification usually doesn't exist. The board pack and the lunch menu sit side by side, equally unlabelled and equally exposed.

The access layer: In the source systems, a junior analyst can't open the acquisition folder. But once those documents are indexed so the assistant can answer questions about them, those permissions often disappear. Ask the right question and the assistant will helpfully summarise a document you were never allowed to open. A hacker who phishes that junior analyst's login can do the same. The data didn't leak out of the building. It leaked sideways, to the wrong person inside it.

The model and provider layer: If the assistant calls a third-party model, the provider's own logging, abuse monitoring and subprocessors become part of your risk. If it's privately hosted, the model server, its fine-tuning data and its operators are. Either way, there's now one more party, or one more system, that sees everything. And because the assistant reads whatever it's given, a hacker doesn't even need a login: a hidden instruction in an uploaded file or a shared document can tell it to pull data from elsewhere and hand it over.

No single one of these is exotic. What makes them dangerous is that they all point at the same target.

What good looks like

None of this means you should switch the assistant off. Pushing people back to public tools would only scatter the risk somewhere you can't see it. The goal is to treat the assistant as what it has become: a crown-jewel system. Here are six things to do, and a quick test for each that you can run this week.

1. Put an expiry date on every store

Do: List every place the assistant keeps data: chat history, uploads, the search index, logs and backups. Set a retention limit for each one on purpose, for example 30 days for chat history and no raw prompts in debug logs at all. Data you don't store can't be stolen.
Test: Ask the team that runs it: "Where is one prompt stored, and when is it deleted?" If nobody can answer, that's your first finding.

2. Catch secrets before they're saved

Do: Add a redaction step in the gateway, between the user and the model, that catches passwords, API keys, card numbers and personal identifiers before they're stored or sent anywhere. Think of it as the security scanner at the airport, not a sign asking passengers not to bring anything dangerous.
Test: Paste a fake API key into the assistant, then search the logs and chat history for it.

3. Search as the user, not as a service account

Do: Use on-behalf-of authentication (OAuth token exchange), so the assistant reads SharePoint and Confluence with each user's own token, not one shared account that can see everything. Then the assistant can only find what that person could already open.
Test: Sign in as a junior user and ask about a document only executives can open. The assistant should find nothing.

4. Split it by sensitivity

Do: Not every department needs the same instance. Give legal deal work, HR casework and general productivity separate workspaces, each with its own storage, encryption keys and admins, so one breach doesn't open everything.
Test: Ask: "If the general workspace is breached, can the attacker reach legal's files?"

5. Lock down who can read everything

Do: List every admin and service account with access to the assistant's chats, stores and logs. Remove standing access, require MFA, and send access logs to a system the admins themselves can't edit. Review these privileges as you would for your most sensitive database.
Test: Ask: "Who can read every employee's chat history today?" Count the names.

6. Get the provider's answers in writing

Do: Read the provider terms like a risk owner, not a buyer. Confirm in the contract what the provider logs, how long flagged prompts are kept, who reviews them and which subprocessors see your data.
Test: Ask the provider for their current subprocessor list. If it takes a week to arrive, note it.

The cost of getting this wrong

When data lived in separate systems, a breach was usually one bad day for one team. A breach of a concentrated AI platform is a bad day for the whole company, because it exposes strategy, staff, customers and code all at once. For a hacker, it's one break-in instead of fifty.

GRC teams are often the only people positioned to see this coming. Engineers built the platform to be useful. Business teams adopted it because it is useful. The question of what it has quietly turned into usually belongs to no one, unless someone in risk can look at the architecture and ask the right questions about each layer.

That's exactly the skill we build in the AI Security and Governance track at AIFoundry. It's six weeks of hands-on work, designed by practitioners for GRC, risk and audit professionals, with no coding required. You map how a real AI system works in your browser, find where its data collects, test the controls and see whether they hold. Every module ends with something you can take back to work, from a threat model to a full risk assessment. You get certified by assessing a realistic company rolling out AI, not by sitting a multiple-choice exam.

Your company built its AI assistant to be safe. Make sure you're the person who can prove it still is.

Sources

Help Net Security: Your employees uploaded over a gig of files to GenAI tools last quarter