Skip to content
A global perspective on digital financeAbout The Token Press
CRYPTO. FINANCE. WHAT’S NEXT.The Token PressYour daily briefing ↗

Intermediate · 6 min read

How to Test an AI App for Private Document Leaks

Use two test accounts and fictional files to check document access, retrieved passages, cached answers and revoked permissions before sharing an AI app.

The Token Press education desk · Updated

Learning path: Build with AI: ChatGPT, Astra, Gemini and a Practical Learning Path →

Start with a question the login screen cannot answer

Can User B retrieve User A's private file, even when the app hides it from B's library? Test that question before introducing real documents. A working login proves that the app can recognize an account; it does not demonstrate that every file, search result and saved answer is restricted correctly.

This guide is a release exercise for an app you own or are authorized to test. Use a staging environment, two ordinary test accounts and invented documents. It extends our app-building and RAG guides with an access matrix, a cache test and a revocation test. It does not require probing somebody else's service or uploading customer records.

An AI answer can disclose a file without providing a download button. Check the material selected for the model as well as the visible reply. A polite refusal is insufficient evidence if the application already sent another user's private passage to the model.

Create three files and write the access rules

Our original exercise uses two fictional workspaces. Account A belongs only to Cedar; Account B belongs only to Maple. A third session is signed out. In this exercise, neither account is an administrator, and there is no intentional sharing between workspaces.

Create CEDAR.txt containing 'Cedar test delivery word: amber-482'; MAPLE.txt containing 'Maple test delivery word: violet-731'; and PUBLIC.txt containing 'Public workshop opens at 10:00.' These markers are invented test text, not passwords. Give Cedar ownership of the first file, Maple ownership of the second, and everyone access to the public file.

The expected retrieval matrix has nine cases: A may retrieve Cedar and Public but not Maple; B may retrieve Maple and Public but not Cedar; the signed-out session may retrieve only Public. That gives five allowed cases and four denied cases. If your real product requires login for all material, change the public row's expectations before testing.

OWASP recommends expressing authorizations in a matrix of roles, features and, where needed, data. The point is to define the expected result before observing the app. Otherwise, an accidental disclosure can be mistaken for an intended sharing feature.

Run the matrix using ordinary sessions

Use separate browser profiles for A and B, plus a signed-out session. Confirm the displayed account and workspace before each case. Do not test everything through the owner's dashboard: elevated access can hide the restrictions ordinary users encounter.

For each matrix case, ask the test assistant for the delivery word in the specified file, or the opening time in Public. Record the account, workspace, document ID, operation, expected result, actual retrieved document IDs, response and request reference. Keep the identity values in a private test record; never include session cookies or access tokens in a shared report.

An allowed case passes only if the intended file can actually be retrieved. A denied case passes only if the restricted source is excluded and its content is absent from the response. An empty answer caused by a broken search service is not a successful permissions test.

Ask the developer to expose a redacted staging trace showing which test document IDs reached the retrieval and model-input stages. You do not need production document contents in logs. If that trace is unavailable, mark retrieval isolation unverified rather than inferring it from the final sentence.

Check database, search and file access separately

In Supabase, row-level security can restrict database operations using the caller's identity. Administrative secret-key access can bypass those policies when no user access token is supplied. The current documentation also explains that a request carrying a user token runs with that user's policies. Test the identity and permissions used by the actual request, not just the label of the client library.

Microsoft's application-managed search-filter pattern uses identity values to select results; the filter string itself does not authenticate anyone. The application must obtain trusted identity information, apply the appropriate filter to every query and prevent bypasses. A workspace name supplied by the browser or by a document is not sufficient evidence of membership.

For your own staging app, open the known Cedar document route while signed in as B. Have the developer test the same documented read operation with B's normal session, without changing credentials or guessing real customer IDs. Inspect search snippets, citation previews and export/download routes too. Hiding a library row does not exercise these entry points.

File hosting has its own rules. Supabase public buckets let anyone with the asset URL retrieve a file. Private buckets support authorized downloads and time-limited signed URLs. Decide whether a link is intended to be shareable: a deliberately issued signed link is a temporary access capability, so possession of that link can matter independently of the viewer's login. Do not treat it like an ordinary private page URL.

Repeat after a cached answer and revoked membership

A cache stores a result for reuse. OWASP advises separating tenant- or user-dependent cached values, including the attributes that affect the result, and checking authorization before returning protected cached data. A separate cache key does not replace the access check.

Add case 10 to our worksheet: let A ask for Cedar's delivery word, then let B ask the identical question. This intentionally warms the answer cache. B must not receive amber-482, a Cedar citation preview or Cedar's retrieved passage. Record whether the response used a cache; a fresh answer alone does not exercise the cached path.

Now revoke A's Cedar membership through your staging administrator workflow. Case 11 is a new request by A to the ordinary authenticated Cedar download route. Case 12 is a new answer request by A about Cedar. Under our exercise's immediate-revocation requirement, both must deny access. Record when the membership change took effect and whether the app refreshed or invalidated its authorization state.

Do not use a previously issued signed URL as a substitute for case 11. Its expiry and revocation behavior depend on the service and the way it was issued. Test that separate policy explicitly. Also distinguish denying future requests from erasing a file or answer already downloaded: access removal cannot retrieve a copy the reader already possesses.

Worked result: eleven passes still leave a release blocker

Imagine the nine retrieval cases behave correctly. Case 10 returns Cedar's cached delivery word to B, while cases 11 and 12 deny the revoked account. The worksheet records 11 passes out of 12, or about 91.7%. These are invented results, not a benchmark for any platform.

The useful finding is the one cross-account disclosure, not the attractive percentage. Keep the affected feature out of use with private documents until the cache path is corrected. Then repeat case 10 and the legitimate A-to-Cedar case: disabling all answers would stop the leak while also breaking the intended feature.

For a reproducible result, record a build identifier, configuration version, test time, expected matrix and redacted request references. Mark each case passed, failed or unverified. Do not combine an unobserved backend step with a pass just because the screen looked normal.

The twelve cases are a starting worksheet. They do not cover every role, organization, concurrent request or file lifecycle. Expand the matrix when adding collaborators, administrator functions, shared folders or background exports. Keep the small baseline runnable after each relevant change.

Visual explainerA failed boundary cannot be averaged away
Retrieval matrix9 checksThree sessions × three fictional files: five allowed, four denied.
Cached answer1 checkB must not receive A's cached private answer.
Access revoked2 checksA's new download and answer requests must respect the change.
11 of 12 passes ≈ 91.7%, but the one fictional cross-account leak still blocks private-document use.Original hypothetical worksheet. These are example outcomes, not completed security testing of a product.

Give your AI builder a bounded test brief

Provide the fictional fixture and your intended permissions, then ask for a map of the enforcement points before accepting a repair. Keep the existing interface and unrelated features outside the change. A failed access test calls for a specific control fix, not a different chatbot personality.

Prompt to adapt: In my authorized staging app, use only the two test accounts and three fictional files described here. Map database reads, retrieval filters, model inputs, answer caches and file-download routes. Write expected results for all twelve cases. Identify which server-side check enforces each rule. Add a failing regression test for any demonstrated leak, make the smallest repair, and show the observed result after rerunning it. Do not use administrative credentials as ordinary users, real customer documents or production secrets. Label anything you cannot inspect as unverified.

Review the test output and the actual changed permission rule. An AI-generated checklist is a plan until it has been run. These checks support a practical release decision; they do not certify that the application is secure.

  • Verify both ordinary account identities before testing.
  • Confirm allowed access still works as well as denied access.
  • Inspect retrieved test document IDs and the final answer separately.
  • Exercise cached responses, direct downloads and membership changes.
  • Keep failed and unverified cases visible in the release record.

Sources & further reading

Educational content. Examples are illustrative. Protocols, products and rules can change; consult the linked documentation for current details.

Explore the newsroom

Connect what you have learned with the latest reporting.