Detecting Inadvertent Disclosure of Confidential Business Data in Enterprise Generative AI Usage

Detecting Inadvertent Disclosure of Confidential Business Data in Enterprise Generative AI Usage Problem. As organizations adopt generative AI platforms for everyday work, employees increasingly paste unstruc- tured, free-text content into prompts. Industry research indicates that roughly one in five data breaches now involves unauthorized or ungoverned (shadow) AI use (IBM, 2025). This gives rise to a growing and largely unaddressed risk of confidential business data leakage like unreleased sales figures, financial projections, and internal plans to AI providers. Most extant work on AI-related data exposure has focused on models leaking their own training data or personally identifiable information (PII) through adversarial queries (Carlini et al., 2021; Lukas et al., 2023; Kim et al., 2023). The subproblem of employees voluntarily disclosing confidential business content through ordinary use still remains underserved. Unlike PII, this class of sensitive content lacks fixed structural patterns, thereby making them poorly suited to traditional rule-based data loss prevention (DLP) tools (which rely on structural matching). This class of methods cannot be extended handling disclosures of confidential business data as they do not reason about semantic or organizational context. Distinguishing a genuinely confidential disclosure (e.g., our unreleased Q3 revenue is $4.2M) from benign or already-public business language (e.g., publicly reported earnings) requires a model that captures contextual meaning rather than structured form. Approach. Our goal is to propose a supervised, classifier-based framework for post-hoc auditing of employee-AI interaction logs to flag prompts and uploads that disclose confidential business information. We propose to fine-tune a lightweight transformer encoder (DistilBERT; Sanh et al., 2019) as a binary sequence classifier over prompt text. The goal is to distinguish confidential business disclosures from non-sensitive business language. As real corporate data cannot be shared or used for training due to their sensitivity, we propose to construct a synthetic dataset of simulated business documents and prompts for our exploration. We will compare the outcomes from the proposed methodology against a traditional bag-of-words baseline (TF-IDF with logistic regression/SVM; Salton & Buckley, 1988) to isolate the contribution of contextual embeddings over keyword-level signal. Tentative Results. This work is in progress. We plan to report precision, recall, and MCC scores on a held-out synthetic test set. In addition, we will include an ablation study to compare the proposed classifier with the TF-IDF baseline (with the goal of quantifying the benefit of contextual representations for this task). Context. Prior work has largely focused on pattern-matchable data types and keywords. On the contrary, confiden- tial business data contains several diverse unstructured constructs like financial figures, strategic plans and unreleased product information. ML security literature has devoted comparatively little attention to detecting employee-initiated disclosures during ordinary use, despite this being a primary concern for enterprises adopting generative AI. Our proposed work differs from extant lines of research by i] targeting employee-originated disclosure of confidential business content rather than model-originated leakage of memorized or personal data, and ii] using a smaller, more deployable classifier suited to scalable, low-latency log auditing rather than adversarial probing or LLM-based rea- soning. Given the sensitivity and unavailability of the real corporate text for research use, we plan to contribute an evaluation framework involving synthetic data. References Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, Ú., Oprea, A., & Raffel, C. (2021). Extracting training data from large language models. In Proceedings of the 30th USENIX Security Symposium (pp. 2633–2650). USENIX Association. IBM. (2025). Cost of a data breach report 2025: The AI oversight gap. IBM Security / Ponemon Institute. Kim, S., Yun, S., Lee, H., Gubri, M., Yoon, S., & Oh, S. J. (2023). ProPILE: Probing privacy leakage in large language models. In Advances in Neural Information Processing Systems (Vol. 36). 1Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L., & Zanella-Béguelin, S. (2023). Analyzing leakage of personally identifiable information in language models. In 2023 IEEE Symposium on Security and Privacy (SP) (pp. 346–363). IEEE. https://doi.org/10.1109/SP46215.2023.10179300 Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Manage- ment, 24(5), 513–523. https://doi.org/10.1016/0306-4573(88)90021-0 Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. 2