We’ve seen a lot of talks, frameworks, and guidelines over the past year on regulating ML. In this talk, I’ll discuss why the current norms and practices in the ML research and practice community don’t lend themselves well to being regulated and what types of changes need to take place, from scoping and formulation, to design and development, to evaluation and monitoring, in order to result in ML systems that could be reliably deployed to help achieve societal, business, and policy outcomes that we want.

It is often said that while the US innovates Europe regulates. But is this actually true? AI, machine learning and generative AI pose new risks and opportunities to our society and many hope or fear that Europe will have a heavy regulatory hand. However, this talk will demystify this assumption and show how EU-laws often under- AND overregulate at the same time. I will explore the actual obligations, shed light on their loopholes, expose shortfalls and explain how to fix them.

3 short talks in this session:

• Can copyright be reduced to privacy by Niva Elkin-Koren, Uri Hacohen, Roi Livni, Shay Moran.

• Merging (EU)-Regulation and Model Reporting by Danilo Brajovic, Vincent Philipp Göbels, Janika Kutz, Marco Huber

• Necessity of Processing Sensitive Data for Bias Detection and Monitoring: A Techno-Legal Exploration by Ioanna Papageorgiou, Carlos Mougan

Major advances in machine learning (ML) and artificial intelligence (AI) increasingly take the form of developing and releasing general-purpose models. These models are designed to be adapted by other businesses and agencies to perform a particular, domain-specific function. This process has become known as adaptation or fine-tuning. In order to understand questions about responsibility and regulation in an ecosystem where multiple parties produce applications in this way, we need reasonable models of the incentives that drive this type of production. Here we offer a model of this multi-party process, in which a general provider of machine learning technology brings the system to a certain level of performance, and one or more Domain-specialists adapt it for use in particular domains. Our model provides high-level takeaways for how incentives operate in this setting, and in this way it suggests how we might think about responsible development and regulation of these technologies. This is joint work with Ben Laufer and Hoda Heidari.

2 short talks in this session:

• SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore by Sewon Min, Suchin Gururangan, Eric Wallace, Weijia Shi, Hannaneh Hajishirzi, Noah Smith, Luke Zettlemoyer

• Detecting Pretraining Data from Large Language Models by Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, Luke Zettlemoyer

With AI already changing the way in which society addresses economic and national security challenges and opportunities, AI technologies must be developed and used in a trustworthy and responsible manner. Experts in industry, academia, and government are still assessing how to best measure and manage risks and impacts of AI systems. At NIST we believe that working together to develop and advance the science of AI safety will harness the power of AI to serve humanity. We have been and will continue developing tests and facilitating the development of standards – measurement science and standards that will allow industry, academia, and government to better map, measure and manage AI risks. This work is a necessary precursor to any compliance or conformity assessment, either voluntary or required. Delivering these needed measurements, standards, and other tools is a primary focus for NIST’s portfolio of AI efforts. This talk provides an overview of NIST Trustworthy and Responsible AI program.

As ML is becoming ubiquitous and creating new and impressive artifacts, regulatory agencies around the world are grappling with the unique properties of this new category of technology. In order to properly address these challenges, we argue for an approach centered on the dual pillars of rights and transparency to ensure that the technology is subject to the appropriate democratic governance. We outline some of the recent developments and proposals made by policymakers in this direction, how they connect to AI, and provide both organizational and technical tools to support well-informed regulation aligned with technology development now and in the future.

2 short talks in this session:

• Learning from Label Proportions: Bootstrapping Supervised Learners via Belief Propagation by Shreyas Havaldar, Navodita Sharma, Shubhi Sareen, Karthikeyan Shanmugam, Aravindan Raghuveer

• Towards a Post-Market Monitoring Framework for Machine Learning-based Medical Devices: A case study by Jean Feng, Adarsh Subbaswamy, Alexej Gossmann, Harvineet Singh, Berkman Sahiner, Mi-Ok Kim, Gene Pennello, Nicholas Petrick, Romain Pirracchio, Fan Xia

The complexity and black-box nature of LLMs makes it difficult to provide meaningful guarantees, which in turn complicates efforts to regulate and audit LLMs. In this talk, I will discuss how statistical guarantees on various properties of LLMs such as privacy (via differential privacy) and provenance (via watermarking or membership inference) provide powerful primitives for thinking about important regulatory issues such as copyright. At the same time, implementing and deploying these primitives can be challenging and I will discuss pitfalls and open problems in the interaction of various statistical guarantees and LLM deployment environments.

3 short talks in this session:

• Can LLM-Generated Misinformation Be Detected? by Canyu Chen, Kai Shu

• Towards Responsible Governance of Biological Design Tools by Richard Moulange, Max Langenkamp, Tessa Alexanian, Samuel Curtis, Morgan Livingston

• Learning to Walk Impartially on the Pareto Frontier of Fairness, Privacy, and Utility by Mohammad Yaghini, Patty Liu, Franziska Boenisch, Nicolas Papernot

Learning from Label Proportions (LLP) is a learning problem where only aggregate level labels are available for groups of instances, called bags, during training, and the aim is to get the best performance at the instance-level on the test data. This setting arises in domains like advertising and medicine due to regulatory guidelines concerning privacy. We propose a novel algorithmic framework for this problem that iteratively performs two main steps. For the first step (Pseudo Labeling) in every iteration, we define a Gibbs distribution over binary instance labels that incorporates a) covariate information through the constraint that instances with similar covariates should have similar labels and b) the bag level aggregated label. We then use Belief Propagation (BP) to marginalize the Gibbs distribution to obtain pseudo labels. In the second step (Embedding Refinement), we use the pseudo labels to provide supervision for a learner that yields a better embedding. Further, we iterate on the two steps again by using the second step's embeddings as new covariates for the next iteration. In the final iteration, a classifier is trained using the pseudo labels. Our algorithm displays strong gains against several SOTA baselines (upto ~12%) for the LLP Binary Classification problem on various dataset types - tabular and Image. We achieve these improvements with minimal computational overhead above standard supervised learning due to Belief Propagation, for large bag sizes, even for a million samples.

Existing work on trustworthy machine learning (ML) often focuses on a single aspect of trust in ML (e.g., fairness, or privacy) and thus fails to obtain a holistic trust assessment. Furthermore, most techniques often fail to recognize that the parties who train models are not the same as the ones who assess their trustworthiness. We propose a framework that formulates trustworthy ML as a multi-objective multi-agent optimization problem to address these limitations. A holistic characterization of trust in ML naturally lends itself to a game theoretic formulation, which we call regulation games. We introduce and study a particular game instance, the SpecGame, which models the relationship between an ML model builder and regulators seeking to specify and enforce fairness and privacy regulations. Seeking socially optimal (i.e., efficient for all agents) solutions to the game, we introduce ParetoPlay. This novel equilibrium search algorithm ensures that agents remain on the Pareto frontier of their objectives and avoids the inefficiencies of other equilibria. For instance, we show that for a gender classification application, the achieved privacy guarantee is 3.76× worse than the ordained privacy requirement if regulators do not take the initiative to specify their desired guarantees first. We hope that our framework can provide policy guidance.

Federated learning (FL) emerges as an effective collaborative learning framework to coordinate data and computation resources from massive and distributed clients in training. Such collaboration results in non-trivial intellectual property (IP) represented by the model parameters that should be protected and shared by the whole party rather than an individual user. Meanwhile, the distributed nature of FL endorses a malicious client the convenience to compromise IP through illegal model leakage to unauthorized third parties. To block such IP leakage, it is essential to make the IP identifiable in the shared model and locate the anonymous infringer who first leaks it. The collective challenges call for accountable federated learning, which requires verifiable ownership of the model and is capable of revealing the infringer's identity upon leakage. In this paper, we propose Decodable Unique Watermarking (DUW) for complying with the requirements of accountable FL. Specifically, before a global model is sent to a client in an FL round, DUW encodes a client-unique key into the model by leveraging a backdoor-based watermark injection. To identify the infringer of a leaked model, DUW examines the model and checks if the triggers can be decoded as the corresponding keys. Extensive empirical results show that DUW is highly effective and robust, achieving over 99% watermark success rate for Digits, CIFAR-10, and CIFAR-100 datasets under heterogeneous FL settings, and identifying the IP infringer with 100% accuracy even after common watermark removal attempts.

The draft AI Act is an effort led by European institutions to regulate the deployment and use of artificial intelligence.It is a notably difficult task, in part due to the polysemy of concepts such as artificial intelligence, covering topics such as foundational models, optimisation routines and rule-based models, among others.Furthermore, it gives a prism by which we can observe the wide variety of stakes different actors are pushing for. After an initial draft proposed by the Commission in 2021, the European Commission, Council and Parliament will now discuss and draft the final version as part of the trilogue phase.The existence of these three versions gives us a chance to understand the negociations happening between the different European institutions, and as such is an interesting look into the currents that shape the artificial intelligence ecosystem. In this paper we focus on the Commission, Council and Parliament proposals for the Act, and read them with a technical lens.In particular, we examine the technical concepts mobilized in the Act, and contextualize them in the wider sociotechnical environment surrounding artificial intelligence. For each main concept, we make a comparative analysis of each version, highlighting their differences and their impact.This paper is primarily geared towards computer scientists, data analysts and machine learning researchers, in order to clarify the tenets and decisions made in the current versions of the act.

Large-Language-Models (LLMs) are arguable the biggest breakthrough in artificial intelligence to date. Recently, they have come to the public Zeitgeist with a surge of media attention surrounding ChatGPT, a large generative language model released by OpenAI which quickly became the fastest growing application in history. This model achieved unparalleled human-AI conversational skills, and even passed various mutations of the popular Turing test which measures if AI systems have achieved general intelligence. Naturally, the world at large wants to utilize these systems for various applications, but in order to do-so in truly sensitive domains, the models must often be regulatable in order to be legally used. In this short paper, we propose one approach towards such systems by forcing them to reason using a combination of (1) human-defined concepts, (2) Case-Base Reasoning (CBR), and (3) counterfactual explanations. All of these have support in user testing and psychology that they are understandable and useful to practitioners of AI systems. We envision this approach will be able to provide transparent LLMs for text classification tasks and be fully regulatable and auditable.

Predictive models are often introduced under the rationale that they improve performance over an existing decision-making policy. However, it is challenging to directly compare an algorithm against a status quo policy due to uncertainty introduced by confounding and selection bias. In this work, we develop a regret estimator which evaluates differences in classification metrics across decision-making policies under confounding. Theoretical and experimental results demonstrate that our regret estimator yields tighter regret bounds than existing auditing frameworks designed to evaluate predictive models under confounding. Further, we show that our regret estimator can be combined with a flexible set of causal identification strategies to yield informative and well-justified policy comparisons. Our experimental results also illustrate how confounding and selection bias contribute to uncertainty in subgroup-level policy comparisons. We hope that our auditing framework will support the operationalization of regulatory frameworks calling for more direct assessments of predictive model efficacy.

Data (as well as computation) is key to the functionality of ML systems. Data protection has therefore become a focal point of policy proposals and existing laws that are pertinent to the governance of ML systems. Privacy laws and legal scholarship have long emphasized privacy responsibilities developers have to protect individual data subjects. As a consequence, technical methods for privacy-preservation have been touted as solutions to prevent intrusions to individual data in the development of ML systems while preserving their resulting functionality. Further, privacy-preserving machine learning (PPML) has been offered up as a way to address the tension between being "seen" and "mis-seen" - to build models that can be fair, accurate, and conservative in data use. However, a myopic focus on privacy-preserving machine learning obscures broader privacy harms facilitated by ML models. In this paper, we argue that the use of PPML techniques to "un-see" data subjects introduces privacy costs of a fundamentally different nature. Your data may not be used in its raw or "personal" form, but models built from that data still make predictions and influence you and people like you. Moreover, PPML has allowed data collectors to excavate crevices of data that no one could touch before. We illustrate these privacy costs with an example on targeted advertising and models built with private set intersection.

There is an increasing drive to integrate machine learning (ML) tools into the drug development pipeline, to improve success rates and efficiency in the clinical development pathway. The ML regulatory framework being developed is closely aligned with ML best practices. However, there remain significant and tangible practical gaps in translating best practice standards into a real-world clinical trial context.To illustrate the practical challenges to regulating ML in this context, we present a theoretical oncology trial in which a ML tool is applied to support toxicity monitoring in patients. We explore the barriers in the highly regulated clinical trial environment to implementing data representativeness, model interpretability, and model usability.

This paper explores the intersection of the upcoming AI Regulation and fair ML research, specifically examining the legal principle of "necessity" in the context of processing sensitive personal data for bias detection and monitoring in AI systems. Drawing upon Article 10 (5) of the AI Act, currently under negotiation, and the General Data Protection Regulation, we investigate the challenges posed by the nuanced concept of "necessity" in enabling AI providers to process sensitive personal data for bias detection and bias monitoring. The lack of guidance regarding this binding textual requirement creates significant legal uncertainty for all parties involved and risks a purposeful and inconsistent legal application. To address this issue from a techno-legal perspective, we delve into the core of the necessity principle and map it to current approaches in fair machine learning. Our objective is to bridge operational gaps between the forthcoming AI Act and the evolving field of fair ML and support an integrative approach of non-discrimination and data protection desiderata in the conception of fair ML, thereby facilitating regulatory compliance.

There is a growing concern that generative AI models may generate outputs that closely resemble the copyrighted input content used for their training. This worry has intensified as the quality and complexity of generative models have immensely improved, and the availability of extensive datasets containing copyrighted material has expanded. Researchers are actively exploring strategies to mitigate the risk of producing infringing samples, and a recent line of work suggests employing techniques such as differential privacy and other forms of algorithmic stability to safeguard copyrighted content.In this work, we examine whether algorithmic stability techniques such as differential privacy are suitable to ensure the responsible use of generative models without inadvertently violating copyright laws. We argue that there are fundamental differences between privacy and copyright that should not be overlooked. In particular, we highlight that although algorithmic stability may be perceived as a practical tool to detect copying, it does not necessarily equate to copyright protection. Therefore, if it is adopted as a standard for copyright infringement, it may undermine the intended purposes of copyright law

Governments and industries have widely adopted differential privacy as a measure to protect users’ sensitive data, creating the need for new implementations of differentially private algorithms. In order to properly test and audit these algorithms, a suite of tools for testing the property of differential privacy is needed. In this work we expand this testing suite and introduce RényiTester, an algorithm that can verify if a mechanism is Rényi differentially private. Our algorithm computes computes a lower bound of the Rényi divergence between the distributions of a mechanism on neighboring datasets, only requiring black-box access to samples from the audited mechanism. We test this approach on a variety of pure and Rényi differentially private mechanisms with diverse output spaces and show that RényiTester detects bugs in mechanisms' implementations and design flaws. While detecting that a general mechanism is differentially private is known to be NP hard, we empirically show that tools like RényiTester provide a way for researchers and engineers to decrease the risk of deploying mechanisms that expose users' privacy.

Conformal Prediction (CP) is a method of estimating risk or uncertainty when using Machine Learning to help abide by common Risk Management regulations often seen in fields like healthcare and finance. CP for regression can be challenging, especially when the output distribution is heteroscedastic, multimodal, or skewed. Some of the issues can be addressed by estimating a distribution over the output, but in reality, such approaches can be sensitive to estimation error and yield unstable intervals.~Here, we circumvent the challenges by converting regression to a classification problem and then use CP for classification to obtain CP sets for regression.~To preserve the ordering of the continuous-output space, we design a new loss function and present necessary modifications to the CP classification techniques.~Empirical results on many benchmarks shows that this simple approach gives surprisingly good results on many practical problems.

Randomized response and label aggregation are two common ways of sharing sensitive label information in a private way. In spite of their popularity in the privacy literature, there is a lack of consensus on how to compare the privacy properties of these two different mechanisms. In this work, we investigate the privacy risk of sharing label information for these privacy enhancing technologies through the lens of label reconstruction advantage measures. A reconstruction advantage measure quantifies the increase in an attacker's ability to infer the true label of an unlabeled example when provided with a private version of the labels in a dataset (e.g., averages of labels from different users or noisy labels output by randomized response), compared to an attacker that only observes the feature vectors, but may have prior knowledge of the correlation between features and labels. We extend the Expected Attack Utility (EAU) and Advantage of previous work to mechanisms that involve aggregation of labels across different examples. We theoretically quantify this measure for Randomized Response and random aggregates under various correlation assumptions with public features, and then empirically corroborate these findings by quantifying EAU on real-world data. To the best of our knowledge, these are the first experiments where randomized response and label proportions are placed on the same privacy footing.We finally point out that simple modifications to the random aggregate approach can provide extra DP-like protection.

Although large language models (LMs) are widely deployed, the data used to train them is rarely disclosed. Given the incredible scale of this data, up to trillions of tokens, it is all but certain that it inadvertently includes potentially problematic text such as copyrighted materials, personally identifiable information, and test data for widely reported reference benchmarks. However, we currently have no way to know which data of these types is included or in what proportions. In this paper, we study the pretraining data detection problem; given a piece of text and black-box access to an LM with no knowledge of its training data, can we determine if the model was trained on our text. To study this problem, we introduce a dynamic benchmark WIKIMIA and a new detection method MIN-K PROB. Our method is based on a simple hypothesis: an unseen example is likely to contain a few outlier words with low probabilities under the LM, while a seen example is less likely to have words with such low probabilities. MIN-K PROB can be applied without any knowledge about the pretrainig corpus or any additional training, departing from previous detection methods that require training a reference model on data that is similar to the pretraining data. Moreover, our experiments demonstrate that MIN-K PROB achieves a 7.4% improvement over these previous methods. Our analysis demonstrates that MIN-K PROB is an effective tool for detecting contaminated benchmark data and copyrighted content within LMs.

Anthropomorphization is the tendency to attribute human-like traits to non-human entities. It is prevalent in many social contexts - children anthropomorphize toys, adults do so with brands, and it is a literary device. It is also a versatile tool in science, with behavioral psychology and evolutionary biology meticulously documenting its consequences. With widespread adoption of AI systems, and the push from stakeholders to make it human-like through alignment techniques, human voice, and pictorial avatars, the tendency for users to anthropomorphize it increases significantly. We take a dyadic approach to understanding this phenomenon with large language models (LLMs) by studying (1) the objective legal implications, as analyzed through the lens of the recent blueprint of AI bill of rights and the (2) subtle psychological aspects customization and anthropomorphization. We find that anthropomorphized LLMs customized for different user bases violate multiple provisions in the legislative blueprint. In addition, we point out that anthropomorphization of LLMs affects the influence they can have on their users, thus having the potential to fundamentally change the nature of human-AI interaction, with potential for manipulation and negative influence. With LLMs being hyper-personalized for vulnerable groups like children and patients among others, our work is a timely and important contribution. We propose a conservative strategy for the cautious use of anthropomorphization to improve trustworthiness of AI systems.

In fairness audits, a standard objective is to detect whether a given algorithm performs substantially differently between subgroups. Properly powering the statistical analysis of such audits is crucial for obtaining informative fairness assessments, as it ensures a high probability of detecting unfairness when it exists. However, limited guidance is available on the amount of data necessary for a fairness audit, lacking directly applicable results concerning commonly used fairness metrics. Additionally, the consideration of unequal subgroup sample sizes is also missing. In this tutorial, we address these issues by providing guidance on how to determine the required subgroup sample sizes to maximize the statistical power of hypothesis tests for detecting unfairness. Our findings are applicable to audits of binary classification models and multiple fairness metrics derived as summaries of the confusion matrix. Furthermore, we discuss other aspects of audit study designs that can increase the reliability of audit results.

This work explores the evolution of watermarking techniques designed to preserve the integrity of digital image content, especially against perturbations encountered during image transmission. An overlooked vulnerability is unveiled: existing watermarks' detectability significantly drops against even moderate generative model modifications, prompting a deeper investigation into the societal implications from a policy viewpoint. In response, we propose ANCHMARK, a robust watermarking paradigm, which remarkably achieves a detection AUC exceeding 0.93 against perturbations from unseen generative models, showcasing a promising advancement in reliable watermarking amidst evolving image modification techniques.

Compact user representations (such as embeddings) form the backbone of personalization services. In this work, we present a new theoretical framework to measure re-identification risk in such user representations. Our framework, based on hypothesis testing, formally bounds the probability that an attacker may be able to obtain the identity of a user from their representation. As an application, we show how our framework is general enough to model important real-world applications such as the Chrome's Topics API for interest-based advertising. We complement our theoretical bounds by showing provably good attack algorithms for re-identification that we use to estimate the re-identification risk in the Topics API. We believe this work provides a rigorous and interpretable notion of re-identification risk and a framework to measure it that can be used to inform real-world applications.

Can governments build AI? In this paper, we describe an ongoing effort to develop "public AI"—publicly accessible AI models funded, provisioned, and governed by governments or other public bodies. Public AI presents both an alternative and a complement to standard regulatory approaches to AI, but it also suggests new technical and policy challenges. We present a roadmap for how the ML research community can help shape this initiative and support its implementation, and how public AI can complement other responsible AI initiatives.

Recent advancements in generative machine learning have enabled rapid progress in biological design tools (BDTs) such as protein structure and sequence prediction models. The unprecedented predictive accuracy and novel design capabilities of BDTs present new and significant dual-use risks. BDTs have the potential to improve vaccine design and drug discovery, but may also be misused deliberately or inadvertently to design biological agents capable of doing more harm or evading current screening techniques. Similar to other dual-use AI systems, BDTs present a wicked problem: how can regulators uphold public safety without stifling innovation? We highlight how current regulatory proposals that are primarily tailored toward large language models may be less effective for BDTs, which require fewer computational resources to train and are often developed in a decentralized, non-commercial, open-source manner. We propose a range of measures to mitigate misuse risks. These include measures to control model development, assess risks, encourage transparency, manage access to dangerous capabilities, and strengthen cybersecurity. Implementing such measures will require close coordination between developers and governments.

Deploying machine learning (ML) models often requires both fairness and privacy guarantees. Both objectives often present notable trade-offs with the accuracy of the model—the primary focus of most applications. Thus, utility is prioritized while privacy and fairness constraints are treated as simple hyperparameters. In this work, we argue that by prioritizing one objective over others, we disregard more favorable solutions where at least certain objectives could have been improved without degrading any other. We adopt impartiality as a design principle: ML pipelines should not favor one objective over another. We theoretically show that a common ML pipeline design that features an unfairness mitigation step followed by private training is non-impartial. Then, parting from the two most common privacy frameworks for ML, we propose FairDP-SGD and FairPATE to train impartially specified private and fair models. Because impartially specified models recover the Pareto frontiers, i.e., the best trade-offs between different objectives, we show that they yield significantly better trade-offs than models optimized for one objective and hyperparameter-tuned for the others. Thus, our approach allows us to mitigate tensions between objectives previously found incompatible.

The advent of Large Language Models (LLMs) has made a transformative impact. However, the potential that LLMs such as ChatGPT can be exploited to generate misinformation has posed a serious concern to online safety and public trust. A fundamental research question is: will LLM-generated misinformation cause more harm than human-written misinformation? We propose to tackle this question from the perspective of detection difficulty. We first build a taxonomy of LLM-generated misinformation. Then we categorize and validate the potential real-world methods for generating misinformation with LLMs. Then, through extensive empirical investigation, we discover that LLM-generated misinformation can be harder to detect for humans and detectors compared to human-written misinformation with the same semantics, which suggests it can have more deceptive styles and potentially cause more harm. We also discuss the implications of our discovery on combating misinformation in the age of LLMs and the countermeasures.

Algorithmic tools in employment contexts are often evaluated via the ""four-fifths rule," which measures disparities in selection rates between legally protected groups. While they have their origins in anti-discrimination law, the "four-fifths rule" and related statistical parity tests are flawed measures of discrimination. In this paper, we trace the origins of this class of tests through the law and computer science literatures and detail their limitations as applied to algorithmic employment tools, with a particular focus on the shift from retrospective auditing to prospective optimization. We then discuss the appropriate role for statistical parity tests in algorithmic governance, suggesting a combination of measures that may be more suitable for building and auditing models.

Advances in machine learning are closely tied to the creation of datasets. While dataset documentation is widely recognized as essential to the reliability, reproducibility, and transparency of ML, we lack a systematic empirical understanding of current dataset documentation practices. To shed light on this question, here we take Hugging Face - one of the largest platforms for sharing and collaborating on ML models and datasets - as a prominent case study. By analyzing all 7,433 dataset documentation on Hugging Face, our investigation provides an overview of the Hugging Face dataset ecosystem and insights into dataset documentation practices, yielding 5 main findings: (1) The dataset card completion rate shows marked heterogeneity correlated with dataset popularity: While 86.0\% of the top 100 downloaded dataset cards fill out all sections suggested by Hugging Face community, only 7.9\% of dataset cards with no downloads complete all these sections. (2) A granular examination of each section within the dataset card reveals that the practitioners seem to prioritize Dataset Description and Dataset Structure sections, accounting for 36.2\% and 33.6\% of the total card length, respectively, for the most downloaded datasets. In contrast, the Considerations for Using the Data section receives the lowest proportion of content, accounting for just 2.1\% of the text. (3) By analyzing the subsections within each section and utilizing topic modeling to identify key topics, we uncover what is discussed in each section, and underscore significant themes encompassing both technical and social impacts, as well as limitations within the Considerations for Using the Data section. (4) Our findings also highlight the need for improved accessibility and reproducibility of datasets in the Usage sections. (5) In addition, our human annotation evaluation emphasizes the pivotal role of comprehensive dataset content in shaping individuals' perceptions of a dataset card's overall quality. Overall, our study offers a unique perspective on analyzing dataset documentation through large-scale data science analysis and underlines the need for more thorough dataset documentation in machine learning research.

After a machine learning (ML)-based system is deployed in clinical practice, performance monitoring is important to ensure the safety and effectiveness of the algorithm over time.The goal of this work is to highlight the complexity of designing a monitoring strategy and the need for a systematic framework that compares the multitude of monitoring options.One of the main decisions is choosing between using real-world (observational) versus interventional data.Although the former is the most convenient source of monitoring data, it exhibits well-known biases, such as confounding, selection, and missingness.In fact, when the ML algorithm interacts with its environment, the algorithm itself may be a primary source of bias.On the other hand, a carefully designed interventional study that randomizes individuals can explicitly eliminate such biases, but the ethics, feasibility, and cost of such an approach must be carefully considered.Beyond the decision of the data source, monitoring strategies vary in the performance criteria they track, the interpretability of the test statistics, the strength of their assumptions, and their speed at detecting performance decay.As a first step towards developing a framework that compares the various monitoring options, we consider a case study of an ML-based risk prediction algorithm for postoperative nausea and vomiting (PONV).Bringing together tools from causal inference and statistical process control, we walk through the basic steps of defining candidate monitoring criteria, describing potential sources of bias and the causal model, and specifying and comparing candidate monitoring procedures.We hypothesize that these steps can be applied more generally, as techniques from causal inference can address other sources of biases as well.

Auditing data usage in machine learning models is crucial for regulatory compliance, especially with sensitive data like medical records. In this study, we scrutinize potential vulnerabilities within an acknowledged baseline method, Ensembled Membership Auditing (EMA), which employs membership inference attacks to determine if a specific model was trained using a particular dataset. We discover a novel False Negative Error Pattern in EMA when applied to large datasets, under adversarial methods like dropout, model pruning, and MemGuard. Our analysis across three datasets shows that larger convolutional models pose a greater challenge for EMA, but a novel metric-set analysis improves performance by up to $5\%$. To extend the applicability of our improvements, we introduce EMA-Zero, a GAN-based dataset auditing method that does not require an external calibration dataset. Notably, EMA-Zero performs comparably to EMA with synthetic calibration data trained on as few as 100 samples.

Personalization services, such as recommender systems, operate on vast amounts of user-item interactions to provide personalized content. To do so, they identify patterns in the available interactions and group users based on pre-existing offline or online social relations, or algorithmically determined similarities and differences. We refer to the relations created between users based on algorithmically determined constructs as algorithmically mediated user relations. However, prior works in the fields of law, technology policy, and philosophy, have identified the lack of existing algorithmic governance frameworks to account for this relational aspect of data analysis. Algorithmically mediated user relations have also not been adequately acknowledged in technical approaches, such as for data importance and privacy, where users are usually considered independent from one another. In this paper, we highlight this conceptual discrepancy in the context of recommendation algorithms and provide empirical evidence of the limitations of the user independence assumption. We discuss related implications and future practical directions for accounting for algorithmically mediated user relations.

As ML models have increased in capabilities and accuracy, so has the complexity of their deployments. Increasingly, ML model consumers are turning to service providers to serve the ML models in the ML-as-a-service (MLaaS) paradigm. As MLaaS proliferates, a critical requirement emerges: how can model consumers verify that the correct predictions were served, in the face of malicious, lazy, or buggy service providers?We present the first practical ImageNet-scale method to verify ML model inference non-interactively, i.e., after the inference has been done. To do so, we leverage recent developments in ZK-SNARKs (zero-knowledge succinct non-interactive argument of knowledge), a form of zero-knowledge proofs. ZK-SNARKs allows us to verify ML model execution non-interactively and with only standard cryptographic hardness assumptions. We provide the first ZK-SNARK proof of valid inference for a full-resolution ImageNet model, achieving 79% top-5 accuracy, with verification taking as little as one second. We further use these ZK-SNARKs to design protocols to verify ML model execution in a variety of scenarios, including verifying MLaaS predictions, verifying MLaaS model accuracy, and using ML models for trustless retrieval. Together, our results show that ZK-SNARKs have the promise to make verified ML model inference practical.

The realization that algorithms can perpetuate or exacerbate racial disparities in society has spurred significant research in the field of algorithmic fairness. Concisely, contending with racism has been a primary motivation and driver of research in this area. Though racism is a primary motivation, developing strategies to correct and/or prevent racist outcomes is an ongoing challenge in the field. In particular, racism tends to be concealed by seemingly “race-neutral” methods and rhetoric making it difficult to identify. How do we solve a problem that we cannot see? Scholars refer to this modern form of racism as colorblind racism which occurs when we observe a racially discriminatory outcome but the mechanism responsible for said outcome appears to have nothing to do with race. In this paper, we introduce the concept of racial disregard in algorithmic fairness. The three components are i) disregard for racial context or racial issues ii) disregard for lived experience and perspectives of minoritized people iii) disregard for ongoing harms caused by a legacy of discrimination. With this definition in hand, we explore racial disregard within existing algorithmic fairness research. We discuss how this more nuanced form of racism can enhance the ongoing research agenda in the field. Understanding racial disregard is crucial for addressing the racial disparities that have motivated much of the algorithmic fairness research agenda. Furthermore, recognizing the invisibility of racial disregard is essential in developing effective solutions to combat racial bias in algorithms. Ultimately, our simple conceptual framework helps identify occurrences of racial disregard to promote regard by attending to racism more directly instead of avoiding a main motivator and driver of research.

Data-driven tools are increasingly used to make consequential decisions. In recent years, they have begun to advise employers on which job applicants to interview, judges on which defendants to grant bail, lenders on which homeowners to give loans, and more. In such settings, different data-driven rules result in different decisions. The problem is, for every data-driven rule, there are exceptions. While a data-driven rule may be appropriate for some, it may not be appropriate for all. In this piece, we argue that existing frameworks do not fully encompass this view. As a result, individuals are often, through no fault of their own, made to bear the burden of being data-driven exceptions. We discuss how data-driven exceptions arise and provide a framework for understanding how we can relieve the burden on data-driven exceptions. Our framework requires balancing three considerations: individualization, uncertainty, and harm. Importantly, no single consideration trumps the rest. We emphasize the importance of uncertainty, advocating that decision-makers should utilize data-driven recommendations only if the levels of individualization and certainty are high enough to justify the potential harm resulting from those recommendations. We argue that data-driven decision-makers have a duty to consider the three components of our framework before making a decision, and connect these three components to existing methods.

Recently, diffusion models have demonstrated great potential for image synthesis due to their ability to generate high-quality synthetic data. However, when applied to sensitive data, privacy concerns have been raised about these models. In this paper, we evaluate the privacy risks of diffusion models through a \emph{membership inference (MI) attack}, which aims to identify whether a target example is in the training set when given the trained diffusion model. Our proposed MI attack learns a single quantile regression model that predicts (a quantile of) the distribution of reconstruction loss for each example. This enables us to identify a unique threshold on the reconstruction loss tailored to each example when determining their membership status. We show that our attack outperforms the prior state-of-the-art MI attack and avoids their high computational cost from training multiple shadow models. Consequently, our work enriches the set of practical tools for auditing the privacy risks of large-scale generative models.

It is widely known that state-of-the-art machine learning models — including vision and language models — can be seriously compromised by adversarial perturbations, so it is also increasingly relevant to develop capability to certify their performance in the presence of the most effective adversarial attacks. Our paper offers a new approach to certify the performance of machine learning models in the presence of adversarial attacks, with population level risk guarantees. In particular, given a specific attack, we introduce the notion of a $(\alpha,\zeta)$ machine learning model safety guarantee: this guarantee, which is supported by a testing procedure based on the availability of a calibration set, entails one will only declare that a machine learning model adversarial (population) risk is less than $\alpha$ (i.e. the model is safe) given that the model adversarial (population) risk is higher than $\alpha$ (i.e. the model is in fact unsafe), with probability less than $\zeta$. We also propose Bayesian optimization algorithms to determine very efficiently whether or not a machine learning model is $(\alpha,\zeta)$-safe in the presence of an adversarial attack, along with their statistical guarantees. We apply our framework to a range of machine learning models — including various sizes of vision Transformer (ViT) and ResNet models — impaired by a variety of adversarial attacks such as AutoAttack, SquareAttack and natural evolution strategy attack, in order to illustrate the merit of our approach. Of particular relevance, we show that ViT's are generally more robust to adversarial attacks than ResNets and ViT-large is more robust than smaller models. Overall, our approach goes beyond existing empirical adversarial risk based certification guarantees, paving the way to more effective AI regulation based on rigorous (and provable) performance guarantees.

The widespread adoption of diffusion models for creative uses such as image, video, and audio synthesis has raised serious legal and ethical concerns surrounding the use of training data and its regulation. Due to the size and complexity of these models, the effect of training data is difficult to characterize with existing methods, confounding regulatory efforts. In this work we propose a novel approach to trace the impact of training data using an encoded ensemble of diffusion models. In our approach, individual models in an ensemble are trained on encoded subsets of the overall training data to permit the identification of important training samples. The resulting ensemble allows us to efficiently remove the impact of any training sample. We demonstrate the viability of these ensembles for assessing influence and consider the regulatory implications of this work.

Regulating AI systems remains a complex and unsolved issue despite years of active research. Various governmental approaches are currently underway, with the European AI Act being a significant initiative in this domain. In the absence of official regulations, researchers and developers have been exploring their own methods to ensure the secure application of AI systems. One well-established practice is the usage and documentation of AI applications through data and model cards. Although data and model cards do not explicitly address regulation, they are widely adopted in practice and share common characteristics with regulatory efforts. This paper presents an extended framework for reporting AI applications based on use-case, data, model and deployment cards, specifically designed to address upcoming regulations by the European Union. The proposed framework aligns with industry practices and provides comprehensive guidance for regulatory compliance and transparent reporting. By documenting the development process and addressing key requirements, the framework aims to support the responsible and accountable deployment of AI systems in line with EU regulations, positioning developers well for future legal requirements.

We examined the world's first regulation on generative AI, China's Provisional Administrative Measures of Generative Artificial Intelligence Services, which came into effect in August 2023. Our assessment reveals that these Measures, while recognizing the technical advances of generative AI and seeking to govern its full life-cycle, present unclear distinctions regarding different roles in the value chain of generative AI including upstream foundational model providers and downstream deployers. The lack of distinction and clear legal status between different players in the AI value chain can have profound consequences. It can lead to ambiguity in accountability, potentially undermining the governance and overall success of AI services.

Artificial Intelligence Impact Assessments ("AIIAs"), a family of tools that provide structured processes to imagine the possible impacts of a proposed AI system, have become an increasingly popular proposal to govern AI systems in the US and EU. Recent efforts from government or private-sector organizations have proposed many diverse instantiations of AIIAs, which take a variety of forms ranging from open-ended questionnaires to graded score-cards. However, to date that has been limited evaluation of existing AIIA templates.We conduct a preliminary classroom study (N = 38) at an R1 university in an elective course focused on the societal and ethical implications of AI. We assign students to different organizational roles (e.g., an ML scientist or product manager) and ask participant teams to complete an AI impact assessment for two imagined AI systems and deployment contexts. In our thematic analysis of participants' responses to post-activity questionnaires, we find a consistent set of limitations shared by several existing AIIA instruments, which we group into concerns about their format and content, as well as the feasibility and effectiveness of the activity in foreseeing and mitigating potential harms. Drawing on the findings of this study, we provide recommendations for future work on developing and validating more effective AIIAs.

The legality of training language models (LMs) on copyrighted or otherwise restricted data is under intense debate. However, as we show, model performance significantly degrades if trained only on low-risk text (e.g., out-of-copyright books or government documents), due to its limited size and domain coverage. We present SILO, a new language model that manages this risk-performance tradeoff during inference. SILO is built by (1) training a parametric LM on the Open License Corpus (OLC), a new corpus we curate with 228B tokens of public domain and permissively licensed text and (2) augmenting it with a more general and easily modifiable nonparametric datastore (e.g., containing copyrighted books or news) that is only queried during inference. The datastore allows use of high-risk data without training on it, supports sentence-level data attribution, and enables data producers to opt out from the model by removing content from the store. These capabilities can foster compliance with data-use regulations such as the fair use doctrine in the United States and the GDPR in the European Union. Our experiments show that the parametric LM struggles on its own with domains not covered by OLC. However, access to the datastore greatly improves out of domain performance, closing 90% of the performance gap with an LM trained on the Pile, a more diverse corpus with mostly high-risk text. We also analyze which nonparametric approach works best, where the remaining errors lie, and how performance scales with datastore size. Our results suggest that it is possible to build high quality language models while mitigating legal risk.

Research in machine learning is making progress in fixing its own reproducibility crisis. Reinforcement learning (RL), in particular, faces its own set of unique challenges. Comparison of point estimates, and plots that show successful convergence to the optimal policy during training, may obfuscate overfitting or dependence on the experimental setup. Although researchers in RL have proposed reliability metrics that account for uncertainty to better understand each algorithm's strengths and weaknesses, the recommendations of past work do not assume the presence of out-of-distribution observations. We propose a set of evaluation methods that measure the robustness of RL algorithms under distribution shifts. The tools presented here argue for the need to account for performance over time while the agent is acting in its environment. In particular, we recommend time series analysis as a method of observational RL evaluation. We also show that the unique properties of RL and simulated dynamic environments allow us to make stronger assumptions to justify the measurement of causal impact in our evaluations. We then apply these tools to single-agent and multi-agent environments to show the impact of introducing distribution shifts during test time. We present this methodology as a first step toward rigorous RL evaluation in the presence of distribution shifts.