In this talk, we discuss the adaptation of non-parametric retrieval models to evolving online conversations. We demonstrate that a static neural encoder can simply replace datastores with up-to-date information to accommodate adaptation and deletion without degradation. Modern deep learning frameworks can achieve these goals by fine-tuning at regular time intervals, but require a great computational budget. Our best non-parametric approach consistently outperforms parametric models (BART’s encoder and sequence-to-sequence models) over the course of a year (48 weeks) with an average relative gain of 64.12% recall when the test distribution shifts and outperforms fine-tuned models with an average relative gain of 11.58% recall. Our empirical analysis highlights non-parametric techniques as a practical and promising direction for adaptation to distribution shifts, and may facilitate future work arising from temporality in real-world deployment of NLP systems that require minimal computational costs.

The prediction of Electronic Stopping Power for general ions and targets is a problem that lacks a closed-form solution. While full approximate solutions from first principles exist for certain cases, the most general model in use is a pseudo-empirical model. This paper presents our advances towards creating predictive models that leverage state-of-the-art Machine Learning techniques. A key component of our approach is the training data selection. We show results that outperform or are on par with the current best pseudo-empirical Stopping Power model as quantified by the Mean Absolute Percentage Error metric.

This paper presents a novel anomaly detection method, called AD-DMKDE, based on the use of Kernel Density Estimation (KDE) along with density matrices (a powerful mathematical formalism from quantum mechanics) and Fourier features. The proposed method was systematically compared with eleven state-of-the-art anomaly detection methods on various data sets is presented, and AD-DMKDE shows competitive performance. The method uses neural-network optimization to find the parameters of data embedding, and the prediction phase complexity of the proposed algorithm is constant relative to the training data size.

Privacy-preserving machine learning in data-sharing processes is an ever-critical task that enables collaborative training of Machine Learning (ML) models without the need to share the original data sources. It is especially relevant when an organization must assure that sensitive data remains private throughout the whole ML pipeline, i.e., training and inference phases. This paper presents an innovative framework that uses Representation Learning via autoencoders to generate privacy-preserving embedded data. Thus, organizations can share the data representation to increase machine learning models' performance in scenarios with more than one data source for a shared predictive downstream task.

The widespread of false information is a rising concern worldwide with critical social impact, inspiring the emergence of fact-checking organizations to mitigate misinformation dissemination. However, human-driven verification leads to a time-consuming task and a bottleneck to have checked trustworthy information at the same pace they emerge. Since misinformation relates not only to the content itself but also to other social features, this paper addresses automatic misinformation checking in social networks from a multimodal perspective. Moreover, as simply naming a piece of news as incorrect may not convince the citizen and, even worse, strengthen confirmation bias, the proposal is a modality-level explainable-prone misinformation classifier framework. Our framework comprises a misinformation classifier assisted by explainable methods to generate modality-oriented explainable inferences. Preliminary findings show that the misinformation classifier does benefit from multimodal information encoding and the modality-oriented explainable mechanism increases both inferences' interpretability and completeness.

Modern high-scoring models of vision in the brain score competition do not stem from Vision Transformers. However, in this paper, we provide evidence against the unexpected trend of Vision Transformers (ViT) being not perceptually aligned with human visual representations by showing how a dual-stream Transformer, a CrossViT a la Chen et al. (2021), under a joint rotationally-invariant and adversarial optimization procedure yields 2nd place in the aggregate Brain-Score 2022 competition (Schrimpf et al., 2020b) averaged across all visual categories, and at the time of the competition held 1st place for the highest explainable variance of area V4. In addition, our current Transformer-based model also achieves greater explainable variance for areas V4, IT and Behavior than a biologically-inspired CNN (ResNet50) that integrates a frontal V1-like computation module (Dapello et al., 2020). To assess the contribution of the optimization scheme with respect to the CrossViT architecture, we perform several additional experiments on differently optimized CrossViT’s regarding adversarial robustness, common corruption benchmarks, mid-ventral stimuli interpretation and feature inversion. Against our initial expectations, our family of results provides tentative support for an “All roads lead to Rome” argument enforced via a joint optimization rule even for non biologically-motivated models of vision such as Vision Transformers.

Health records and patient generated data (in health forums or social media) constitute what the FDA and the CDC refer to as "real world data", which can be extremely valuable and become ‘real world evidence’ to advance health research. However, using these data presents many challenges for large-scale studies, as it is sometimes misused.

In this talk, I will showcase some of the approaches my team has deployed to identify cohorts, reduce data bias, find what drives patients to switch medications, and enrich metadata for SARS-CoV-2 sequences in public repositories, among other projects that incorporate RWD using natural language processing and artificial intelligence techniques.

Direct Sampling is an algorithm that can generate synthetic data using only one training image and a set of conditioning points. This algorithm implicitly learns the conditional distribution of the probable values the data could take given a set of conditioning points and the training image.This algorithm does not learn an internal state, like parametric Machine Learning algorithms, but instead, it contains a pattern-matching algorithm that implicitly learns such conditional distribution. Thus, it is a non-parametric Machine learning algorithm that resembles the KNN approach. In this work, we explore the application of Direct Sampling for generating extreme precipitation events, which are precipitation weather fields with out-of-sample precipitation values. To this end, we propose to conditioning Direct Sampling not only in the training image and the conditioning points but also in a set of control points and a return precipitation level map to guide the out-of-sample precipitation value generation. We validate our approach with statistical metrics and connectivity metrics.

The performance of a reinforcement learning (RL) system depends on the computational architecture used to approximate a value function. Deep learning methods provide architectures for approximating nonlinear functions from noisy, high-dimensional observations. However, their prevailing optimization techniques are not designed for strictly-incremental online updates. Standard architectures are also not designed to efficiently represent observational patterns from an a priori unknown structure: for example, light receptors randomly dispersed in space. Nor are standard architectures designed to efficiently represent observational patterns from an a priori unknown structure: for example, light receptors randomly dispersed in space. We propose an online RL algorithm for adapting a value function’s architecture and efficiently finding useful nonlinear features. The algorithm is evaluated in a spatial domain with high-dimensional, stochastic observations. We further show that the algorithm outperforms baselines and approaches the performance of an architecture given side-channel information about observational structure.

In this work, we propose a framework relying solely on chat-based customer support (CS) interactions for predicting the recommendation decision of individual users. For our case study, we analyzed a total number of 16.4k users and 48.7k customer support conversations within the financial vertical of a large e-commerce company in Latin America. Consequently, our main contributions and objectives are to use Natural Language Processing (NLP) to assess and predict the recommendation behavior where, in addition to using static sentiment analysis, we exploit the predictive power of each user's sentiment dynamics. Our results show that, with respective feature interpretability, it is possible to predict the likelihood of a user to recommend a product or service, based solely on the message-wise sentiment evolution of their CS conversations in a fully automated way.

We study the problem of recognizing personality from videos depicting users' social interaction. Multimodal information is represented using pretrained models, and multi-stream sequential models are considered for prediction. Experimental results of the proposed method in the recently released UDIVA dataset are reported and compared to related work. We show that the proposed methodology is competitive with the state-of-the-art while using less complex models.

Experts across diverse academic and professional disciplines struggle with making sense of large amounts of linguistic data. Automatically uncovering latent themes from large textual resources remains an open challenge in natural language processing. Traditionally, researchers and practitioners approach this challenge using noisy unsupervised techniques such as topic models, or by manually identifying the relevant themes and annotating them in the text. In this paper, we propose an interactive framework that combines computational and qualitative techniques to discover and ground latent themes in large text collections. Our framework strikes a balance between automated techniques and manual coding, allowing experts to maintain control of their study while reducing the manual effort required.

Pathways into AI: Learn about the diverse pathways Meta researchers have taken from education into industry. They will share from personal experience how they arrived in their current role along with their perspective on how to expand the field to enable a more representative community of researchers within industry.

Keren Fuentes, AI Resident, FAIR Labs Keren is an AI Resident on the Responsible AI team working on the evaluation of fairness metrics for language models. Prior to joining Meta, she was a software engineer at Microsoft. She is originally from Mexico. Maria Lomeli, Research Engineer, FAIR Labs Maria is a Research Engineer at FAIR Labs. Prior to joining Meta, she was a research scientist at Babylon Health working in ML for healthcare. She completed a PhD in Machine Learning at the Gatsby Computational Neuroscience Unit, University College London and a postdoc at the Machine Learning group in the University of Cambridge. She is originally from Mexico. Ricardo Monti, Research Scientist, CTRL Research Ricardo is a Research Scientist within the CTRL team at Meta Reality Labs. Prior to joining Meta, he completed his PhD within the Statistics Department of Imperial College London and a PostDoc with Aapo Hyvarinen at the Gatsby Computational Neuroscience Unit, UCL. He is originally from Argentina.

It is known that adversarial examples can exploit deep learning models’ vulnerabilities to cause harm or gain unfair advantages. Recently, many works have tried to improve adversarial examples to re-train models and make them more robust. However, we have shown in our prior work that those mechanisms negatively affect different measures of fairness, which are critical in practice. This talk will show how adversarial training decreases fairness scores and how we can make assessments and estimations of such behavior. Furthermore, we show how to evaluate adversarial robustness in classic and generative models using computer vision datasets. These estimations can help researchers define creative objective functions for safe, robust, trustworthy models.

It has become an important goal of machine learning to develop methods that are exactly (or approximately) equivariant to group actions. Equivariant functions obey relations like f(g x) = g f(x); that is, if the inputs x are transformed by group element g, then the outputs f(x) are correspondingly transformed. There are two different kinds of symmetries that can be encoded by these equivariances: active symmetries that are observed regularities in the laws of physics, and passive symmetries that arise from redundancies in the allowed representations of the physical objects. In the first category are the symmetries that lead to conservation of momentum, energy and angular momentum. In the second category are coordinate freedom, units equivariance, and gauge symmetry, among others. Passive symmetries always exist, even in situations in which the physical law is not actively symmetric. For example, the physics near the surface of the Earth is very strongly oriented (free objects fall in the down direction, usually), and yet the laws can be expressed in a perfectly coordinate-free way by making use of the local gravitational acceleration vector. The passive symmetries seem trivial, but they can lead naturally to the discovery of scalings, mechanistic structures, and missing geometric and dimensional quantities, even with very limited training data. Our conjecture is that enforcing passive symmetries in machine-learning models will improve generalization (both in and out of sample) in all areas of engineering and the natural sciences. In this talk we explain how to parameterize functions that satisfy (some) symmetries, using classical invariant theory.

The prediction of Electronic Stopping Power for general ions and targets is aproblem that lacks a closed-form solution. While full approximate solutionsfrom first principles exist for certain cases, the most general model in use isa pseudo-empirical model. This paper presents our advances towards creatingpredictive models that leverage state-of-the-art Machine Learning techniques. Akey component of our approach is the training data selection. We show results thatoutperform or are on par with the current best pseudo-empirical Stopping Powermodel as quantified by the Mean Absolute Percentage Error metric.

Low Surface Brightness Galaxies (LSBGs) constitute an important segment of the galaxy population, however, due to their diffuse nature, their search is challenging. The detection of LSBGs is usually done with a combination of parametric methods and visual inspection, which becomes unfeasible for future astronomical surveys that will collect petabytes of data. Thus, in this work we explore the usage of Locality-Sensitive Hashing for the approximate similarity search of LSBGs in large astronomical catalogs. We use 11670190 objects from the Dark Energy Survey Y3 Gold coadd catalog to create an approximate k nearest neighbors model based on the properties of the objects, developing a tool able to find new LSBG candidates while using only one known LSBG. From just one labeled example we are able to find various known LSBGs and many objects visually similar to LSBGs but not yet catalogued. Also, due to the generality of similarity search models, we are able to search for and recover other rare astronomical objects without the need of retraining or generating a large sample. Our code is available on https://github.com/zysymu/lsh-astro.

This paper presents a novel anomaly detection method, called AD-DMKDE, based on the use of Kernel Density Estimation (KDE) along with density matrices (a powerful mathematical formalism from quantum mechanics) and Fourier features. The proposed method was systematically compared with eleven state-of-the-art anomaly detection methods on various data sets is presented, and AD-DMKDE shows competitive performance. The method uses neural-network optimization to find the parameters of data embedding, and the prediction phase complexity of the proposed algorithm is constant relative to the training data size.

Experts across diverse disciplines are often interested in making sense of large text collections. Traditionally, this challenge is approached either by noisy unsupervised techniques such as topic models, or by following a manual theme discovery process. In this paper, we expand the definition of a theme to account for more than just a word distribution, and include generalized attributes and concepts emerging from the data. Then, we propose an interactive neuro-symbolic framework that receives expert feedback at different levels of abstraction. Our framework strikes a balance between automation and manual coding, allowing experts to maintain control of their study while reducing the manual effort required.This is a non-archival submission.

Heart valves consist of leaflets that can degrade due to a range of disease processes. To better design prostheses, it is critical to study leaflet mechanics. Although mechanical testing of heart valve leaflets (HVLs) is the standard for evaluating mechanical behavior, imaging and deep learning (DL) networks, such as convolutional neural networks (CNNs), are more readily available and cost-effective. In this work, we determined the influence that a dataset that we curated had on the ability of a CNN to predict the stress-strain response of the leaflets. Our findings indicate that CNNs can be used to predict the polynomial coefficients needed for reconstructing the toe and linear regions of typically observed mechanical behavior, which lie near the physiological strain, 10\% strain.

The widespread of false information is a rising concern worldwide with critical social impact, inspiring the emergence of fact-checking organizations to mitigate misinformation dissemination. However, human-driven verification leads to a time-consuming task and a bottleneck to have checked trustworthy information at the same pace they emerge. Since misinformation relates not only to the content itself but also to other social features, this paper addresses automatic misinformation checking in social networks from a multimodal perspective. Moreover, as simply naming a piece of news as incorrect may not convince the citizen and, even worse, strengthen confirmation bias, the proposal is a modality-level explainable-prone misinformation classifier framework. Our framework comprises a misinformation classifier assisted by explainable methods to generate modality-oriented explainable inferences. Preliminary findings show that the misinformation classifier does benefit from multimodal information encoding and the modality-oriented explainable mechanism increases both inferences' interpretability and completeness.

Modern high-scoring models of vision in the brain score competition do not stem from Vision Transformers. However, in this paper, we provide evidence against the unexpected trend of Vision Transformers (ViT) being not perceptually aligned with human visual representations by showing how a dual-stream Transformer, a CrossViT a la Chen et al. (2021), under a joint rotationally-invariant and adversarial optimization procedure yields 2nd place in the aggregate Brain-Score 2022 competition (Schrimpf et al., 2020b) averaged across all visual categories, and at the time of the competition held 1st place for the highest explainable variance of area V4. In addition, our current Transformer-based model also achieves greater explainable variance for areas V4, IT and Behavior than a biologically-inspired CNN (ResNet50) that integrates a frontal V1-like computation module (Dapello et al., 2020). To assess the contribution of the optimization scheme with respect to the CrossViT architecture, we perform several additional experiments on differently optimized CrossViT’s regarding adversarial robustness, common corruption benchmarks, mid-ventral stimuli interpretation and feature inversion. Against our initial expectations, our family of results provides tentative support for an “All roads lead to Rome” argument enforced via a joint optimization rule even for non biologically-motivated models of vision such as Vision Transformers.

Direct Sampling is an algorithm that can generate synthetic data using only one training image and a set of conditioning points. This algorithm implicitly learns the conditional distribution of the probable values the data could take given a set of conditioning points and the training image.This algorithm does not learn an internal state, like parametric Machine Learning algorithms, but instead, it contains a pattern-matching algorithm that implicitly learns such conditional distribution. Thus, it is a non-parametric Machine learning algorithm that resembles the KNN approach. In this work, we explore the application of Direct Sampling for generating extreme precipitation events, which are precipitation weather fields with out-of-sample precipitation values. To this end, we propose to conditioning Direct Sampling not only in the training image and the conditioning points but also in a set of control points and a return precipitation level map to guide the out-of-sample precipitation value generation. We validate our approach with statistical metrics and connectivity metrics.

Recently, with the advancement of deep learning, several applications in text classification have advanced significantly. However, this improvement comes with a cost because deep learning is vulnerable to adversarial examples. This weakness indicates that deep learning is not very robust. Fortunately, the input of a text classifier is discrete. Hence, it can prevent the classifier from state-of-the-art attacks. Nonetheless, previous works have generated black-box attacks that successfully manipulate the discrete values of the input to find adversarial examples. Therefore, instead of changing the discrete values, we transform the input into its embedding vector containing real values to perform the state-of-the-art white-box attacks. Then, we convert the perturbed embedding vector back into a text and name it an adversarial example. In summary, we create a framework that measures the robustness of a text classifier by using the gradients of the classifier.

Sign Language Recognition (SLR) models rely heavily on advances reached by the Human Action Recognition (HAR). One of the simplest and most dimensional-reduced modality is the skeleton joints and limbs represented with key-point landmarks and edges connecting these landmarks. These skeletons can be obtained by pose estimation, depth maps or motion capture. For HAR, models are usually interested in less granularity of pose estimation, compared to SLR, where it is highly important the landmark estimation of not only the pose and body but the facial gestures, hands and fingers. In this work, we compare three whole-body estimation libraries/models that are gaining attraction in the SLR task. We first find their relation by identifying common keypoints in their landmark structure and analyzing their quality. Then, we complement this analysis by comparing their annotations in three sign language datasets with videos of different quality, background, and region (Peru and USA). We test a sign language recognition model to compare the quality of the annotations provided by these libraries/models.

The development of autonomous vehicles arises new challenges in urban trafficscenarios where vehicle-pedestrian interactions are frequent e.g. vehicle yields topedestrians, pedestrian slows down due approaching to the vehicle. Over the lastyears, several datasets have been developed to model these interactions. However,available datasets do not cover near-accident scenarios that our dataset covers.We introduce I see you, a new vehicle-pedestrian interaction dataset that tacklesthe lack of trajectory data in near-accident scenarios using YOLOv5 and cameracalibration methods. I see you consist of 170 near-accident occurrences in sevenintersections in Cusco-Peru. This new dataset and pipeline code are available onGitHub.

Discriminating fine movements within the same limb using electroencephalography (EEG) signals is a current challenge to non-invasive BCIs systems due to the close spatial representations on the motor cortex area of the brain, the signal-to-noise ratio, and the stochastic nature of this kind of signals. This research presents the performance evaluation of different strategies of classification using Linear Discriminant Analysis (LDA) method and power spectral density (PSD) features for three tasks: make a fist, open the hand, and keep the anatomical position of the hand. For this, EEG signals were collected from 10 healthy subjects and evaluated with different cross-validation methods: Monte Carlo, to implement an Offline Analysis and Leave-one-out for a pseudo-online implementation. The results show that the average accuracy for classifying the start of each task is approximately 76% for offline and Pseudo-online Analysis, classifying just the start of movement is 54% and 62% respectively for same both methods and 45% for and 32% classifying between classes. Based on these results, it can be said that the implementation of a BCI based on PSD features and LDA method could work to detect the start of one of the proposed tasks, but to discriminate the movement it is necessary to implement a different strategy for increase accuracy in the classification problem.

Procedural step generation (i.e. instruction generation) is an important task, particularly because of its applicability on fields such as usability of technology and education. An effective procedural step generation system should be capable of generating steps to guide a person to accomplish a procedure, and in the case of a mishap in the procedure, generate steps to correct the mishap and continue and complete the original procedure. Procedural step generation is complicated for modern natural language systems, particularly because of knowledge that may be implicit in the steps, and essential for its completion. We present a work-in-progress of a multi-step system that is capable of, given a goal and facts related to a goal, to generate procedural steps to accomplish a procedure. The system is also capable of reusing past knowledge through a neural memory to handle goal changes (i.e., mishaps) while performing procedures. To accomplish this, we leverage a contextual commonsense inference model which can generate contextually relevant facts (i.e., assertions) about a procedure, and train a model which selects facts that are necessary to accomplish a goal and translate these into a procedural step.

Deep learning architectures lead the state-of-the-art in several computer vision, natural language processing, and reinforcement learning tasks due to their ability to extract multi-level representations without human engineering. The model's performance is affected by the amount of labeled data used in training. Hence, novel approaches like self-supervised learning (SSL) extract the supervisory signal using unlabeled data.Although SSL reduces the dependency on human annotations, there are still two main drawbacks. First, high-computational resources are required to train a large-scale model from scratch. Second, knowledge from an SSL model is commonly finetuning to a target model, which forces them to share the same parameters and architecture and make it task-dependent.This paper explores how SSL benefits from knowledge distillation in constructing an efficient and non-task-dependent training framework. The experimental design compared the training process of an SSL algorithm trained from scratch and boosted by knowledge distillation in a teacher-student paradigm using the video-based human action recognition dataset UCF101. Results show that knowledge distillation accelerates the convergence of a network and removes the reliance on model architectures.

The performance of a reinforcement learning (RL) system depends on the compu- tational architecture used to approximate a value function. We propose an online RL algorithm for adapting a value function’s architecture and efficiently finding useful nonlinear features. The algorithm is evaluated in a spatial domain with high-dimensional, stochastic observations. Our method outperforms non-adaptive baseline architectures and approaches the performance of an architecture given side- channel information about observational structure. These results are a step towards scalable RL algorithms for more general problem settings, where observational structure is unavailable.

In this work, we propose a framework relying solely on chat-based customer support (CS) interactions for predicting the recommendation decision of individual users. For our case study, we analyzed a total number of 16.4k users and 48.7k customer support conversations within the financial vertical of a large e-commerce company in Latin America. Consequently, our main contributions and objectives are to use Natural Language Processing (NLP) to assess and predict the recommendation behavior where, in addition to using static sentiment analysis, we exploit the predictive power of each user's sentiment dynamics. Our results show that, with respective feature interpretability, it is possible to predict the likelihood of a user to recommend a product or service, based solely on the message-wise sentiment evolution of their CS conversations in a fully automated way.

We study the problem of recognizing personality from videos depicting users' social interaction. Multimodal information is represented using pretrained models, and multi-stream sequential models are considered for prediction. Experimental results of the proposed method in the recently released UDIVA dataset are reported and compared to related work. We show that the proposed methodology is competitive with the state-of-the-art while using less complex models.

Understanding and measuring the effect of human mobility on the spread of epidemics is key to addressing these threats. GPS human mobility data represents an enormous advancement in the field of epidemics as it could reveal population-level contact patterns that can replace homogeneous mixing assumptions or the unjustified use of synthetic random networks in epidemic models. However, a standing challenge in the estimation of contacts from GPS signals is addressing the high sparsity in this type of data. Alas, most users are observed only for a small fraction of the time. In this paper, we address this issues by proposing a novel methodology that can fill in the gaps in the data. Our framework is based on link prediction using deep learning to predict missing links in a temporal bipartite graph connecting users and locations. We demonstrate and validate our methodology on privacy-enhanced location data from thousands of mobile devices in the city of Philadelphia during 2020.

Privacy-preserving machine learning in data-sharing processes is an ever-critical task that enables collaborative training of Machine Learning (ML) models without the need to share the original data sources. It is especially relevant when an organization must assure that sensitive data remains private throughout the whole ML pipeline, i.e., training and inference phases. This paper presents an innovative framework that uses Representation Learning via autoencoders to generate privacy-preserving embedded data. Thus, organizations can share the data representation to increase machine learning models' performance in scenarios with more than one data source for a shared predictive downstream task.