MMLS 2024

Topic: ThemeLLM: A Retrieval and Structuring Approach for Theme-Focused, LLM-Guided Scientific Exploration

Abstract: Large Language Models (LLMs) may bring unprecedent power for scientific discovery. However, current LLMs may still encounter major challenges for effective scientific exploration due to their lack of in-depth, theme-focused data and knowledge. Retrieval augmented generation (RAG) has recently become an interesting approach for augmenting LLMs with grounded, theme-specific datasets. We discuss the challenges of RAG and propose a retrieval and structuring (RAS) approach, which enhances RAG by improving retrieval quality and mining structures (e.g., extracting entities and relations and building knowledge graphs) to ensure its effective integration of theme-specific data with LLM. We show the promise of retrieval and structuring approach at augmenting LLMs and discuss its potential power for future LLM-enabled science exploration.

• Chong Liu - Communication-Efficient Federated Non-Linear Bandit Optimization
• Yiping Lu - Simulation-Calibrated Scientific Machine Learning
• Sepehr Dehdashtian - FairerCLIP: Debiasing CLIP’s Zero-Shot Predictions Using Functions in RKHSs
• Vishnu Boddeti - On the Biometric Capacity of Generative Face Models
• Rachel Newton - Optimality of POD for Data-Driven LQR With Low-Rank Structures
• Paul Hieu Nguyen - Oblique Bayesian Additive Regression Trees with non-axis aligned splits
• Shaoming Xu - Message Propagation Through Time: An Algorithm for Sequence Dependency Retention in Time Series Modeling
• Eng Hock Lee - Enhancing Materials Discovery with the LUPI Framework: A Novel Approach for Predicting Material Properties
• Tanawan Premsri - Tuning Language Models with Spatial Logic for Complex Reasoning

• Speaker 1: Tian Li (U Chicago)
Title: Tilted Losses in Machine Learning: Theory, Applications, and Recent Advances


Abstract: Exponential tilting is a technique commonly used in fields such as statistics, probability, information theory, and optimization to create parametric distribution shifts. In this talk, I introduce its usage in machine learning by exploring the use of tilting in risk minimization. The tilted empirical risk minimization (TERM) framework is a simple extension of ERM, which uses exponential tilting to flexibly tune the impact of individual losses. I describe several useful theoretical properties of TERM including its connections with other non-ERM objectives, and a multitude of applications regarding fairness and robustness. I will conclude the talk by discussing recent advances on leveraging the tilted risk framework to improve non-convex optimization and the statistical properties of TERM.



• Speaker 2: Samet Oymak (UMich)
Title: Hybrid Architectures for Next-Generation of Language Models


Abstract: The transformer architecture has catalyzed revolutionary advances in language modeling. However, recent convolution-based recipes, such as state-space models and Mamba, have become competitive with transformers. Motivated by this, we examine the shortcomings of purely-attention or purely-convolutional designs and how augmenting attention with convolution can provably overcome them. We first describe a diverse suite of associative recall (AR) and in-context learning tasks which aim to assess the model's capability to search the context window and retrieve relevant information to the query. We show that equipping a transformer with "short convolutions" empowers the model to solve AR tasks with length generalization and without positional encoding. Secondly, we show that, "long convolutions" provide a mechanism to effectively summarize the long context window into few summary tokens. Through this, we describe a fundamental tradeoff between the required amount of attention vs convolution: Specifically, the model can solve AR tasks by only attending these summary tokens rather than all of the context window, thereby facilitating computational efficiency. Finally, we describe MambaFormer, a hybrid Attention+Mamba model, and demonstrate its best-of-both-world performance for in-context learning. In summary, our findings reveal the fundamental benefits of augmenting transformers with convolution and advocate the use of hybrid architectures for next-generation LLMs and foundation models.



• Speaker 3: Raghu Pasupathy (Purdue)
Title: Deterministic and Stochastic Frank-Wolfe Recursion on Probability Spaces


Abstract: Motivated by an emergency response application, we consider smooth stochastic optimization problems over probability measures supported on compact subsets of the Euclidean space. With the \emph{influence function} as the first variational object, we construct a deterministic Frank-Wolfe (dFW) recursion for probability spaces. As in Euclidean spaces, dFW is made possible by a key lemma that expresses the solution to the infinite-dimensional Frank-Wolfe sub-problem as the solution to a finite-dimensional optimization problem. This in turn allows each iterate of the solution sequence to be expressed as a convex combination of the incumbent iterate and a Dirac measure concentrating on the minimum of the influence function at the incumbent iterate. To address common application contexts that have access only to Monte Carlo observations of the objective and influence function, we construct a stochastic Frank-Wolfe (sFW) variation that generates a random sequence of probability measures constructed using minima of increasingly accurate estimates of the influence function. We demonstrate that sFW's optimality gap sequence exhibits $O(1/k)$ complexity almost surely and in expectation for smooth convex objectives, and $O(1/\sqrt{k})$ (in Frank-Wolfe gap) for smooth non-convex objectives, where $k$ is the iteration count. Furthermore, we show that an easy-to-implement fixed-step, fixed-sample version of (sFW) exhibits exponential convergence to $\varepsilon$-optimality. We end with a central limit theorem on the observed objective values at the sequence of generated random measures. To further intuition, we include several illustrative examples with exact influence function calculations.

• Speaker 1: Jiayu Zhou (MSU)
Title: Knowledge Integration Enabling AI for Scientific Discovery


Abstract: Recent strides in artificial intelligence (AI) have set the stage for groundbreaking innovations. However, the journey towards fully harnessing AI's potential, particularly in the context of AI+X across numerous fields, has been filled with challenges. Central among these are the integration of heterogeneous data modalities, the analysis of data characterized by complex spatiotemporal structures, the adept handling of missing values, and the incorporation of domain-specific knowledge. In this talk, I will overview the scientific areas under my investigation, highlighting how each is influenced by these critical challenges. I will introduce the development of methodologies and theories aimed at enhancing knowledge integration to support multimodal learning, manage noisy datasets effectively, and facilitate the integration of domain expertise.



• Speaker 2: Aditya Balu (ISU)
Title: 3D Microstructure Generation Using Conditional Latent Diffusion Models for Organic Photovoltaics


Abstract: The development of a microstructure generation framework tailored to user-specific needs is crucial for understanding materials behavior through distinct processing-structure-property relationships. Recent advancements in generative modeling, particularly with Latent Diffusion Models (LDM), have significantly enhanced our ability to create high-quality images that fulfill specific user requirements. In this talk, we present a scalable framework that employs LDM to sample 3D microstructures (128x128x64) with over a million voxels customized to user specifications. This framework can also predict manufacturing conditions that facilitate the synthesis of sampled microstructures experimentally. Our work focuses on organic photovoltaics (OPV), but the architecture allows for potential extensions into other fields of materials science by adjusting the training dataset.



• Speaker 3: Chris Bartel (UMN)
Title: Accelerating materials synthesis with machine learning


Abstract: The emergence of high-throughput quantum chemical calculations has accelerated the rate at which we can predict new materials for various applications (batteries, solar cells, catalysts, etc.), but the successful synthesis of these materials has often become the slow step in materials design. Autonomous laboratories hold the potential to systematically explore various synthesis routes to new materials, alleviating the painstaking manual trial-and-error approach. However, for an autonomous laboratory to work for inorganic synthesis, we need methods to initialize, interpret, and optimize synthesis recipes without any human intervention. This talk will focus on the application of machine learning to the initialization (recommending precursors) and interpretation (identifying phases from X-ray diffraction) steps.

Topic: The puzzle of dimensionality and feature learning in neural networks and kernel machines
Abstract: Remarkable progress in AI has far surpassed expectations of just a few years ago.
At their core, modern models, such as transformers, implement traditional statistical models -- high order Markov chains. Nevertheless, it is not generally possible to estimate Markov models of that order given any possible amount of data. Therefore these methods must implicitly exploit low-dimensional structures present in data. Furthermore, these structures must be reflected in high-dimensional internal parameter spaces of the models. Thus, to build fundamental understanding of modern AI, it is necessary to identify and analyze these latent low-dimensional structures. In this talk, I will discuss how deep neural networks of various architectures learn low-dimensional features and how the lessons of deep learning can be incorporated in non-backpropagation-based algorithms that we call Recursive Feature Machines. I will provide a number of experimental results on different types of data, as well as some connections to classical sparse learning methods, such as Iteratively Reweighted Least Squares.

• Speaker 1: Kirthevasan (kirthi) Kandasamy (U Wisc)
Title: Data without Borders: Game-theoretic Challenges in Democratizing Data


Abstract: Due to the popularity of machine learning, many organizations view data as an invaluable resource, likening it to the "new oil/gold". However, unlike many types of resources, data is nonrivalrous: it can be freely replicated and used by many. Hence, data produced by one organization, can, in principle, generate limitless value to many others. This will accelerate economic, social, and scientific breakthroughs and benefit society at large. However, considerations of free-riding and competition may prevent such open sharing of data between organizations. An organization may be wary that others may not be contributing a sufficient amount of data, or contributing fabricated/poisoned datasets. Organizations may also wish to monetize the data they have for profit. In some recent work, we leverage ideas from game theory, market design, and robust statistics to design protocols for data sharing. Our methods incentivize organizations to collect and truthfully contribute large amounts of data, so that socially optimal outcomes can be achieved.
In this talk, I will present a high level view of some of our recent approaches to solving these challenges and focus on a mean estimation problem. Here, a set of strategic agents collect i.i.d samples from a high dimensional distribution at a cost, and wish to estimate the mean of this distribution. To facilitate collaboration, we design mechanisms that incentivize agents to collect a sufficient amount of data and share it truthfully, so that they are all better off than working alone. Our approach prevents under-collection and data fabrication via two key techniques: first, when sharing the others’ data with an agent, the mechanism corrupts this dataset proportional to how much the data reported by the agent differs from the others; second, we design minimax optimal estimators for the corrupted dataset. Our mechanism, which is Nash incentive compatible and individually rational, achieves a social penalty (sum of all agents’ estimation errors and data collection costs) that is close to the global minimum.



• Speaker 2: Brian Ziebart (UIC)
Title: Aligned Imitation Learning for More Capable Imitators


Abstract: Given demonstrations of sequential decision making, imitation learning seeks a policy that performs competitively with the demonstrator (when evaluated on the demonstrator's unknown reward function). Prevalent imitation learning methods assume that demonstrations are (near-)optimal. For example, inverse reinforcement learning estimates a reward function that best rationalizes demonstrations, and then imitates using a policy that optimizes the estimated reward function. As imitators become more capable than demonstrators, the (near-)optimality assumption does not hold and these methods can lead to value misalignment. This talk presents subdominance minimization as an alternative imitation learning objective for robustly aligning the imitator with the demonstrator's reward function, even under differences in demonstrator-imitator capabilities.



• Speaker 3: Frederic Sala (U Wisc)
Title: Data-Efficient Adaptation for Pretrained Decision-Making Models


Abstract: The use of pretrained models forms the major paradigm change in machine learning workflows this decade, including for decision making. These powerful and typically massive models have the promise to be used as a base for diverse applications. Unfortunately, it turns out that adapting these models for downstream tasks tends to be difficult and expensive, often requiring collecting and labeling additional data to further train or fine-tune the model. In this talk, I will describe my group's work on addressing this challenge via efficient adaptation. First, when adapting vision-language models to make robust predictions, we show how to self-guide the adaptation process, without any additional data. Second, we show how to integrate relational structures like knowledge graphs into model prediction pipelines, enabling models to adapt to new domains unseen during training, without additional annotated examples. Lastly, in the most challenging scenarios, when the model must be fine-tuned on labeled data, we show how to obtain this data efficiently through techniques called weak supervision.



• Speaker 4: Yulong Lu (UMN)
Title: On the generalization of diffusion models in high dimensions


Abstract: Diffusion models, particularly score-based generative models (SGMs), have emerged as powerful tools in diverse machine learning applications, spanning from computer vision to modern language processing. In this talk, I will discuss about the generalization theory of SGMs for learning high-dimensional distributions. Our analysis show that SGMs achieve a dimension-free generation error bound when applied to a class of sub-Gaussian distributions characterized by certain low-complexity structures.

• Speaker 1: Ismini Lourentzou (UIUC)
Title: Zero-Shot Natural Language Video Localization


Abstract: Zero-shot Natural Language-Video Localization (NLVL) has emerged as a promising approach by training NLVL models exclusively on raw video data. Most methods employ dynamic video proposal and pseudo-query annotation generation modules to extract video segments and their corresponding text queries. However, a common challenge encountered is effectively grounding the generated textual annotations within the source video context. This talk will explore the importance of commonsense as a cross-modal grounding mechanism for zero-shot NLVL and highlight possible future directions in cross-modal understanding.



• Speaker 2: Parisa Kordjamshidi (MSU)
Title: Compositional Reasoning over Natural Language Leveraging Neuro-Symbolic AI


Abstract: Recent research indicates that large language models lack consistent reliability in tasks requiring complex reasoning. While they may impress us with fluently written articles prompted by user input, they can easily disappoint us by displaying shortcomings in basic reasoning skills, such as understanding that "left" is the opposite of "right." To address real-world problems, computational models often need to involve multiple interdependent learners, along with significant levels of composition and reasoning based on additional knowledge beyond available data. In this talk, I will discuss our findings and novel models for compositional reasoning over complex linguistic structures. I will highlight our efforts in neuro-symbolic modeling to integrate explicit symbolic knowledge and enhance the compositional generalization of neural learning models. Additionally, I will introduce DomiKnowS, our library that facilitates neuro-symbolic modeling. DomiKnowS framework exploits both symbolic and sub-symbolic representations to solve complex, AI-complete problems. It seamlessly integrates domain knowledge in the form of logical constraints in deep models through various underlying algorithms.



• Speaker 3: Manling Li (Northwestern)
Title: The Missing Knowledge in LLMs to Perceive and Interact with Physical World


Abstract: Recent breakthroughs in foundation models have unlocked exciting possibilities for embodied AI agents that can perceive and interact with the physical world. However, despite their impressive performance on various benchmarks, these models perceive images as bags of words. In detail, they use object understanding as a shortcut but lacks ability to do abstraction and reasoning, such as solving a maze. To acquire knowledge about the physical world, we initially categorize it based on its low-level physical and geometric visual features (from semantic to geometric) and its long horizon (from short/fast thinking to long/slow thinking). My research aims to bring this knowledge view to the multimodal world. Such a transformation poses significant challenges: (1) abstracting multimodal low-level geometric structures by introducing and training a low-level abstract layer that serves as a mental model; (2) enabling long-horizon reasoning by inducing complex patterns. Subsequently, we will examine the reason of hallucinations and explore potential methods for ensuring factuality through knowledge-driven approaches, with applications such as meeting summarization, timeline generation, and question answering. I will then lay out how I plan to promote factuality and truthfulness in multimodal information access, through a structured knowledge abstraction that is easily explainable, highly compositional, and capable of long-horizon reasoning.



• Speaker 4: Scott McCloskey (Kitware)
Title: End-to-end Machine Learning for Co-optimized Sensing and Computer Vision


Abstract: Many sensors produce data that rarely, if ever, is viewed by a human, and yet sensors are often designed to maximize subjective image quality. For sensors whose data is intended for embedded exploitation, maximizing the subjective image quality to a human will generally decrease the performance of downstream exploitation. In recent years, computational imaging researchers have developed end-to-end learning methods that co-optimize the sensing hardware with downstream exploitation via end-to-end machine learning. This talk will describe two such approaches at Kitware. In the first, we use an end-to-end ML approach to design a multispectral sensor that’s optimized for scene segmentation and, in the second, we optimize post-capture super-resolution in order to improve the performance of airplane detection in overhead imagery.