Active Feature Acquisition: Is AI System-Building a Machine Learning Problem?
Thursday, November 19, 2026;
1:30 pm
W375 Westgate
Speaker: Jason Eisner from Johns Hopkins University
Abstract: Everyone nowadays is building and evaluating LLM workflows. It's easy to get something running, but we don't yet have an engineering discipline for zeroing in on the most accurate and cost-effective system. There are many ways to break a given task into subtasks -- each of which may benefit from evaluation and revision. There are many components to try, including prompts, examples, rewards, LLMs, and external tools -- as well as humans in the loop, human annotators before the loop, and human evaluators after the loop. Many of these components are trainable or configurable and have varying costs. Do we really have to search this space by hand? Or could AI guide our use of annotation and computation?
The problem is more general than NLP. For example, medicine develops workflows for diagnosis and treatment. I'll outline a general approach based on active feature acquisition: every human annotation, LLM output, tool call, or medical test result is a random variable that we could pay to observe. Since random variables are often correlated, observing cheap variables can help us predict more expensive variables and evaluate whether they are worth observing as well. (Which step should we try next, and with what prompt? Should we double-check or revise the result? Should we ask a human?)
Ultimately, gathering information is a reinforcement learning problem. I'll describe how to train an environment model that predicts distributions over any missing variables, much as BERT predicts distributions over any missing tokens. This model could be extended into a policy model that chooses which missing variable to observe next. Any data gathered by the system can immediately be used to train it further, including on counterfactual trajectories, which exploit the fact that our actions have no side effects (observing variables does not change the values of other variables).
Bio: Jason Eisner is Professor of Computer Science at Johns Hopkins University and a Fellow of the Association for Computational Linguistics. At Johns Hopkins, he is also affiliated with the Center for Language and Speech Processing, the Mathematical Institute for Data Science, the Cognitive Science Department, and the Data Science and AI Institute. His goal is to develop the probabilistic modeling, inference, and learning techniques needed for a unified model of all kinds of linguistic structure, and to connect existing models (such as LLMs) to commonsense reasoning, formal reasoning, and downstream applications. His 190+ papers have presented various algorithms for parsing, machine translation, and weighted finite-state machines; formalizations, algorithms, theorems, and empirical results in computational phonology; unsupervised or semi-supervised learning methods for syntax, morphology, and word-sense disambiguation; and principled methods for conversational AI, including neural language modeling and semantic parsing. From 2019-2024 he was Director of Research at Microsoft Semantic Machines, which developed new approaches to conversational AI. He is also the lead designer of Dyna, a declarative programming language that provides an infrastructure for AI algorithms. He has received 3 school-wide awards for excellence in teaching, most recently in 2025, as well as recent Best Paper Awards at ACL 2017, EMNLP 2019, and NAACL 2021 and Outstanding Paper Awards at ACL 2022, EMNLP 2024, and COLM 2025.
Hosted by: Wenpeng Yin, wenpeng@psu.edu
Toward Robust and Efficient Reasoning with LLM Agents
Wednesday, November 18, 2026;
254 Health and Human Development Building
3:35-4:25 p.m.
Speaker: Aram Galstyan from
Abstract: Test-time scaling (TTS) aims to improve the reasoning capabilities of large language models (LLMs) and agents by allocating more compute at inference time. However, TTS introduces its own challenges, such as redundant reasoning, error propagation across multiple steps, and increased inference latency. In this talk, I will present our recent work addressing these challenges. First, I will show how selectively pruning redundant tokens from reasoning trajectories can improve accuracy without extra training. Next, I will introduce a framework for structured communication among parallel agents that detects and resolves factual inconsistencies during exploration while preserving diversity in reasoning trajectories. Finally, I will present an approach that uses idle time during tool execution for adaptive speculative planning, improving agent performance without increasing latency. Together, these results show how more effective use of test-time computation can make LLM agents more robust and efficient.
BIO: Aram Galstyan is a recognized leader in AI and machine learning with more than 25 years of experience and 150+ peer-reviewed publications in leading AI, machine learning, and natural language processing venues. He has a strong track record of leading interdisciplinary research teams, setting scientific strategy, and delivering large-scale, high-impact projects. His current research spans large language models, agentic systems, personalization, human-AI collaboration, and trustworthy AI.
As an Amazon Scholar in Amazon AGI Foundations, Aram provides technical leadership across initiatives in large language model safety, efficiency, and personalization. His work includes shaping research roadmaps, fostering collaboration between research and engineering teams, and developing academic partnerships that accelerate strategic AI initiatives.
At the USC Information Sciences Institute, Aram established and led the Machine Intelligence and Data Science group and directed the institute’s AI Division, its largest research unit. He led and mentored an organization of more than 100 researchers, engineers, faculty, postdoctoral scholars, and Ph.D. students. He also initiated and led major interdisciplinary research programs supported by DARPA, IARPA, NSF, and NIH, contributing to more than $60 million in sponsored research funding.
Hosted by: Lana Fulton, lub18@psu.edu