ERCIM News 145
July 2026
Special theme E-values: Statistical Testing for the 21st Century
Guest editors Peter Grünwald (CWI and Leiden University), Wouter Koolen (CWI and University of Twente), and Johanna Ziegel (ETH Zurich)
From P-values to E-values
- Details
- Category: Keynote
- Hits: 401
Foreword by Michael I. Jordan
What is an “e-value” and why has it become an object of intense study in statistics and in the allied fields of machine learning, signal processing, and econometrics? To briefly introduce the basic idea, let us consider one of the core problems in statistics – the “hypothesis testing problem” of deciding whether observed data is consistent with some particular data-generating mechanism (often referred to as a “null hypothesis”) or is better explained by another mechanism (referred to as an “alternative hypothesis”). This problem is addressed by defining some function of the data (a “statistic”) whose distribution is as different as possible under the null and the alternative. Given an observed value of such a statistic, one then makes a choice between the two distributions, doing so in a way that minimizes the probability of errors. Classical statistical theory provides a unifying framework – the “p-value” – by which the choice between the null and alternative hypotheses reduces to a thresholding procedure.
E-values: Statistical Testing for the 21st Century - Introduction to the Special Theme
- Details
- Category: Special Theme
- Hits: 554
by the guest editors Peter Grünwald (CWI and Leiden University, Wouter Koolen (CWI and University of Twente) and Johanna Ziegel (ETH Zurich)
As new measurements become available over time, we face the classic problem of updating our information state. In science, this typically means refining our view of hypotheses based on experimental outcomes – either determining if the data allow us to reject a null hypothesis, or estimating which parameter values remain statistically plausible. Anytime-valid methods allow us to reliably refine these assessments sequentially while guaranteeing at most a controlled fraction of mistakes.
On Testing by Imaginary Betting
- Details
- Category: Special Theme
- Hits: 416
by Glenn Shafer
Data analysis requires principles as well as mathematics. Traditionally, we have relied on Cournot’s principle when we use probability theory for data analysis. But when we test by betting instead of relying on small probabilities, we can formulate principles that dig deeper into statistical practice and apply more broadly.
Optimal Gambling Strategies when Testing by Betting
- Details
- Category: Special Theme
- Hits: 1147
by Martin Larsson (Carnegie Mellon University), Aaditya Ramdas (Carnegie Mellon University), and Johannes Ruf (London School of Economics)
Under what conditions do optimal bets against a given probabilistic hypothesis exist? Answer: they always do!
E-values Are "Just" (Some) P-values
- Details
- Category: Special Theme
- Hits: 375
by Eugenio Clerico (University of Oxford)
While e-values and p-values are often presented as competitors, they share deep structural connections. We highlight a functional perspective linking the two, and suggest how it might lead to new ways of translating p-value methods into the e-value framework.
E-values, P-values and Counterfactuals
- Details
- Category: Special Theme
- Hits: 328
by Peter Grünwald (CWI and Leiden University)
A major criticism of p-values and standard confidence intervals, first coined around 1960, is their sensitivity to counterfactuals: their validity depends on how data would have been collected in situations that never occurred, which is often unknown or even unknowable. The fact that e-based methods remain valid under optional continuation implies that they do not suffer from this problem…or does it?
E-processes, Predictive Recursion, and Objective Empirical Probability
- Details
- Category: Special Theme
- Hits: 354
by Vaidehi Dixit (University of Nottingham) and Ryan Martin (North Carolina State University)
Good e-processes can be constructed for testing a specific null hypothesis against a specific alternative, but general inference need not have a specific alternative in mind. In such cases, one might seek an alternative hypothesis-agnostic e-process with fast growth rate under a wide range of alternatives. Our predictive recursion-based e-process construction offers just that, along with some deeper insights related to “objective” empirical probability.
Rethinking Conformal Prediction for Constrained Environments
- Details
- Category: Special Theme
- Hits: 617
by Etienne Gauthier (Inria, Ecole Normale Supérieure, PSL Research University)
How can we trust a model’s predictions in the presence of uncertainty? Conformal prediction provides a principled framework for attaching reliable confidence guarantees to machine learning outputs. By incorporating e-values, this framework moves beyond rigid, pre-specified guarantees. It enables finer control over the predictions in a dynamic way, adapting the reliability of AI systems to the constraints of real-world applications.
More Discoveries and Flexibility in Multiple Testing
- Details
- Category: Special Theme
- Hits: 270
by Rianne de Heide (University of Twente and CWI)
Modern data analysis can test thousands of scientific questions at once, from genes in cancer studies to voxels in brain scans. A new general principle called e-closure gives researchers more freedom to explore these results after seeing the data, while keeping false discoveries under control.
Combining Rejection Sets with False Discovery Control
- Details
- Category: Special Theme
- Hits: 251
by Zhimei Ren (University of Pennsylvania)
How can we combine discoveries from multiple FDR-controlling rejection sets without losing statistical validity? In joint work with Rina Foygel Barber, we show that the knockoff procedure can be represented through e-values and e-BH, allowing rejection sets from multiple randomized runs to be aggregated by averaging e-values while preserving FDR control. More broadly, this e-value perspective provides a general framework for merging FDR-controlling procedures and opens new directions for understanding, improving, and aggregating large-scale testing methods.
E-Values for Stable and Robust Variable Selection in Microbiome Studies
- Details
- Category: Special Theme
- Hits: 218
by Gianna Serafina Monti (University of Milano-Bicocca), and Peter Filzmoser (TU Wien)
E-values provide a principled foundation for false discovery rate control in high-dimensional microbiome data analysis. We show how their aggregation properties enable a derandomized, robust knockoff filter that outperforms classical approaches in stability and reproducibility.
Betting on Bets: Comparing the Upsides of Uncertain Prospects over Time with E-values
- Details
- Category: Special Theme
- Hits: 291
by Yo Joong Choe (INSEAD) and Sebastian Arnold (CWI)
New research develops novel statistical methodology, based on e-values, for monitoring whether one uncertain prospect (say, a new investment option) has an upside over another (say, the current investment). These e-values can flexibly and effectively test for multiple notions of upside over time, as defined by the decision-maker’s preferences, and they come with a direct monetary interpretation that guides the decision (say, whether to invest in the new option).
A Map of Sub-Gaussian E-variables
- Details
- Category: Special Theme
- Hits: 239
by Wouter M. Koolen (CWI & University of Twente), Shubhada Agrawal (IISc Bangalore) and Martin Larsson (CMU)
Could our data be sub-Gaussian noise? We explore rejecting that null hypothesis with the help of e-variables. We map the landscape of optimal e-variables against two-point alternatives.
E-Values for Contingency Tables
- Details
- Category: Special Theme
- Hits: 495
by Thorsten Dickhaus (University of Bremen), Francesca Giuffrida (Leiden University and IMT School for Advanced Studies Lucca) and Yonqqi Wang (CWI)
We present powerful and easy-to-compute e-values for the classical statistical task of testing associations between two binary traits based on contingency table data. Genetic case-control association studies are our main intended use case.
E-values Instead of P-values in Clinical Trials: What Happens?
- Details
- Category: Special Theme
- Hits: 438
by Yongxi Long and Erik van Zwet (Leiden University Medical Centre)
Anytime valid testing with e-values offers great flexibility allowing both optional stopping (peeking) and optional continuation (collecting more data). The price to pay is a reduction of statistical power. Using data from more than 20,000 randomized trials, we evaluate how e-values compare with classical p-values in balancing flexibility and efficiency.
Racing to the Truth: How E-values Can Speed Up Science
- Details
- Category: Special Theme
- Hits: 202
by Sebastian Arias , Alexander Ly, Michele Meziu (CWI) and Angel Reyero Lobo (CWI and Inria)
Modern science generates data continuously, but the statistical methods that still dominate many fields generally require data collection to end before reliable meta-analysis can begin. New research on e-values offers a way to analyse evidence in real time, without sacrificing statistical reliability. The approach could make science not just more robust to modern research practices, but significantly more efficient.
How Small is Negligible? Rethinking Equivalence Testing with E-values
- Details
- Category: Special Theme
- Hits: 202
by Stan Koobs and Nick W. Koning (Erasmus University Rotterdam)
A classical statistical problem is to assess whether an unknown quantity is negligible: that is, practically equivalent to zero. It is standard to define ‘negligible’ as being smaller in magnitude than some threshold margin. Specifying this margin has plagued statisticians for decades: if it is set too large, then one can hardly speak of negligibility, but if the margin is set too small, then one may need an enormous amount of data to statistically establish negligibility. In recent work, we study this problem in depth and show how e-values can be used to bypass it, by enabling one to select the margin post-hoc: after seeing the data.
Safe Preference-Based Political Polls
- Details
- Category: Special Theme
- Hits: 207
by Adrienne Tuynman and Timothée Mathieu (Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 – CRIStAL)
Political polls before elections are useful to identify promising candidates, and to allow parties to make compromises or build alliances. We are interested in conducting polls sequentially, so that one can stop acquiring data as soon as possible while safely yielding statistically significant results.
Assessing Generative AI Systems Using E-scores
- Details
- Category: Special Theme
- Hits: 242
by Guneet Singh Dhillon (University of Oxford), Teodora Pandeva (Microsoft Research), and Alicia Curth (Microsoft Research)
Generative AI systems are becoming ubiquitous, but their outputs can still be inaccurate or misleading. Using e-values, the e-scores framework provides a statistically rigorous assessment of AI-generated responses while accommodating the adaptive and post-hoc nature of human-AI interactions.
Winning by Watching: Evaluating Policies Safely for Reinforcement Learning
- Details
- Category: Special Theme
- Hits: 783
by Stephan Bongers (CWI)
Standard statistical guarantees fail when analysts repeatedly check incoming data. By integrating anytime-valid inference with reinforcement learning, this work enables safe policy evaluation under continuous monitoring.
Scrutinizing Banks’ Risk Forecasts with E-backtesting
- Details
- Category: Special Theme
- Hits: 343
by Ruodu Wang (University of Waterloo)
A model-free method lets regulators and financial institutions continuously monitor tail-risk forecasts using e-values that remain valid whenever checked.
Anytime-Valid Testing in the Age of AI-Assisted Software Development
- Details
- Category: Special Theme
- Hits: 947
by Michael Scott Lindon (Netflix)
The statistical guarantees designed to protect against human failures in sequential experimentation turn out to be exactly what is needed to govern autonomous AI agents conducting experiments.
Large Language Models as Design Partners: Automating Graphical Mockups to Refine Requirements
- Details
- Category: Research and Innovation
- Hits: 203
by Giovanna Broccia, Maurice H. ter Beek (CNR–ISTI), and Alessio Ferrari (University College Dublin and CNR–ISTI)
Can large language models help designers move faster without sacrificing human centrality? Researchers from CNR–ISTI and University College Dublin are exploring how large language models can support the rapid creation and refinement of industrial graphical user interfaces through a case study involving the Italian railway operator Trenord, helping development teams move more quickly from textual requirements to interactive mockups, while keeping humans at the centre of the design process.
Challenges in Small-Scale Medical Data Exchange Platform Development
- Details
- Category: Research and Innovation
- Hits: 207
by Hubert Schölnast, Peter Kieseberg, Patrick Kochberger and Henri Ruotsalainen (University of Applied Sciences St. Pölten)
While the concepts of data sharing and data reuse are simple in theory, they face a plethora of challenges and obstacles when transferred into real-life applications. In this article we discuss the major challenges encountered in the successful construction of a sharing infrastructure for oncological data, as well as best practices and learnings in order to overcome similar issues.
Building Sovereign AI for Europe’s Energy Networks
- Details
- Category: Research and Innovation
- Hits: 537
by Antonello Monti (Fraunhofer Institute for Applied Information Technology FIT, Germany)
European electricity networks are becoming increasingly complex as renewable energy sources, electrification and cross-border interconnections continue to grow. The AI.Grids initiative brings together 48 European organizations to develop open, trustworthy and sovereign AI models and data foundations tailored to the needs of Europe’s critical energy infrastructure.
SCIANCE: AI for Scientific Discovery in Europe
- Details
- Category: Research and Innovation
- Hits: 467
by András Benczúr, Edina Nemeth (SZTAKI), Jonas L'Haridon (European Science Foundation) and Magdalena Brus (EGI Foundation)
Artificial Intelligence (AI) is changing how scientific research is conceived, executed and interpreted, from analysing massive astrophysical data streams to accelerating drug discovery and improving climate and environmental modelling. Yet, the European landscape of AI enabled research remains fragmented: scientific communities, AI experts and research infrastructures often work in parallel rather than together, and strategic guidance on where to invest and how to coordinate efforts is still emerging. The SCIANCE project was launched to address this fragmentation and to help Europe turn AI into a coherent, shared engine for scientific discovery.
Spring ERCIM Meetings 2026 Held in Vienna
- Details
- Category: Joint ERCIM Actions
- Hits: 382
Event report
ERCIM warmly thanks SBA Research for hosting the ERCIM Days on 20–21 May 2026 at its offices in Vienna. Representatives from leading European research institutions gathered for two days of strategic discussions, collaborative planning, and networking focused on the future of informatics and mathematics research in Europe.
Workshop Report: 14th International Workshop on Computational Intelligence for Multimedia Understanding
- Details
- Category: Joint ERCIM Actions
- Hits: 412
by Behçet Uğur Töreyin (İTÜ), Maria Trocan (ISEP) and Davide Moroni (CNR-ISTI)
The 14th International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM 2026), organised by the ERCIM Working Group Multimedia Understanding through Semantics, Computation and Learning (MUSCLE), was held as a special session of IEEE ISCAS 2026 in Shanghai, China, on 26 May 2026.
Call for Participation: FMICS 2026: 31st International Conference on Formal Methods for Industrial Critical Systems
- Details
- Category: Announcements
- Hits: 282
Liverpool, UK, 2-4 September 2026
FMICS is the annual conference of the ERCIM Working Group on Formal Methods for Industrial Critical Systems and the leading forum at the intersection of formal methods research and industrial applications. The conference aims to bring together researchers, practitioners and tool developers interested in the development and deployment of formal methods for safety-critical systems. ERCIM is pleased to sponsor this year's edition.
Call for Participation: MODELS 2026 in Málaga: Advancing Model-Driven Engineering in the Age of AI
- Details
- Category: Announcements
- Hits: 322
Malaga, Spain, 4-9 October 2026,
The ACM/IEEE 29th International Conference on Model Driven Engineering Languages and Systems (MODELS 2026) will take place in Málaga, Spain, bringing together researchers, practitioners and industry leaders working on model driven engineering, modelling languages, and model-based software and systems engineering. Since 1998, the MODELS conference series has been the leading international forum for advances in modelling methods, languages, tools and applications.
Dagstuhl Seminars and Perspectives Workshops
- Details
- Category: Announcements
- Hits: 255
Call for Proposal
Schloss Dagstuhl – Leibniz-Zentrum für Informatik is accepting proposals for scientific seminars/workshops in all areas of computer science, in particular also in connection with other fields.
ERCIM Forum Beyond Compliance on Digital Ethics in Research
- Details
- Category: Announcements
- Hits: 407
Call for Participation
The 5th edition of the ERCIM Forum Beyond Compliance on Research Ethics in the Digital Age will take place on 29–30 October 2026 in Porto, Portugal, hosted by INESC TEC.
Training in Digital Ethics
- Details
- Category: Announcements
- Hits: 222
Call for Participation
Registrations are now open for a one-day Training in Digital Ethics, organised within the Horizon Europe AIOLIA project (Artificial Intelligence in Human Cognition and Behaviour), in which ERCIM is a project partner. The training is delivered by INESC TEC, an ERCIM member institute, and explores the ethical challenges arising from AI systems that shape human decision-making and interaction.
ERCIM Working Group “Inclusive Digital Futures” Held Kick-off Meeting
- Details
- Category: Announcements
- Hits: 191
Event report
The kick-off meeting of the ERCIM Working Group “Inclusive Digital Futures: Developing Technology and Culture” took place online on 18 May 2026. The meeting introduced the Working Group’s vision and scope, which focuses on moving beyond isolated institutional initiatives towards a more systematic approach within the Computer Science community to foster inclusion and diversity in both the digital sphere and the broader research and innovation ecosystem.