Adaptive Data Analysis

3 Credits · Level: Graduate, Undergraduate

Data-based inferences can become invalid if the studied hypotheses are selected based on the data itself. Yet, it is common that data is being re-used and hence a potential to overfitting and false discoveries. In this class, we will study mathematical approaches for mitigating false discovery and overfitting in data analysis when data is being reused adaptively, i.e., when hypotheses are selected based on previous finding from the same data. We will study examples where adaptivity leads to invalid inferences, fundamental limitations on ensuring statistical validity in face of adaptivity, and novel mathematically rigorous approaches for this task.