X Hacking and Misguided AutoML | Sumantrak Mukherjee

Summary

This project examines failure modes in AutoML pipelines where optimization pressure can amplify spurious patterns and produce convincing but misguided conclusions.

Research Question

How do automated model-search procedures create opportunities for unintentional hacking of the target metric or scientific claim?

Methodology

Empirical stress-testing of AutoML behavior
Analysis of optimization-induced failure modes
Framing around reliability rather than benchmark-only performance

Key Contributions

Studies the interaction between automated search and methodological misuse
Highlights a systems-level failure mode rather than an isolated algorithmic bug
Connects ML tooling with scientific reliability concerns

Outputs

Publication: (Sharma et al., 2025)

References

2025

adjacent interests

X-Hacking: The Threat of Misguided AutoML

Rahul Sharma, Sumantrak Mukherjee, Andrea Sipka, and 4 more authors

In International Conference on Machine Learning, 2025

Bib Paper

@inproceedings{sharma2025xhacking,
  title = {X-Hacking: The Threat of Misguided AutoML},
  author = {Sharma, Rahul and Mukherjee, Sumantrak and Sipka, Andrea and H{\"u}llermeier, Eyke and Vollmer, Sebastian Josef and Redyuk, Sergey and Selby, David A.},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
}