How Ghost Hunting Changed the Way I Do Science

Imagine you’re an undergrad student in your second year with a lab. You’re finally comfortable with all of the lab techniques, and are confident enough to run your own experiments. You spend months testing a hypothesis, encouraged by positive results, only to hit a brick wall, and realize that the data you thought supported your idea was actually driven by something else, whether an overlooked variable or a well-known process. Almost every researcher I've met experiences this arc: the excitement of discovery followed by the disappointment of realizing that the result was a covariate-induced illusion. I’ve fallen victim to this arc a number of times, and each time it can drain the curiosity and motivation of both myself and my colleagues. But what can we do about it? First, I’d argue that this is a “canon event” in the life of a young scientist, and it's important for growth. But more importantly, we should reframe how we approach our work. Instead of working towards proving our hypotheses, I propose that it can also be beneficial to treat our hypotheses as starting points, and the explore every possible way we could disprove them.

This thought came to me randomly while thinking about how I used to watch ghost hunting shows that I would rent from the local library. For something often considered pseudoscience, I remember those investigators following a surprisingly rigorous pattern. They would start with an observation, a strange noise or a door slamming, and then systematically try to rule out every mundane explanation, testing acoustics, airflow, and environmental conditions before jumping to conclusions. Many experienced scientists would probably say this is basic protocol, but sitting here as a 23 year old MS student, I found it to be a fun way to reshape the way I approach experiments and analysis.

This past weekend I participated in the BU MedAI hackathon, and I ended up using the ghost-hunting method to help make sure my classification algorithm wasn’t being influenced by anything other than the true important signals in the proteomics and clinical data. After a few hours of iteration, my model was finally showing a strong log-loss score on the held-out test subset. But before getting too excited and claiming I’d found a ghost, I spent the next two hours interrogating the result. Was this score a product of overfitting? Could any clinical variables be introducing batch effects? How sensitive was my performance to the ratio of controls versus affected samples? Doing this, I discovered that my log-loss score was overfitting to the training data and was able to fix the issue. Only after ruling out all those possibilities did I feel confident that the model was picking up on something real. I ended up finishing 5th which while a little bit disappointing to miss out on the prize money, I took the experience as a win.

There's an uncomfortable truth underlying all of this. In my experience, science in how it's practiced and rewarded tends to quietly discourage this type of self-skepticism. Whether it's in industry or academia, funding, publications, and prestige tend to flow toward positive results, which creates a subtle pressure to stop interrogating your data the moment it shows you what you want to see. This is even more noticeable in biotech and pharma where deadlines are more pressing and competition is fierce. Despite this, I'd argue that building in deliberate doubt, where you treat every promising result as a ghost until proven otherwise, is one of the safest practices you can do as a scientist. I’d argue that it doesn't have to be elaborate, but before you celebrate a result, spend equal time trying to kill it. Sometimes you have to ask the boring questions, test the mundane explanations, and see what survives. It’s hard to try to disprove something that you’re passionate about and have worked hard for, but findings that make it through that process are the ones worth getting excited about.