Finding a signal in the haystack – Confirmatory versus exploratory research

Chartering into the unknown to find something new can be driven by playfully experimenting with whatever is at hand or trying to prove a specific hypothesis derived from prior knowledge.

Copyright: Sabrina Herold and Artem Chakirov at TomCapital.ch – “Finding a signal in the haystack” – Confirmatory versus exploratory research

With traditional science, we commonly think about the latter rather than the former. This article sheds light on both approaches’ usefulness and examines what machine learning technology can contribute.

Confirmatory versus exploratory research

The confirmatory method, also known as hypothesis testing, is driven by five stages: formulating a clear hypothesis or question, designing a study, collecting data, analyzing data, and drawing a conclusion. It is a common approach on which most of today’s empirical finance is built. To us, it is comparable to putting a specific combination of data series (features) on trial and proving with new data that this combination will be able to extract the desired alpha.

On the other hand, the exploratory method emphasizes the “search” part of the word research. It assumes a broader perspective by stepping away from strong assumptions and focusing on experimentation with the data at hand to understand patterns better and to find the question worth asking. Here, instead of having a specific feature combination under examination, the entire data environment is explored to understand patterns of interaction, discover anomalies across features, and generate a hypothesis from the data. For Tukey (1980), exploration involves three components: attitude, flexibility, and a graphical illustration. He may suggest that openness, curiosity, and bravery, coupled with transparency, are relevant prerequisites for a good (data) explorer.

In an output-driven and expert-knowledge-focused society, fast and positive results based on expert intuition are rewarded, and false positives should be avoided at all costs. The confirmatory method, which is by nature designed to confirm the expected, suits these needs well. Will it solve tomorrow’s problems and create the breakthrough innovations we need? Or is more extensive exploration necessary to unsee the obvious and look at the relevant? For us, the potential this question holds to reshape our understanding of the financial market in a scientific way is undoubtedly attractive.

Data favors the explorers or: how new technology makes exploratory approaches more accessible

Today, data can tell us many stories that haven’t been told before. As a relatively nascent field that started to emerge with the convergence of statistics and personal computers in the 1970s, data science as we know it today continues to experience a surge of available data and tools to process them effectively and efficiently. The days of financial data being a scarce treasure and the possibilities of data exploration being limited are over.

Combining big data with the speed of analytical tools, such as machine learning, continues to open opportunities for financial innovation. We argue that this shift from few data points to an ever-increasing data abundance makes the exploratory method necessary to innovate today. In addition, applying machine learning to big data allows forecasting models to accommodate much more complex relationships within data, which we humans cannot derive ourselves.[…]