Testing the accuracy of your model’s predictions

In forecasting, a lot depends on the accuracy of the prediction you make. There are multiple ways to test the accuracy of your model. A common way to measure how well a strategy would have performed in the past is called a ‘back-test‘.

Most so-called ‘back-tests’ are engineered by people who have become very good at what they do over a long time. Unfortunately, the better someone becomes at it, the more likely errors such as selection bias and overfitting will come to show in what is called curve fitting. Essentially, in back-tests, experts fit the model to the particular historical curve they are looking at, including random market noise. As soon as the same model is exposed to new market data, it can fail drastically, in part because it failed to correctly identify a pattern. See the graph below for an illustration of this behavior (1).

While human experts try to understand market mechanisms based on knowledge and experience, machines understand markets only in terms of data and patterns. Based on the input data, the algorithms find patterns that have occurred in the past and assume that they will repeat in the future. In this way, machine learning can identify connections that are unknown to the human expert. At the same time, this data-driven approach offers a significant advantage in verifying the accuracy of the algorithm’s predictions. Unlike humans, who cannot voluntarily erase a certain part of their memory, we can keep machines uninformed about a certain part of history.[…]

In forecasting, a lot depends on the accuracy of the prediction you make. There are multiple ways to test the accuracy of your model. A common way to measure how well a strategy would have performed in the past is called a ‘back-test‘.

Share this:

Related