Comments on: Fit vs. predictive power of statistical models

By: Bernard

Bernard — Tue, 15 Apr 2008 22:35:56 +0000

It also seem to me that overfitting might seemingly improve relations in the small space under study but might lead to very bad prediction capability under another set of circumstances.

I would tend to think that the simplest model possible in terms of mathematical complexity should also be the most powerful in terms of prediction.

The factors identified in a part of the space might not have the same weight in another space of data.

By: Anthony

Anthony — Thu, 20 Mar 2008 22:52:14 +0000

Brian, I think that your comment points out something really important. The decision between these types of approaches to data is a philosophical matter: it’s the epistemology of statistics that is the hard part. Do we want “underlying mechanism” or “predictive power” and are either of those even achievable? Once I decide what I want to know, then it becomes easier because it’s a mathematical issue of picking an appropriate algorithm. In other words, knowing what the right question is and whether that question is answerable can be harder than answering it. I’d like to see statisticians address this type of thing more often, perhaps with help from the epistemology experts in the philosophy department.

By: Brian Seok

Brian Seok — Wed, 12 Mar 2008 06:12:38 +0000

When I think about models in the context of science, I want them to fit! A model that can simulate observed data well can help us better understand the underlying mechanism of how things work. Isn’t that what we want in science, better understanding of the natural world? Models with predictive power are good for policy and economics. But that doesn’t mean predictive models shouldn’t be tested and developed for scientific purposes, they help sell the science to our potential sponsors and make scientists prophets for the public. Our crystal ball. Bwahahaha~!