>When their accuracy is touted it is usually based on the prediction the day before the election.
That's not true in Nate's case at least. One of his biggest wins was debunking some of the BS "narratives" that pundits tried to spring during the election long before election day, the one about "Romney's momentum" in the final weeks of the election being the most memorable one.
Linzer also did a postmortem where he looked at, among other things, accuracy and predictive power of the model at different points in the election [1].
>Is that really useful?
An order of magnitude moreso than traditional punditry, not least because it's honest about what it can and can't tell you, namely "if the election were held today, this is the most likely outcome". No more, no less.
>More importantly you can get the same predictive power by just directly using recent polling data.
You do realize that's exactly what these guys do, right? The problem is, which polling data do you use? They provide a scientific answer to that question. Instead of cherry picking, they use it all, and weight it based on past accuracy and other factors.
I didn't communicate my point effectively. Let me try again.
I believe strongly in quantitative analysis of election data. I place zero value in punditry, especially from main stream media. What I don't believe is that some of these complicated models, Silver's in particular, are meaningfully superior to predicting the election by trivially applying recent polling data augmenting it perhaps with a simple weighted combination of polls based on recency or sample size.
I take as a given that Silver's model is better than punditry. I am skeptical that it is better than the trivial model which any undergraduate stats student would cook up.
> The problem is, which polling data do you use? They provide a scientific answer to that question. Instead of cherry picking, they use it all, and weight it based on past accuracy and other factors.
What I dispute is whether the "other factors", which is the secret-sauce that allows Silver to give the impression he has a uniquely predictive model, have any real value.
Thanks for the Linzer link. I have to take time to read it carefully but on first glance it again shows one of the things I take issue with: If you are going to claim that a model is accurate you should be asking, compared to what? How can you justify a complex model if you aren't even going to try to show that it is better than some trivial baseline?
> I take as a given that Silver's model is better than punditry. I am skeptical that it is better than the trivial model which any undergraduate stats student would cook up.
This seems quite easy to test. Has anyone done so yet?
I wouldn't say there is a canonical baseline model but your example would be a reasonable place to start. Weighted average by sample size is a straightforward modification. Just using the most recent reliable poll would also be interesting.
I think most of the extra value you could add would be from analyzing the poll data to get a sense of which polls were unreliable.
That's not true in Nate's case at least. One of his biggest wins was debunking some of the BS "narratives" that pundits tried to spring during the election long before election day, the one about "Romney's momentum" in the final weeks of the election being the most memorable one.
Linzer also did a postmortem where he looked at, among other things, accuracy and predictive power of the model at different points in the election [1].
>Is that really useful?
An order of magnitude moreso than traditional punditry, not least because it's honest about what it can and can't tell you, namely "if the election were held today, this is the most likely outcome". No more, no less.
>More importantly you can get the same predictive power by just directly using recent polling data.
You do realize that's exactly what these guys do, right? The problem is, which polling data do you use? They provide a scientific answer to that question. Instead of cherry picking, they use it all, and weight it based on past accuracy and other factors.
[1]: http://votamatic.org/evaluating-the-forecasting-model/