The New Republic discusses prediction markets, the Condorcet Jury Theorem, the Columbia space shuttle disaster, fat oxen, and much besides.
Posted by Alan Allport at June 30, 2004 01:58 PM"Oxen," no?
Sorry, haven't replied to this because not feeling sufficiently erudite. Question mainly being how you can tell when bettors (or investors) are in one of these believing-their-own-hype loops. I didn't read closely enough to see if they found any infallible ways of telling when that's happening (other than that, e.g., the law of gravity does apply to the NASDAQ...). Did they find any?
Posted by: Martha Bridegam at July 2, 2004 10:07 AMOxen corrected. The point is, I think, that with a crowd - so long as it's constructed of a fairly diverse group of people who come to their decisions independent of one another - you tend to have the hype element marginalized. It's not that the individual people within the crowd are thoroughly objective - they don't have to be; it's that as the crowd gets larger the element of self-deludedness cancels itself out, and you get a collective decision that is wiser than the sum of all the individual decisions. That's the theory anyway. I think Cass Sunstein makes some sensible remarks about it; it's an important way of thinking about problems and it can be a useful aid to certain types of decision-making, but it's not the answer to everything.
Posted by: Alan Allport at July 2, 2004 10:48 AMMy first urge on reading this article was to go over it point-by-point, or at least pick a large number of what should be fairly obvious holes in it, but...
I haven't time right now. Let me just point out one strange statement in this article that I think shows that the prediction market is one of those concepts that pop up once in a while that make smart people go wooley between the ears.
Asking two hundred students to rank items by weight, one experimenter found that the group's estimate was 94 percent accurate--a figure excelled by only five individuals in that group.
Excelled by only five individuals? So, in other words, the experiment suggests exactly the opposite of what the author claims: that certain individuals (call them experts, if you want) make better estimations than groups.
The first major piece of empirical evidence for the efficacy of prediction markets is the success of the Iowa Electionic Markets in predicting the outcome of presidential elections:
As a predictor, the Iowa Electronic Markets have produced extraordinarily accurate judgments, often doing better than professional polling organizations.
It should be obvious that this is weak evidence for the use of markets to make wider policy predictions. The people participating in these markets are the people upon whom the outcome of the election rests, i.e., this is a circular exercise. The Iowa Electronic Markets are not predicting, they are measuring, and therefore are useful perhaps as an alternate polling model, but nothing more.*
In general, it bothers me that no one talking about this concept ever seems to make any distinctions between prediction, measurement and estimation, which, it seems to me, are all very different things. If the quality of this review accurately reflects the book, I'm going to have to pass.
Why do certain ideas seem to make people into sloppy thinkers? Seems to me it always has something to do with self interest, as in the internet boom of the 90's when so many convinced themselves that economies no longer worked the way we knew they always had.
Could it be that many people are just too thrilled to have their faith in markets confirmed on a mysterious, metaphysical level, as if by God?
---
* Furthermore, I would suggest that the improved results may be more a result of better de facto sampling than any mysterious market wisdom. Exactly why this would be is unclear to me, but it seems at least as likely an explanation.
Excelled by only five individuals? So, in other words, the experiment suggests exactly the opposite of what the author claims: that certain individuals (call them experts, if you want) make better estimations than groups.
No, I don't think that's quite right. What I think they're getting at is that, while a handful of people might get a better one-off individual result through blind chance, over time the group will always do better than any one of them. In the particular example cited, I don't think the five "experts" did well because they had special knowledge - they did well because they were lucky. There's no reason to think they'd be similarly lucky on another occasion (others might, but there's no way to predict in advance which "experts" will come up trumps), whereas the group would still produce a pretty good answer no matter how many times you repeated the experiment.
As I said above, this isn't the catch-all answer to all of life's problems. But it may have its place.
Posted by: Alan Allport at July 2, 2004 11:13 AMWell, I don't find it the least unlikely that some people in that group were, in fact, good at estimating the weight of objects. I know a carpenter who can eyeball something and estimate its dimensions much more accurately than I ever could. I realize this is just one minor point, but I think it is an example of a kind of wishful thinking going on which Sunstein does not seem to have caught.
Likewise, I should say I'm not denying that something real is being talked about here, but I do think that people are missing some crucial distinctions. Looking back over the article, I'd like to retract somewhat my criticism of Sunstein, though. His discussion of the Condorcet theorem strikes me as unusually level-headed for this topic, and his criticisms seem useful.
Posted by: Alan Hogue at July 2, 2004 11:27 AMGoing over it again, this passage caught my eye:
In the Iowa Electronic Markets, it turns out that 85 percent of the traders aren't so smart. They hold onto their shares for a long period and then just accept someone else's prices. The market's predictions appear to be driven by the other 15 percent--frequent traders who post their offers rather than accepting those made by other people.
This is interesting.
If you have a market running like the one described above, where it appears that a small percentage of "smart" traders are somehow molding the outcome, then perhaps all you've done is find a reliable mechanism for identifying those most fit to answer a question or make a prediction. Just a thought.
Posted by: Alan Hogue at July 2, 2004 11:45 AM