I enjoyed reading this book although I did not feel that it was as new as many think. There are some comments, I think I should warn any potential reader of this book. The first is that there is much about American sports in this book, if you do not know baseball, you will struggle in parts. Although if you keep going, I think you will find it rewarding. Some of his observations about how sportsman rarely did well after getting major publicity are valid for many sports. Furthermore, the writer is concentrating on many fields that at present we are struggling to predict. He actually bypasses many fields that we are rather good at predicting. For example, astronomers struggled to forecast the orbit of planets, today we can predict the path of planets. Another example would be bread sales in a modern city is reasonably predictable as is the sale of train tickets. Another point is that the writer stress Bayesian probability. Some of the problems he notes such as the importance of the initial probability that you use, which can cause many major differences in the result. Another problem that he does not mention is for Bayesian to work you generally need to know a lot about all the options. If your theory on one option is wrong, or you do not know one fact often it will fail. For example, if a coin is biased, and you want to measure how much biased it is, Bayesian can help, but if say there is a magnet, that can be turned on and off that affects the coin, and the statistician does not know about, Bayesian will generally fail. This is a big problem if you are dealing with competitive situations where players can and do change the rules like terrorism. Finally, I think the writer should edit his chapter on terrorism, after rereading his chapter on the stock market. Terrorism I agree with him is a publicity stunt. One event is not independent of another. If terrorists kills a few times ten people in an operations after a while to get into the mass media, a terrorist groups will have to kill twenty people. After they do that for a while, they need to kill thirty and so on. Another point is that as the terrorist gets experienced in killing ten, they are trained up to try something bigger. Israel by dealing forcefully with even small terrorist operations may be despite the writer's doubt be on to something. They definitely have the figures as the writer shows. The same would be true of theft. Most thieves start small, often they have a history of juvenile delinquency and as they get older advance to something bigger. Having said this, I do think that this book is worth reading, I intend to keep my copy to read again sometime.

This book is similar to Steven Levitt's Freakonomics: A Rogue Economist Explores the Hidden Side of Everything (P.S.) , Nassim Taleb's The Black Swan: Second Edition: The Impact of the Highly Improbable: With a new section: "On Robustness and Fragility" , and James Surowiecki's The Wisdom of Crowds . All four books explore the intersection of data, human behavior, and outcomes. They explain how to quantify outcomes within the financial markets, professional sports or elections. This book is especially interesting because Nate Silver has honed firsthand his statistical skills onto numerous domains including professional poker, baseball performance forecasting (he developed one of the best software program to do that), political elections (his "fivethirtyeight" blog). And, when he is not a firsthand practitioner he is a first class investigator. The first seven chapters cover the errors and successes people have had in forecasting in various disciplines. Chapter eight is the most pedagogical, as the author explains the basics of Bayes Theorem that he considers as an overall solution to many of the errors we make in forecasting. The last five chapters focus on Bayesian thinking within various disciplines. Nate Silver's coverage of the credit rating agencies "Catastrophic failure of prediction" (first chapter title) is excellent. In a single sentence on page 13, he captures the cause of the financial crisis: "In advance of the financial crisis, the system was so highly leveraged that a single lax assumption in the credit rating agencies played a huge role in bringing down the whole global financial system." Silver states that the AAA rated CDOs were deemed to have a default rate of only 0.12%. The actual default rate was 28% or over 200 times greater! This was because the rating agencies missed out the correlation between mortgage default rates at different locations when a nationwide home price downturn hit (see figure 1.2 on page 28. Watch out that he mislabeled column 3 and 4 from the right). Silver assesses that overall leverage was too high during the housing bubble. Fannie Mae and Freddie Mac had a debt-to-equity leverage of 70-to-1. Lehman Brothers and other investment banks were leveraged over 30-to-1. Borrowers had often loan-to-value ratios of 100% on their homes. The volume of credit default swaps, MBS, CDOs represented 30 to 60 times the volume of home sales during the bubble years (fig. 1.5 page 35). Nate Silver summarizes the errors made. Investors trusted the rating agencies. The rating agencies assumed home prices would never decline on a nationwide basis because they never had since the Great Depression. Lenders and borrowers believed rising home prices would bail them out through refinancing. Policymakers believed the financial system had enough capital and was self-disciplined. And, economists completely missed the ensuing severe recession. Nate Silver focuses next on political predictions. This field of experts was so bad at predicting it motivated him to enter it by starting his fivethirtyeight blog. He documents their failings extensively. Within this chapter he refers to the theory of Philip Tetlock, professor of psychology and political science at Berkeley. Tetlock had surveyed predictions of experts in various fields. And, he categorized them within two archetypes: the hedgehogs and the foxes. The hedgehogs are dogmatic, rarely change their minds, and are very confident of their forecast. The foxes are just the opposite. They update their forecasts as often as new information warrants it. As a result, they make better forecasts. The chapter on baseball is one of the best because of Silver's extensive firsthand experience. He uncovers many concepts applicable to many sports such as the age-curve of baseball performance (pg. 81). All sports have a predetermined age-curve. Actually, every single aspects of life including life itself have predetermined age-curves. His description of what it takes to be a successful professional baseball player (pg. 97) has also surprisingly broad applications. The conclusion of the chapter is also fascinating. It describes baseball management as a competitive arms race of intelligence gathering to extract small competitive edges. And, that those competitive edges are short-lived. That's a very interesting application of the Efficient Market Hypothesis. The chapter on economists documents how inaccurate their forecasts are. The majority can't forecast a recession that has already started as they missed out on the three most recent ones (1990, 2001, 2007). In November 2007, the average economic forecast was 2.4% real GDP growth in 2008. Instead, real GDP shrank by -3.3%. Economists assigned only a 1-in-2000 chance of the economy shrinking that much. Yet, home prices were already declining. Foreclosures had picked up. Bear Stearns had gone belly up six months ago. Those were powerful signals the housing and financial markets were on the edge of a cliff. Also, economists are way too confident. The few times you can extract confidence intervals from the economic profession they are invariably way too narrow because they underestimate the error level within their forecasts (pg. 182). Nate Silver states that: "this property of overconfident prediction has been observed also in medical research, political science, finance, and psychology" (pg. 183). Despite our having so much more data and computer power at our hands, economic forecasting has not improved since 1968. This is because our underlying understanding of cause and effects has not changed much since. Chapter 8 introduces Bayes's Theorem. Here Nate Silver often refers to a very good book on the subject: The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy by Sharon Bertsch McGrayne. Chapter 9 and 10 about chess and poker are excellent. Kasparov was ultimately beaten by a computer bug. IBM Big Blue made a move late in the last game that did not make any sense (the team who programmed it confirmed it was due to a small programming bug). Kasparov who was in a vulnerable position could not figure out that move and in despair resigned the game and lost the series. The Pareto principle of prediction on page 312 and 314 and the ensuing economics of poker are really interesting. Poker winning are heavily dependent on the one worst player at a table. If he leaves, the winnings are a lot harder to reap. Chapter 11 on the Efficient Market Hypothesis (EMH) is excellent. Nate Silver states that the stock market is efficient most of the time, although it is never perfectly efficient (that would preclude a market). But, it can be wildly inefficient on few occasions associated with bubbles and crashes. Nate Silver demonstrates how both technical analysis and fundamental analysis do not beat the market over the long run. Fig 11.3 on page 340 shows no correlation between the performance of mutual funds over the 2002 to 2006 period vs over the 2007 to 2011 period. Past performance is no guarantee of future returns. Next, Silver refers to Robert Shiller in showing the market is not as efficient as the EMH entails. Shiller looked at the P/E ratio of the S&P 500 over a trailing 10 year period and looked at prospective returns. And, the longer the period contemplated the greater the negative correlation between trailing P/E levels and future average yearly returns. This suggests that the market can get overvalued. But, the return correction is not apparent until looking at average return over a 10 to 20 year period. Next, Nate Silver refers to the works of Richard Thaler and Daniel Kahneman in behavioral economics to outline how market traders are not perfectly rational. They suffer from herd mentality, overconfidence, and being overly emotional rendering their trading pro-cyclical. So, if the market is not so efficient, can you beat it? Probably not. On page 345, Nate Silver demonstrates how a hypothetical investor with perfect timing over a decade (1976-1986) would get killed by very small transaction costs. Even though this investor would handily beat the stock market before transaction costs, he would wipe out most of his capital after transaction costs. Silver next tests a prudent investment strategy over the 1970 to 2009 period. He assumes an investor is prudent and sells his position in the S&P 500 index whenever it had declined 25% from its peak and reinvests whenever it recovered 90% of its value. Such an investor would have earned only 2.6% per year vs close to 10% for a simple buy-and-hold strategy. Nate Silver does believe several hedge funds can beat the market. But, they have intellectual and technological resources that no retail investor and few mutual funds can match. Chapter 12 on climate change is really interesting. He differentiates between where scientists agree and disagree. They all agree that the greenhouse effect exists and keeps the Earth warmer than it would otherwise be; that temperatures have risen over the past century; that greenhouse gases have contributed to that trend; and that water vapor is by far the most potent greenhouse gas (not CO2 as the Media conveys). The majority of scientists agree that rising CO2 concentration does contribute to rising temperature. But, there is a debate regarding how much. Where the scientific community is more divergent is regarding climate models and projections. They acknowledge that Al Gore's An Inconvenient Truth deterministic apocalyptic message was way off base. Nate Silver explains why there is much uncertainty regarding climate models' projections. One uncertainty is figuring out CO2 levels 100 years down the road. Another uncertainty is getting the causal relationships right (there is a lot more than CO2 at play). Another uncertainty concerns whether those models are programmed correctly. Within the vast quantities of computer codes, are there a few bugs that contribute to generating erroneous forecasts? Nate Silver reviews the prediction of the IPCC's 1990 model and observes that temperatures have not risen as fast as the model predicted. Current temperatures are below the model's 95% confidence interval. This lead the IPCC to reduce their baseline temperature increase from 3 degree Celsius per century in 1990 to 1.8 degree in 1995. On page 407, Silver comes up with an interesting application of Bayes theorem applied to rising temperature predictions. The last chapter on terrorism is intriguing. Terrorist attacks follow a similar Power Law as earthquakes. The frequency of events declines exponentially with increase in intensity. More violent events are much rarer than lesser ones. But, the few major events dominate the data in human casualties. For instance, 9/11 represented more than half of the total fatalities from terror attacks in NATO countries since 1979. Thus, it is worth exploring means of mitigating the impact of such events.