The Elements of Statistical Learning. Data Mining_ Inference_ and Prediction (The Elements of Statistical Learning. Data Mining_ Inference_ and Prediction.pdf)

Trevor Hastie • Robert Tibshirani • Jerome FriedmanThe Elements of Statictical LearningThis major new edition features many topics not covered in the original, including graphicalmodels, random forests, ensemble methods, least angle regression & path algorithms for thelasso, non-negative matrix factorization, and spectral clustering. There is also a chapter onmethods for “wide” data (p bigger than n), including multiple testing and false discovery rates.Trevor Hastie, Robert Tibshirani, and Jerome Friedman are professors of statistics atStanford University.

They are prominent researchers in this area: Hastie and Tibshiranideveloped generalized additive models and wrote a popular book of that title. Hastie codeveloped much of the statistical modeling software and environment in R/S-PLUS andinvented principal curves and surfaces. Tibshirani proposed the lasso and is co-author of thevery successful An Introduction to the Bootstrap.

Friedman is the co-inventor of many datamining tools including CART, MARS, projection pursuit and gradient boosting.S TAT I S T I C S ----› springer.comThe Elements of Statistical LearningDuring the past decade there has been an explosion in computation and information technology. With it have come vast amounts of data in a variety of fields such as medicine, biology, finance, and marketing.

The challenge of understanding these data has led to the development of new tools in the field of statistics, and spawned new areas such as data mining,machine learning, and bioinformatics. Many of these tools have common underpinnings butare often expressed with different terminology. This book describes the important ideas inthese areas in a common conceptual framework. While the approach is statistical, theemphasis is on concepts rather than mathematics. Many examples are given, with a liberaluse of color graphics. It should be a valuable resource for statisticians and anyone interestedin data mining in science or industry.

The book’s coverage is broad, from supervised learning(prediction) to unsupervised learning. The many topics include neural networks, supportvector machines, classification trees and boosting—the first comprehensive treatment of thistopic in any book.Hastie • Tibshirani • FriedmanSpringer Series in StatisticsSpringer Series in StatisticsTrevor HastieRobert TibshiraniJerome FriedmanThe Elements ofStatistical LearningData Mining, Inference, and PredictionSecond EditionThis is page vPrinter: Opaque thisTo our parents:Valerie and Patrick HastieVera and Sami TibshiraniFlorence and Harry Friedmanand to our families:Samantha, Timothy, and LyndaCharlie, Ryan, Julie, and CherylMelanie, Dora, Monika, and IldikoviThis is page viiPrinter: Opaque thisPreface to the Second EditionIn God we trust, all others bring data.–William Edwards Deming (1900-1993)1We have been gratified by the popularity of the first edition of TheElements of Statistical Learning.

This, along with the fast pace of researchin the statistical learning field, motivated us to update our book with asecond edition.We have added four new chapters and updated some of the existingchapters. Because many readers are familiar with the layout of the firstedition, we have tried to change it as little as possible.

Here is a summaryof the main changes:1 On the Web, this quote has been widely attributed to both Deming and Robert W.Hayden; however Professor Hayden told us that he can claim no credit for this quote,and ironically we could find no “data” confirming that Deming actually said this.viiiPreface to the Second EditionChapter1. Introduction2. Overview of Supervised Learning3. Linear Methods for Regression4.

Linear Methods for Classification5. Basis Expansions and Regularization6. Kernel Smoothing Methods7. Model Assessment and Selection8. Model Inference and Averaging9. Additive Models, Trees, andRelated Methods10. Boosting and Additive Trees11. Neural Networks12. Support Vector Machines andFlexible Discriminants13.PrototypeMethodsandNearest-Neighbors14. Unsupervised Learning15.16.17.18.Random ForestsEnsemble LearningUndirected Graphical ModelsHigh-Dimensional ProblemsWhat’s newLAR algorithm and generalizationsof the lassoLasso path for logistic regressionAdditional illustrations of RKHSStrengths and pitfalls of crossvalidationNew example from ecology; somematerial split off to Chapter 16.Bayesian neural nets and the NIPS2003 challengePath algorithm for SVM classifierSpectral clustering, kernel PCA,sparse PCA, non-negative matrixfactorization archetypal analysis,nonlineardimensionreduction,Google page rank algorithm, adirect approach to ICANewNewNewNewSome further notes:• Our first edition was unfriendly to colorblind readers; in particular,we tended to favor red/green contrasts which are particularly troublesome.

We have changed the color palette in this edition to a largeextent, replacing the above with an orange/blue contrast.• We have changed the name of Chapter 6 from “Kernel Methods” to“Kernel Smoothing Methods”, to avoid confusion with the machinelearning kernel method that is discussed in the context of support vector machines (Chapter 11) and more generally in Chapters 5 and 14.• In the first edition, the discussion of error-rate estimation in Chapter 7 was sloppy, as we did not clearly differentiate the notions ofconditional error rates (conditional on the training set) and unconditional rates. We have fixed this in the new edition.Preface to the Second Editionix• Chapters 15 and 16 follow naturally from Chapter 10, and the chapters are probably best read in that order.• In Chapter 17, we have not attempted a comprehensive treatmentof graphical models, and discuss only undirected models and somenew methods for their estimation.

Due to a lack of space, we havespecifically omitted coverage of directed graphical models.• Chapter 18 explores the “p ≫ N ” problem, which is learning in highdimensional feature spaces. These problems arise in many areas, including genomic and proteomic studies, and document classification.We thank the many readers who have found the (too numerous) errors inthe first edition. We apologize for those and have done our best to avoid errors in this new edition. We thank Mark Segal, Bala Rajaratnam, and LarryWasserman for comments on some of the new chapters, and many Stanfordgraduate and post-doctoral students who offered comments, in particularMohammed AlQuraishi, John Boik, Holger Hoefling, Arian Maleki, DonalMcMahon, Saharon Rosset, Babak Shababa, Daniela Witten, Ji Zhu andHui Zou.

We thank John Kimmel for his patience in guiding us through thisnew edition. RT dedicates this edition to the memory of Anna McPhee.Trevor HastieRobert TibshiraniJerome FriedmanStanford, CaliforniaAugust 2008xPreface to the Second EditionThis is page xiPrinter: Opaque thisPreface to the First EditionWe are drowning in information and starving for knowledge.–Rutherford D. RogerThe field of Statistics is constantly challenged by the problems that scienceand industry brings to its door.

In the early days, these problems often camefrom agricultural and industrial experiments and were relatively small inscope. With the advent of computers and the information age, statisticalproblems have exploded both in size and complexity. Challenges in theareas of data storage, organization and searching have led to the new fieldof “data mining”; statistical and computational problems in biology andmedicine have created “bioinformatics.” Vast amounts of data are beinggenerated in many fields, and the statistician’s job is to make sense of itall: to extract important patterns and trends, and understand “what thedata says.” We call this learning from data.The challenges in learning from data have led to a revolution in the statistical sciences.

Since computation plays such a key role, it is not surprisingthat much of this new development has been done by researchers in otherfields such as computer science and engineering.The learning problems that we consider can be roughly categorized aseither supervised or unsupervised. In supervised learning, the goal is to predict the value of an outcome measure based on a number of input measures;in unsupervised learning, there is no outcome measure, and the goal is todescribe the associations and patterns among a set of input measures.xiiPreface to the First EditionThis book is our attempt to bring together many of the important newideas in learning, and explain them in a statistical framework.

While somemathematical details are needed, we emphasize the methods and their conceptual underpinnings rather than their theoretical properties. As a result,we hope that this book will appeal not just to statisticians but also toresearchers and practitioners in a wide variety of fields.Just as we have learned a great deal from researchers outside of the fieldof statistics, our statistical viewpoint may help others to better understanddifferent aspects of learning:There is no true interpretation of anything; interpretation is avehicle in the service of human comprehension.

The value ofinterpretation is in enabling others to fruitfully think about anidea.–Andreas BujaWe would like to acknowledge the contribution of many people to theconception and completion of this book. David Andrews, Leo Breiman,Andreas Buja, John Chambers, Bradley Efron, Geoffrey Hinton, WernerStuetzle, and John Tukey have greatly influenced our careers.

Balasubramanian Narasimhan gave us advice and help on many computationalproblems, and maintained an excellent computing environment. Shin-HoBang helped in the production of a number of the figures. Lee Wilkinsongave valuable tips on color production. Ilana Belitskaya, Eva Cantoni, MayaGupta, Michael Jordan, Shanti Gopatam, Radford Neal, Jorge Picazo, Bogdan Popescu, Olivier Renaud, Saharon Rosset, John Storey, Ji Zhu, MuZhu, two reviewers and many students read parts of the manuscript andoffered helpful suggestions.