c14-1 (Numerical Recipes in C)

610Chapter 14.Statistical Description of DataCITED REFERENCES AND FURTHER READING:Bevington, P.R. 1969, Data Reduction and Error Analysis for the Physical Sciences (New York:McGraw-Hill).Stuart, A., and Ord, J.K. 1987, Kendall’s Advanced Theory of Statistics, 5th ed. (London: Griffinand Co.) [previous eds. published as Kendall, M., and Stuart, A., The Advanced Theoryof Statistics].Norusis, M.J. 1982, SPSS Introductory Guide: Basic Statistics and Operations; and 1985, SPSSX Advanced Statistics Guide (New York: McGraw-Hill).Dunn, O.J., and Clark, V.A.

1974, Applied Statistics: Analysis of Variance and Regression (NewYork: Wiley).14.1 Moments of a Distribution: Mean,Variance, Skewness, and So ForthWhen a set of values has a sufficiently strong central tendency, that is, a tendencyto cluster around some particular value, then it may be useful to characterize theset by a few numbers that are related to its moments, the sums of integer powersof the values.Best known is the mean of the values x1 , . . . , xN ,x=N1 XxjN j=1(14.1.1)which estimates the value around which central clustering occurs. Note the use ofan overbar to denote the mean; angle brackets are an equally common notation, e.g.,hxi. You should be aware that the mean is not the only available estimator of thisSample page from NUMERICAL RECIPES IN C: THE ART OF SCIENTIFIC COMPUTING (ISBN 0-521-43108-5)Copyright (C) 1988-1992 by Cambridge University Press.Programs Copyright (C) 1988-1992 by Numerical Recipes Software.Permission is granted for internet users to make one paper copy for their own personal use.

Further reproduction, or any copying of machinereadable files (including this one) to any servercomputer, is strictly prohibited. To order Numerical Recipes books,diskettes, or CDROMsvisit website http://www.nr.com or call 1-800-872-7423 (North America only),or send email to trade@cup.cam.ac.uk (outside North America).In the other category, model-dependent statistics, we lump the whole subject offitting data to a theory, parameter estimation, least-squares fits, and so on. Thosesubjects are introduced in Chapter 15.Section 14.1 deals with so-called measures of central tendency, the moments ofa distribution, the median and mode. In §14.2 we learn to test whether different datasets are drawn from distributions with different values of these measures of centraltendency.

This leads naturally, in §14.3, to the more general question of whether twodistributions can be shown to be (significantly) different.In §14.4–§14.7, we deal with measures of association for two distributions.We want to determine whether two variables are “correlated” or “dependent” onone another. If they are, we want to characterize the degree of correlation insome simple ways. The distinction between parametric and nonparametric (rank)methods is emphasized.Section 14.8 introduces the concept of data smoothing, and discusses theparticular case of Savitzky-Golay smoothing filters.This chapter draws mathematically on the material on special functions thatwas presented in Chapter 6, especially §6.1–§6.4. You may wish, at this point,to review those sections.14.1 Moments of a Distribution: Mean, Variance, Skewness6111 X(xj − x)2N −1NVar(x1 .

. . xN ) =(14.1.2)j=1or its square root, the standard deviation,σ(x1 . . . xN ) =pVar(x1 . . . xN )(14.1.3)Equation (14.1.2) estimates the mean squared deviation of x from its mean value.There is a long story about why the denominator of (14.1.2) is N − 1 instead ofN . If you have never heard that story, you may consult any good statistics text.Here we will be content to note that the N − 1 should be changed to N if youare ever in the situation of measuring the variance of a distribution whose meanx is known a priori rather than being estimated from the data.

(We might alsocomment that if the difference between N and N − 1 ever matters to you, then youare probably up to no good anyway — e.g., trying to substantiate a questionablehypothesis with marginal data.)As the mean depends on the first moment of the data, so do the variance andstandard deviation depend on the second moment. It is not uncommon, in reallife, to be dealing with a distribution whose second moment does not exist (i.e., isinfinite). In this case, the variance or standard deviation is useless as a measureof the data’s width around its central value: The values obtained from equations(14.1.2) or (14.1.3) will not converge with increased numbers of points, nor showany consistency from data set to data set drawn from the same distribution.

This canoccur even when the width of the peak looks, by eye, perfectly finite. A more robustestimator of the width is the average deviation or mean absolute deviation, defined byADev(x1 . . . xN ) =N1 X|xj − x|N(14.1.4)j=1One often substitutes the sample median xmed for x in equation (14.1.4). For anyfixed sample, the median in fact minimizes the mean absolute deviation.Statisticians have historically sniffed at the use of (14.1.4) instead of (14.1.2),since the absolute value brackets in (14.1.4) are “nonanalytic” and make theoremproving difficult. In recent years, however, the fashion has changed, and the subjectof robust estimation (meaning, estimation for broad distributions with significantnumbers of “outlier” points) has become a popular and important one. Highermoments, or statistics involving higher powers of the input data, are almost alwaysless robust than lower moments or statistics that involve only linear sums or (thelowest moment of all) counting.Sample page from NUMERICAL RECIPES IN C: THE ART OF SCIENTIFIC COMPUTING (ISBN 0-521-43108-5)Copyright (C) 1988-1992 by Cambridge University Press.Programs Copyright (C) 1988-1992 by Numerical Recipes Software.Permission is granted for internet users to make one paper copy for their own personal use.

Further reproduction, or any copying of machinereadable files (including this one) to any servercomputer, is strictly prohibited. To order Numerical Recipes books,diskettes, or CDROMsvisit website http://www.nr.com or call 1-800-872-7423 (North America only),or send email to trade@cup.cam.ac.uk (outside North America).quantity, nor is it necessarily the best one. For values drawn from a probabilitydistribution with very broad “tails,” the mean may converge poorly, or not at all, asthe number of sampled points is increased. Alternative estimators, the median andthe mode, are mentioned at the end of this section.Having characterized a distribution’s central value, one conventionally nextcharacterizes its “width” or “variability” around that value.

Here again, more thanone measure is available. Most common is the variance,612Chapter 14.Statistical Description of DataKurtosisSkewnessnegativepositive(b)Figure 14.1.1. Distributions whose third and fourth moments are significantly different from a normal(Gaussian) distribution. (a) Skewness or third moment. (b) Kurtosis or fourth moment.That being the case, the skewness or third moment, and the kurtosis or fourthmoment should be used with caution or, better yet, not at all.The skewness characterizes the degree of asymmetry of a distribution around itsmean. While the mean, standard deviation, and average deviation are dimensionalquantities, that is, have the same units as the measured quantities xj , the skewnessis conventionally defined in such a way as to make it nondimensional.

It is a purenumber that characterizes only the shape of the distribution. The usual definition isSkew(x1 . . . xN ) =3N 1 X xj − xN j=1σ(14.1.5)where σ = σ(x1 . . . xN ) is the distribution’s standard deviation (14.1.3). A positivevalue of skewness signifies a distribution with an asymmetric tail extending outtowards more positive x; a negative value signifies a distribution whose tail extendsout towards more negative x (see Figure 14.1.1).Of course, any set of N measured values is likely to give a nonzero valuefor (14.1.5), even if the underlying distribution is in fact symmetrical (has zeroskewness). For (14.1.5) to be meaningful, we need to have some idea of itsstandard deviation as an estimator of the skewness of the underlying distribution.Unfortunately, that depends on the shape of the underlying distribution, and rathercritically on its tails! For the idealized case ofpa normal (Gaussian) distribution, thestandard deviation of (14.1.5) is approximately 15/N.