Nash - Scientific Computing with PCs (523165), страница 45
Текст из файла (страница 45)
For the growth curveproblem with any loss function or for linear models using some robust or maximum likelihood lossfunctions, this is often a reasonable formulation. However, it is not an efficient way to solve for theparameters b for linear least squares problems. The growth curve problem is usually formulated as anonlinear least squares problem.Note that we could apply calculus minimization techniques to S(b).
That is, solve the equations that arisewhen we set the derivatives to zero. Here we want to find the root(s) b* of the equations(15.2.9)grad S(b) = 015: DATA FITTING AND MODELLINGwhere131grad is the gradient operator that forms the vector of first derivatives of S(b) with respect to b.Subtle changes in the specifications can radically alter the formulation.
For example, simple constraintson the parameters, such as the requirement that they are positive, may hamper use of traditionaltechniques. We have not here specified any extra conditions on our problems. For instance, we could notethat all the growth data must be positive, and further insist that models be monotone increasing. The firstcondition forces b1 to be positive, the second that b2 and b3 have the same sign. At this stage of the work,we should already have plotted the data to see if it a growth function is appropriate. To save space, werefer the reader to Figure 15.5.2 later in the chapter.15.3 Methods and AlgorithmsThe criteria for fit discussed in Section 15.2, namely, least squares, weighted least squares, maximumlikelihood or robust loss functions, all result in either a minimization or maximization mathematicalproblem.
Therefore, we first seek function minimization methods. (Converting maximization tominimization is accomplished by multiplying the objective by -1.) There are many choices available.First, we may look for nonlinear least squares methods, since our linear problems are special cases. In astatistical context, this may be indexed under the more general topic of nonlinear regression. We may evenfind that growth modelling is explicitly indexed.More generally, function minimization or minimization of a function of several variables is a topic coveredin many works on numerical methods. Often minimization is considered a sub-topic of optimization,particularly unconstrained optimization.
For regular solution of problems of one type, we are likely towant algorithms that are both reliable and efficient. Minimization methods vary widely in theirmotivations and detail, giving rise to a very large literature. Nash J C and Walker-Smith (1987) includesa large bibliography. In general, considering only function minimization tools, we may distinguishmethods as (see Table 7.7.1):•Direct search — not requiring derivative information;•Gradient methods — requiring the user to supply first derivative information; and•Newton-type methods — requiring second derivative information to be supplied.
This category maybe broken down further to include the Gauss-Newton family of methods for loss function that aresums of squared terms, including the Marquardt variants.Usually methods become more efficient as we supply more information, but there can be some nastysurprises. Sadly, the computation of derivatives is non-trivial.
It requires attention to detail and generatesquite lengthy program code. About 90% of the "failures" of minimization codes in our experience resultfrom errors in the derivative code. To avoid such errors, we may turn to symbolic manipulation packages(e.g., Maple, DERIVE) that allow output in the form of program code. Automatic differentiation tools suchas AUTODIF (Fournier, 1991) or ADIFOR (Griewank, 1992) essentially differentiate our program sourcecode for the objective function rather than a mathematical expression.
Another choice is to use numericalapproximation of derivatives, with possible consequences of "poor" approximations.For the linear least squares problems, we can clearly do better. We have already seen some options inSections 12.1 and 12.2 so will not repeat that discussion here.
There is no shortage of methods for ourproblems. The work will be to select among the possibilities.15.4 Programs and PackagesSince the growth curve problem (Table 15.1.1c) is the most general, we will start there. A first softwareoption is statistical packages that include nonlinear regression modules. This is clearly sensible if we are132Copyright © 1984, 1994 J C & M M NashNash Information Services Inc., 1975 Bel Air Drive, Ottawa, ON K2C 0X1 CanadaSCIENTIFIC COMPUTING WITH PCsCopy for:Dr. Dobb’s Journalfamiliar with the package and it gives us the results we want.
Operational details can get in the way.Nonlinear estimation usually requires the user to specify the model, starting values for parameters, andsometimes the loss function to be used. Also needed may be settings for convergence tolerances andnumbers of iterations. Users should understand the actions these control. The advantages of a package— in terms of additional output, graphics, and integration with related computations — are worthwhileif the learning cost is acceptable.Packages may employ inappropriate algorithms; they may fail to give correct answers or may fail toconverge. SYSTAT uses a Gauss-Newton nonlinear least squares method, with derivatives computedautomatically, but this information came from the developer rather than documentation.A second approach is to use programs or systems specific to nonlinear estimation or optimization.Subroutine libraries remain useful for building a special-purpose program when many problems of onetype are to be solved over an extended period.
The user must prepare a driver program as well asroutines to tell the subroutine the nature of the minimization, possibly including derivative information.If good examples exist, the coding task may be straightforward. In our own work (Nash J C 1990d), wehave tried to provide moderately detailed examples, but the practice is by no means universal.Special systems exist for nonlinear estimation (Ross, 1990; Nash J C and Walker-Smith, 1987). Differentproblems benefit from different algorithms.
To simplify bringing together problem specifications andminimizers Nash and Walker-Smith use program builder with a one-line command:NL (name of method) (name of problem)Menu-driven and hypertext variants of this program builder have been created (Nash J C and WalkerSmith, 1989b; Nash J C, 1993b). We provide a variety of examples that can be edited by users so they donot have to prepare problem specification files from scratch. Special systems may have particular featureswe need: the Nash and Walker-Smith codes allow parameters to be made subject to upper and lowerbounds or to be temporarily fixed (masked). A "not computable" flag can be used in the functionspecification to avoid calculating the model or the loss function when parameters have unsuitable values.An "add-on" that is especially useful is graphics. If only very simple graphs can be produced (as inNash J C and Walker-Smith, 1987), an easy way to move data and results to other software forpresentation must be found.
For this book we used Stata to graph the original data and fitted line inFigure 15.5.2.For spreadsheet users, the Lotus 1-2-3 "add-in" What-If Solver (Nash J C, 1991b) could be used, but thespecification of the objective function and data are the responsibility of the user. These are entered intoregular spreadsheet cells using the spreadsheet functions. Traditional datasets are less easily used sincethey must be imported carefully into the appropriate spreadsheet locations. There are similar "solvers" forother spreadsheet packages.A fourth approach is provided by general math/calculation tools. MATLAB is our example here.
This offerstoolkits of procedures, e.g., for optimization problems. A variety of additional calculations or graphs maybe prepared. The problem specification must be "programmed", but even the Student MATLAB includesa Nelder-Mead minimization and we can also program other algorithms. These may use derivativeinformation or be specialized to functions that are sums of squares. To learn about the present growthcurve problem, we computed residuals, their sum of squares, the gradient, the Jacobian matrix (partialderivatives of the residuals) and the Hessian (second derivative matrix of the loss function) and itseigenvalues.For the farm income problem (Table 15.1.1a), we specify as our explanatory variables the two datacolumns labelled phosphate and petrol.
The column income is y. The linear least squares problem iscontained within the linear regression problem of statistics, with a vast array of software available for itssolution. From software already installed on just one of our PCs, we can choose among the generalstatistical packages MINITAB, SYSTAT (and its offspring MYSTAT), and Stata, a special-purpose ridge15: DATA FITTING AND MODELLING133regression package RXRIDGE (Obenchain, 1991), our own algorithms from Nash J C (1990d) and NashJ C and Walker-Smith (1987), and simple expressions in MATLAB.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
