Correlation analysis provides a single summary statistic – the correlation coefficient – describing the strength of the association between two variables. The most common types of association which are investigated by correlation analysis are linear relationships, and there are a number of forms of linear correlation coefficients for use with different types of data.

The (Pearson) product-moment correlation coefficients measure a linear relationship, while Kendall's tau and Spearman's rank order correlation coefficients measure monotonicity only. All three coefficients range from -1.0 to +1.0. A coefficient of zero always indicates that no linear relationship exists; a +1.0 coefficient implies a ‘perfect’ positive relationship (i.e., an increase in one variable is always associated with a corresponding increase in the other variable); and a coefficient of -1.0 indicates a ‘perfect’ negative relationship (i.e., an increase in one variable is always associated with a corresponding decrease in the other variable).

The product-moment correlation coefficient can be greatly affected by the presence of a few extreme observations or outliers. There are robust estimation procedures which aim to decrease the effect of extreme values.

Mathematically these methods can be described as follows. A robust estimate of the variance-covariance matrix, C, can be written as where τ2 is a correction factor to give an unbiased estimator if the data is Normal and A is a lower triangular matrix. Let xi be the vector of values for the ith observation and let zi=Axi-θ, θ being a robust estimate of location, then θ and A are found as solutions to and where wt, ut and vt are functions such that they return a value of 1 for reasonable values of t and decreasing values for large t. The correlation matrix can then be calculated from the variance-covariance matrix. If w, u, and v returned 1 for all values then the product-moment correlation coefficient would be calculated.

When there are missing values in the data these may be handled in one of two ways. Firstly, if a case contains a missing observation for any variable, then that case is omitted in its entirety from all calculations; this may be termed casewise treatment of missing data. Secondly, if a case contains a missing observation for any variable, then the case is omitted from only those calculations involving the variable for which the value is missing; this may be called pairwise treatment of missing data. Pairwise deletion of missing data has the advantage of using as much of the data as possible in the computation of each coefficient. In extreme circumstances, however, it can have the disadvantage of producing coefficients which are based on a different number of cases, and even on different selections of cases or samples; furthermore, the ‘correlation’ matrices formed in this way need not necessarily be positive-semidefinite, a requirement for a correlation matrix. Casewise deletion of missing data generally causes fewer cases to be used in the calculation of the coefficients than does pairwise deletion. How great this difference is will obviously depend on the distribution of the missing data, both among cases and among variables.

Pairwise treatment does therefore use more information from the sample, but should not be used without careful consideration of the location of the missing observations in the data matrix, and the consequent effect of processing the missing data in that fashion.

Consider a matrix with elements given by the product-moment correlation of pairs of variables, with any missing values treated in the pairwise sense. Such a matrix may not be positive semi-definite, and therefore not a valid correlation matrix. However, a valid correlation matrix can be calculated that is in some sense ‘close’ to the original. One measure of closeness is the Frobenius norm. This valid correlation matrix is the solution to the nearest correlation matrix problem.

In regression analysis the relationship between one specific random variable, the dependent or response variable, and one or more known variables, called the independent variables or covariates, is studied. This relationship is represented by a mathematical model, or an equation, which associates the dependent variable with the independent variables, together with a set of relevant assumptions. The independent variables are related to the dependent variable by a function, called the regression function, which involves a set of unknown parameters. Values of the parameters which give the best fit for a given set of data are obtained; these values are known as the estimates of the parameters.

The reasons for using a regression model are twofold. The first is to obtain a description of the relationship between the variables as an indicator of possible causality. The second reason is to predict the value of the dependent variable from a set of values of the independent variables. Accordingly, the most usual statistical problems involved in regression analysis are:

(i)	to obtain best estimates of the unknown regression parameters;
(ii)	to test hypotheses about these parameters;
(iii)	to determine the adequacy of the assumed model; and
(iv)	to verify the set of relevant assumptions.

When the regression model is linear in the parameters (but not necessarily in the independent variables), then the regression model is said to be linear; otherwise the model is classified as nonlinear.

One application of regression models is in the analysis of experiments. In this case the model relates the dependent variable to qualitative independent variables known as factors. Factors may take a number of different values known as levels. For example, in an experiment in which one of four different treatments is applied, the model will have one factor with four levels. Each level of the factor can be represented by a dummy variable taking the values 0 or 1. So in the example there are four dummy variables xj, for j=1,2,3,4 such that: along with a variable for the mean x0: If there were 7 observations the data would be: Models which include factors are sometimes known as General Linear (Regression) Models. When dummy variables are used it is common for the model not to be of full rank. In the case above, the model would not be of full rank because This means that the effect of x4 cannot be distinguished from the combined effect of x0,x1,x2 and x3. This is known as aliasing. In this situation, the aliasing can be deduced from the experimental design and as a result the model to be fitted; in such situations it is known as intrinsic aliasing. In the example above no matter how many times each treatment is replicated (other than 0) the aliasing will still be present. If the aliasing is due to a particular dataset to which the model is to be fitted then it is known as extrinsic aliasing. If in the example above observation 1 was missing then the x1 term would also be aliased. In general intrinsic aliasing may be overcome by changing the model, e.g., remove x0 or x1 from the model, or by introducing constraints on the parameters, e.g., β1+β2+β3+β4=0.

If aliasing is present then there will no longer be a unique set of least-squares estimates for the parameters of the model but the fitted values will still have a unique estimate. Some linear functions of the parameters will also have unique estimates; these are known as estimable functions. In the example given above the functions (β0+β1) and (β2-β3) are both estimable.

In many situations there are several possible independent variables, not all of which may be needed in the model. In order to select a suitable set of independent variables, two basic approaches can be used.

(a)

All possible regressions In this case all the possible combinations of independent variables are fitted and the one considered the best selected. To choose the best, two conflicting criteria have to be balanced. One is the fit of the model as measured by the residual sum of squares. This will decrease as more variables are added to the model. The second criterion is the desire to have a model with a small number of significant terms. To aid in the choice of model, statistics such as R2, which gives the proportion of variation explained by the model, and Cp, which tries to balance the size of the residual sum of squares against the number of terms in the model, can be used.

(b)

Stepwise model building In stepwise model building the regression model is constructed recursively, adding or deleting the independent variables one at a time. When the model is built up the procedure is known as forward selection. The first step is to choose the single variable which is the best predictor. The second independent variable to be added to the regression equation is that which provides the best fit in conjunction with the first variable. Further variables are then added in this recursive fashion, adding at each step the optimum variable, given the other variables already in the equation. Alternatively, backward elimination can be used. This is when all variables are added and then the variables dropped one at a time, the variable dropped being the one which has the least effect on the fit of the model at that stage. There are also hybrid techniques which combine forward selection with backward elimination.

Having fitted a model two questions need to be asked: first, ‘are all the terms in the model needed?’ and second, ‘is there some systematic lack of fit?’. To answer the first question either confidence intervals can be computed for the parameters or t-tests can be calculated to test hypotheses about the regression parameters – for example, whether the value of the parameter, βk, is significantly different from a specified value, bk (often zero). If the estimate of βk is β^k and its standard error is seβ^k then the t-statistic is It should be noted that both the tests and the confidence intervals may not be independent. Alternatively F-tests based on the residual sums of squares for different models can also be used to test the significance of terms in the model. If model 1, giving residual sum of squares RSS1 with degrees of freedom ν1, is a sub-model of model 2, giving residual sum of squares RSS2 with degrees of freedom ν2, i.e., all terms in model 1 are also in model 2, then to test if the extra terms in model 2 are needed the F-statistic may be used. These tests and confidence intervals require the additional assumption that the errors, ei, are Normally distributed.

To check for systematic lack of fit the residuals, ri=yi-y^i, where y^i is the fitted value, should be examined. If the model is correct then they should be random with no discernible pattern. Due to the way they are calculated the residuals do not have constant variance. Now the vector of fitted values can be written as a linear combination of the vector of observations of the dependent variable, y, y^=Hy. The variance-covariance matrix of the residuals is then I-Hσ2, I being the identity matrix. The diagonal elements of H, hii, can therefore be used to standardize the residuals. The hii are a measure of the effect of the ith observation on the fitted model and are sometimes known as leverages.

If the observations were taken serially the residuals may also be used to test the assumption of the independence of the ei and hence the independence of the observations.

Let X be the n by p matrix of independent variables and y be the vector of values for the dependent variable. To find the least-squares estimates of the vector of parameters, β^, the QR decomposition of X is found, i.e., where R*= R 0 , R being a p by p upper triangular matrix, and Q an n by n orthogonal matrix. If R is of full rank then β^ is the solution to where c=QTy and c1 is the first p rows of c. If R is not of full rank, a solution is obtained by means of a singular value decomposition (SVD) of R, where D is a k by k diagonal matrix with nonzero diagonal elements, k being the rank of R, and Q* and P are p by p orthogonal matrices. This gives the solution

P1 being the first k columns of P and Q*1 being the first k columns of Q*.

This will be only one of the possible solutions. Other estimates may be obtained by applying constraints to the parameters. If weighted regression with a vector of weights w is required then both X and y are premultiplied by w1/2.

The method described above will, in general, be more accurate than methods based on forming (XTX), (or a scaled version), and then solving the equations

Generalized linear models are an extension of the general linear regression model discussed above. They allow a wide range of models to be fitted. These included certain nonlinear regression models, logistic and probit regression models for binary data, and log-linear models for contingency tables. A generalized linear model consists of three basic components:

(a)	A suitable distribution for the dependent variable Y. The following distributions are common: (i) Normal (ii) binomial (iii) Poisson (iv) gamma In addition to the obvious uses of models with these distributions it should be noted that the Poisson distribution can be used in the analysis of contingency tables while the gamma distribution can be used to model variance components. The effect of the choice of the distribution is to define the relationship between the expected value of Y, EY=μ, and its variance and so a generalized linear model with one of the above distributions may be used in a wider context when that relationship holds.
(b)	A linear model η=∑βjxj, η is known as a linear predictor.
(c)	A link function g· between the expected value of Y and the linear predictor, gμ=η. The following link functions are available: For the binomial distribution ε, observing y out of t: (i) logistic link: η=logμt-μ ; (ii) probit link: η=Φ-1 μt ; (iii) complementary log-log: η=log-log1-μt . For the Normal, Poisson, and gamma distributions: (i) exponent link: η=μa, for a constant a; (ii) identity link: η=μ; (iii) log link: η=log⁡μ; (iv) square root link: η=μ; (v) reciprocal link: η=1μ . For each distribution there is a canonical link. For the canonical link there exist sufficient statistics for the parameters. The canonical links are: (i) Normal – identity; (ii) binomial – logistic; (iii) Poisson – logarithmic; (iv) gamma – reciprocal. For the general linear regression model described above the three components are: (i) Distribution – Normal; (ii) Linear model – ∑βjxj; (iii) Link – identity.

The model is fitted by maximum likelihood; this is equivalent to least-squares in the case of the Normal distribution. The residual sums of squares used in regression models is generalized to the concept of deviance. The deviance is the logarithm of the ratio of the likelihood of the model to the full model in which μ^i=yi, where μ^i is the estimated value of μi. For the Normal distribution the deviance is the residual sum of squares. Except for the case of the Normal distribution with the identity link, the χ2 and F-tests based on the deviance are only approximate; also the estimates of the parameters will only be approximately Normally distributed. Thus only approximate z- or t-tests may be performed on the parameter values and approximate confidence intervals computed.

The estimates are found by using an iterative weighted least-squares procedure. This is equivalent to the Fisher scoring method in which the Hessian matrix used in the Newton–Raphson method is replaced by its expected value. In the case of canonical links the Fisher scoring method and the Newton–Raphson method are identical. Starting values for the iterative procedure are obtained by replacing the μi by yi in the appropriate equations.

In a standard linear model the independent (or explanatory) variables are assumed to take the same set of values for all units in the population of interest. This type of variable is called fixed. In contrast, an independent variable that fluctuates over the different units is said to be random. Modelling a variable as fixed allows conclusions to be drawn only about the particular set of values observed. Modelling a variable as random allows the results to be generalised to the different levels that may have been observed. In general, if the effects of the levels of a variable are thought of as being drawn from a probability distribution of such effects then the variable is random. If the levels are not a sample of possible levels then the variable is fixed. In practice many qualitative variables can be considered as having fixed effects and most blocking, sampling design, control and repeated measures as having random effects.

In a general linear regression model, defined by

where	y is a vector of n observations on the dependent variable,
	X is an n by p design matrix of independent variables,
	β is a vector of p unknown parameters,
and	ε is a vector of n, independent and identically distributed, unknown errors, with ε ~ N 0 , σ 2 .

there are p fixed effects (the β) and a single random effect (the error term ε).

An extension to the general linear regression model that allows for additional random effects is the linear mixed effects regression model, (sometimes called the variance components model). One parameterisation of a linear mixed effects model is

where	y is a vector of n observations on the dependent variable,
	X is an n by p design matrix of fixed independent variables,
	β is a vector of p unknown fixed effects,
	Z is an n by q design matrix of random independent variables,
	ν is a vector of length q of unknown random effects,
	ε is a vector of length n of unknown random errors,

and ν and ε are normally distributed with expectation zero and variance / covariance matrix defined by

The methods currently available in this chapter are restricted to cases where R= σ R 2 I , I is the n×n identity matrix and G is a diagonal matrix. Given this restriction the random variables, Z, can be subdivided into g≤q groups containing one or variables. The variables in the ith group are identically distributed with expectation zero and variance σi2. The model therefore contains three sets of unknowns, the fixed effects, β, the random effects, ν, and a vector of g+1 variance components, γ, with γ = σ12,σ22,…,,, σ g-1 2 ,σg2,σR2 . Rather than work directly with γ and the full likelihood function, γ is replaced by γ* = σ12 / σR2 , σ22 / σR2 ,…, σg-12 / σR2 , σg2 / σR2 ,1  and the profiled likelihood function is used instead.

The model parameters are estimated using an iterative method based on maximizing either the restricted (profiled) likelihood function or the (profiled) likelihood functions. Fitting the model via restricted maximum likelihood involves maximizing the function

Whereas fitting the model via maximum likelihood involves maximizing

In both cases

Once the final estimates for γ *  have been obtained, the value of σR2  is given by

Case weights, Wc , can be incorporated into the model by replacing X′X  and Z′Z  with X′WcX  and Z′WcZ  respectively, for a diagonal weight matrix Wc .

When data on predictor variables x are multicollinear, ridge regression models provide an alternative to variable selection in the multiple regression model. In the ridge regression case, parameter estimates in the linear model are found by penalised least squares: where the value of the ridge parameter h controls the trade-off between the goodness-of-fit and smoothness of a solution.

Regression by means of projections to latent structures PLS, also known as partial least squares, is a latent variable linear model suited to data for which:

• the number of x-variables is high compared to the number of observations;
• x-variables and/or y-variables are multicollinear.

Latent variables are linear combinations of x-variables that explain variance in x and y-variables. These latent variables, known as factors, are extracted iteratively from the data. A choice of the number of factors to include in a model can be made by considering diagnostic statistics such as the variable influence on projections (VIP).

Let SSx be the sum of squares of deviations from the mean, x-, for the variable x for a sample of size n, i.e., and let SCxy be the cross-products of deviations from the means, x- and y-, for the variables x and y for a sample of size n, i.e., Then the sample covariance of x and y is and the product-moment correlation coefficient is

There are five methods which perform nonparametric correlations, each of which is capable of producing both Spearman's rank-order and Kendall's tau correlation coefficients. The basic underlying concept of both these methods is to replace each observation by its corresponding rank or order within the observations on that variable, and the correlations are then calculated using these ranks.

It is obviously more convenient to order the observations and calculate the ranks for a particular variable just once, and to store these ranks for subsequent use in calculating all coefficients involving that variable; this does however require an amount of store of the same size as the original data matrix, which in some cases might be excessive. Accordingly, some methods calculate the ranks only once, and replace the input data matrix by the matrix of ranks, which are then also made available to you on exit from the method, while others preserve the data matrix and calculate the ranks a number of times within the method; the ranks of the observations are not provided as output by methods which work in the latter way. If it is possible to arrange the program in such a way that the first technique can be used, then efficiency of timing is achieved with no additional storage, whereas in the second case, it is necessary to have a second matrix of the same size as the data matrix, which may not be acceptable in certain circumstances; in this case it is necessary to reach a compromise between efficiency of time and of storage, and this may well be dependent upon local conditions.