Linear model

In statistics the linear model can be expressed by saying

• Y=Xβ+ε

where Y is an nx1 column vector of random variables, X is an nxp matrix of "known" (i.e., observable and non-random) quanitities, whose rows correspond to statistical units, β is a px1 vector of (unobservable) parameters, and ε is an nx1 vector of "errors", which are uncorrelated random variables each with expected value 0 and variance σ². Often one takes the components of the vector of errors to be independent and normally distributed. Having observed the values of X and Y, the statistician must estimate β and σ². Typically the parameters β are estimated by the method of least squares.

If, rather than taking the variance of ε to be σ²I, where I is the nxn identity matrix, one assumes the variance is σ²M, where M is a known matrix other than the identity matrix, then one estimates β by the method of "generalized least squares", in which, instead of minimizing the sum of squares of the residuals, one minimizes a different quadratic form in the residuals -- the quadratic form being the one given by the matrix M^-1. If all of the off-diagonal entries in the matrix M are 0, then one normally estimates β by the method of "weighted least squares", with weights proportional to the reciprocals of the diagonal entries.

Ordinary Linear regression is a very closely related topic.

"Generalized linear models", rather than saying

• E(Y)=Xβ,

say

• f(E(Y))=Xβ,

where f is the "link function". An example is the "Poisson regression model", which says

• Y_i has a Poisson distribution with expected value e^γ+δx_i.

The link function is the natural logarithm function. Having observed x_i and Y_i for i=1,...,n, one can estimate γ and δ by the method of maximum likelihood.