The regression model

2.1 Ordinary least squares (OLS) estimation method: two-variable case

In the bivariate case, suppose that we have a dataset on variable \(y\) and on variable \(x\). The data are collected in a sample of observations, say \(N\) different observations, on units indexed by \(i=1, \ldots , N\). Our aim is to approximate the value of \(y\) by a linear combination \(\hat {y}=\alpha +\beta x\), where \(\alpha \) and \(\beta \) are real constants to be determined. The \(i\)-th square residual \(e_i\) is given by \[e_i=y_i-\hat {y}_i=y_i-\alpha – \beta x_i,\] and the procedure consists in the minimization of the sum of squared residuals. Call \(S(\alpha , \beta )\) the function of the parameters indicating such a sum of squares, i.e. \begin {equation} S(\alpha , \beta )=\sum _{i=1}^N e_i^2=\sum _{i=1}^N \left (y_i – \alpha – \beta x_i \right )^2. \label {functionSbivariate} \end {equation} The related minimization problem is unconstrained. It reads as \begin {equation} \min _{\alpha , \ \beta } S(\alpha , \beta ), \label {basicOLSproblem} \end {equation} and the solution procedure obviously involves the calculation of the first order derivatives. The first order conditions (FOCs) are: \begin {equation*} -2 \sum _{i=1}^N \left (y_i – \alpha – \beta x_i \right )=0 \qquad \Longrightarrow \qquad \sum _{i=1}^N y_i – N \alpha – \beta \sum _{i=1}^N x_i=0. \end {equation*} \begin {equation*} -2 \sum _{i=1}^N \left (y_i – \alpha – \beta x_i \right )x_i=0 \qquad \Longrightarrow \qquad \sum _{i=1}^N x_i y_i – \alpha \sum _{i=1}^N x_i – \beta \sum _{i=1}^N x_i^2=0. \end {equation*} After a rearrangement, these \(2\) equations are typically referred to as normal equations of the \(2\)-variable regression model: \begin {equation} \sum _{i=1}^N y_i = N \alpha + \beta \sum _{i=1}^N x_i, \label {normalequation1} \end {equation} \begin {equation} \sum _{i=1}^N x_i y_i = \alpha \sum _{i=1}^N x_i + \beta \sum _{i=1}^N x_i^2. \label {normalequation2} \end {equation} Solving (4) for \(\alpha \) yields: \begin {equation} \alpha =\dfrac {\sum _{i=1}^N y_i – \beta \sum _{i=1}^N x_i}{N}= \overline {y} – \beta \overline {x}, \label {valueofalpha} \end {equation} after introducing the arithmetic means: \(\overline {x}=\dfrac {\sum _{i=1}^N x_i}{N}\), \(\overline {y}=\dfrac {\sum _{i=1}^N y_i}{N}\).

Plugging (6) into (5) amounts to: \begin {equation*} \sum _{i=1}^N x_i y_i – (\overline {y} – \beta \overline {x}) N \overline {x} – \beta \sum _{i=1}^N x_i^2=0, \end {equation*} hence \(\beta \) can be easily determined: \begin {equation*} \sum _{i=1}^N x_i y_i – N \overline {x} \cdot \overline {y} + \beta \left (N \overline {x}^2 – \sum _{i=1}^N x_i^2 \right )=0 \qquad \Longrightarrow \end {equation*} \begin {equation} \Longrightarrow \qquad \beta =\dfrac {\sum _{i=1}^N x_i y_i – N \overline {x} \cdot \overline {y}}{\sum _{i=1}^N x_i^2 – N \overline {x}^2}, \label {betaOLS} \end {equation} and consequently, inserting (7) into (6), we achieve: \begin {equation} \alpha = \overline {y} – \dfrac {\overline {x} \sum _{i=1}^N x_i y_i – N \overline {x}^2 \cdot \overline {y}}{\sum _{i=1}^N x_i^2 – N \overline {x}^2}. \label {alphaOLS} \end {equation} The regression line is given by: \begin {equation} \hat {y}=\alpha + \beta x, \label {regressionline} \end {equation} meaning that for each value of \(x\), taken from a sample, \(\hat {y}\) predicts the corresponding value of \(y\). The residuals can be evaluated as well, by comparing the given values of \(y\) with the ones that would be predicted by taking the given values of \(x\).

It is important to note that \(\beta \) can be also interpreted from the viewpoint of probability, when looking upon both \(x\) and \(y\) as random variables. Dividing numerator and denominator of (7) by \(N\) yields: \begin {equation} \Longrightarrow \qquad \beta =\dfrac {\dfrac {\sum _{i=1}^N x_i y_i}{N} – \overline {x} \cdot \overline {y}}{\dfrac {\sum _{i=1}^N x_i^2}{N} – \overline {x}^2}=\dfrac {Cov(x,y)}{Var(x)}, \label {betaOLSCovandVar} \end {equation} after applying the \(2\) well-known formulas: \[Cov(x,y)=\mathbb {E}[x \cdot y]-\mathbb {E}[x] \mathbb {E}[y], \qquad Var(x)=\mathbb {E}[x^2]-\left (\mathbb {E}[x]\right )^2.\] There exists another way to indicate \(\beta \), by further manipulating (7). Since \[\sum _{i=1}^N x_i y_i – N \overline {x} \cdot \overline {y}=\sum _{i=1}^N x_i y_i – N \overline {x} \cdot \overline {y} + N \overline {x} \cdot \overline {y} – N \overline {x} \cdot \overline {y}=\] \[=\sum _{i=1}^N x_i y_i – \overline {x} \sum _{i=1}^N y_i – \overline {y} \sum _{i=1}^N x_i + N \overline {x} \cdot \overline {y}=\sum _{i=1}^N (x_i – \overline {x})(y_i – \overline {y})\] and \[\sum _{i=1}^N x_i^2 – N \overline {x}^2=\sum _{i=1}^N x_i^2+N \overline {x}^2 – 2N \overline {x} \cdot \overline {x}=\sum _{i=1}^N x_i^2+ \sum _{i=1}^N \overline {x}^2 – 2 \overline {x} \sum _{i=1}^N x_i=\] \[=\sum _{i=1}^N (x_i-\overline {x})^2,\] \(\beta \) can also be reformulated as follows: \begin {equation} \beta =\dfrac {\sum _{i=1}^N (x_i- \overline {x})(y_i- \overline {y})}{\sum _{i=1}^N (x_i – \overline {x})^2}. \label {betaOLSalternative} \end {equation} The following Example illustrates an OLS and the related assessment of the residuals.

Example 1.

Consider the following \(6\) points in the \((x,y)\) plane, which correspond to \(2\) samples of variables \(x\) and \(y\): \[P_1=(0.3,0.5), \qquad P_2=(0.5, 0.7), \qquad P_3=(1,0.5),\] \[P_4=(1.5,0.8), \qquad P_5=(0.8,1), \qquad P_6=(0.7, 1.5).\] Let us calculate the regression parameters \(\alpha \) and \(\beta \) with the help of formulas (8) and (7) to determine the regression line: Since \[\overline {x}=\dfrac {0.3+0.5+1+1.5+0.8+0.7}{6}=0.8, \ \overline {y}=\dfrac {0.5+0.7+0.5+0.8+1+1.5}{6}=0.8\overline {3},\] we obtain: \[\alpha =0.8\overline {3}-\] \[\dfrac {0.8 (0.3 \cdot 0.5 + 0.5 \cdot 0.7 + 1 \cdot 0.5 + 1.5 \cdot 0.8 + 0.8 \cdot 1 + 0.7 \cdot 1.5)-6 \cdot (0.8)^2 \cdot 0.8\overline {3}}{(0.3)^2+(0.5)^2+1^2+(1.5)^2+(0.8)^2+(0.7)^2-6 \cdot (0.8)^2}=\] \[=0.7877.\] \[\beta =\dfrac {0.3 \cdot 0.5 + 0.5 \cdot 0.7 + 1 \cdot 0.5 + 1.5 \cdot 0.8 + 0.8 \cdot 1 + 0.7 \cdot 1.5-6 \cdot 0.8 \cdot 0.8\overline {3}}{(0.3)^2+(0.5)^2+1^2+(1.5)^2+(0.8)^2+(0.7)^2-6 \cdot (0.8)^2}=0.057,\] hence the regression line is: \[y=0.057x+0.7877.\] We can also calculate all the residuals \(e_i\), i.e. the differences between \(y_i\) and \(\hat {y}_i\), and their squares \(e_i^2\) as well.

\(i\)	\(y_i\)	\(\hat {y}_i\)	\(e_i\)	\(e_i^2\)
\(1\)	\(0.5\)	\(0.8048\)	\(-0.3048\)	\(0.0929\)
\(2\)	\(0.7\)	\(0.8162\)	\(-0.1162\)	\(0.0135\)
\(3\)	\(0.5\)	\(0.8447\)	\(-0.3447\)	\(0.1188\)
\(4\)	\(0.8\)	\(0.8732\)	\(-0.0732\)	\(0.0053\)
\(5\)	\(1\)	\(0.8333\)	\(0.1667\)	\(0.0277\)
\(6\)	\(1.5\)	\(0.8276\)	\(0.6724\)	\(0.4521\)

Note that the sum of the squares of the residuals is \(\sum _{i=1}^6 e_i^2=0.7103\). Moreover, the larger contribution comes from point \(P_6\), as can be seen from Figure 2, whereas \(P_2\) and \(P_4\) are ’almost’ on the regression line.

2.2 Assessment of the goodness of fit

Every time we carry out a regression, we need a measure of the fit of the obtained regression line to the data. We are going to provide the definitions of some quantities that will be useful for this purpose:

• Total Sum of Squares: \[SST=\sum _{i=1}^N (y_i-\overline {y})^2.\]
• Regression Sum of Squares: \[SSR= \sum _{i=1}^N (\hat {y}_i-\overline {y})^2.\]
• Error Sum of Squares: \[SSE= \sum _{i=1}^N (\hat {y}_i-y_i)^2.\]

The \(3\) above quantities are linked by the straightforward relation we are going to derive. Since we have: \[y_i-\overline {y}=y_i-\hat {y}_i+\hat {y}_i-\overline {y} \qquad \Longrightarrow \qquad (y_i-\overline {y})^2=(y_i-\hat {y}_i+\hat {y}_i-\overline {y})^2=\] \[=(y_i-\hat {y}_i)^2+(\hat {y}_i-\overline {y})^2 + 2(y_i-\hat {y}_i)(\hat {y}_i-\overline {y}).\] Summing over \(N\) terms yields: \begin {equation*} \sum _{i=1}^N (y_i-\overline {y})^2 = \sum _{i=1}^N (y_i-\hat {y}_i)^2+ \sum _{i=1}^N (\hat {y}_i-\overline {y})^2 + 2 \sum _{i=1}^N (y_i-\hat {y}_i)(\hat {y}_i-\overline {y}). \end {equation*} Now, let us take the last term in the right-hand side into account. Relying on the OLS procedure, we know that: \begin {equation*} 2 \sum _{i=1}^N (y_i-\hat {y}_i)(\hat {y}_i-\overline {y})=2 \sum _{i=1}^N (y_i – \hat {y}_i) (\alpha + \beta x_i – \alpha – \beta \overline {x})=2 \beta \sum _{i=1}^N (y_i – \hat {y}_i) (x_i – \overline {x})= \end {equation*} \begin {equation*} =2 \beta \sum _{i=1}^N (y_i – \hat {y}_i+\overline {y}-\overline {y}) (x_i – \overline {x})=2 \beta \sum _{i=1}^N (y_i – \alpha -\beta x_i+\alpha +\beta \overline {x}-\overline {y}) (x_i – \overline {x}) \end {equation*} \begin {equation*} =2 \beta \sum _{i=1}^N (y_i -\overline {y} – \beta (x_i – \overline {x})) (x_i – \overline {x})=2 \beta \left [\sum _{i=1}^N (y_i – \overline {y})(x_i – \overline {x}) – \beta \sum _{i=1}^N (x_i – \overline {x})^2 \right ]= \end {equation*} \begin {equation*} =2 \beta \left [\sum _{i=1}^N (y_i – \overline {y})(x_i – \overline {x})-\dfrac {\sum _{j=1}^N (x_j- \overline {x})(y_j- \overline {y})}{\sum _{j=1}^N (x_j – \overline {x})^2} \cdot \sum _{i=1}^N (x_i – \overline {x})^2 \right ]=0, \end {equation*} after employing expression (11) to indicate \(\beta \). Since the above term vanishes, we obtain: \begin {equation*} \sum _{i=1}^N (y_i-\overline {y})^2 = \sum _{i=1}^N (y_i-\hat {y}_i)^2+ \sum _{i=1}^N (\hat {y}_i-\overline {y})^2, \end {equation*} then the following relation holds: \[SST=SSE+SSR.\] Now we can introduce a coefficient which is helpful to assess the closeness of fit: the coefficient of determination \(R^2 \in (0,1)\). \[R^2=\dfrac {SSR}{SST}=\dfrac {\sum _{i=1}^N (\hat {y}_i – \overline {y})^2}{\sum _{i=1}^N (y_i-\overline {y})^2}=\beta ^2 \dfrac {\sum _{i=1}^N (x_i-\overline {x})^2}{\sum _{i=1}^N (y_i-\overline {y})^2}.\] An equivalent formulation of \(R^2\) is the following one: \[R^2=1-\dfrac {SSE}{SST}=1 – \dfrac {\sum _{i=1}^N (\hat {y}_i – y_i)^2}{\sum _{i=1}^N (y_i-\overline {y})^2}.\] The regression line fits the scatter of points better as close as \(R^2\) is to \(1\). We can calculate \(R^2\) in the previous Example, obtaining the value: \(R^2=0.004\).

2.3 Ordinary least squares (OLS) estimation method: multiple variable case

When \(N>2\), we are in a standard scenario, because typically more than \(2\) variables are involved in an economic relationship. The standard linear equation that we are faced with reads as: \begin {equation} y=\beta _1 + \beta _2 x_2 + \beta _3 x_3 + \cdots + \beta _N x_N + \epsilon , \label {Nvariableslinearequation1} \end {equation} where we chose not to use \(x_1\) to leave the intercept alone, and \(\epsilon \) represents the above-mentioned disturbance. Another possible expression of the same equation is: \begin {equation} y=\beta _1 x_1 + \beta _2 x_2 + \beta _3 x_3 + \cdots + \beta _N x_N + \epsilon . \label {Nvariableslinearequation2} \end {equation} In (12) there are \(N\) regression parameters to be estimated. Taking the expectations and assuming \(\mathbb {E}(\epsilon )=0\), we have: \begin {equation} \mathbb {E}[y]=\beta _1 + \beta _2 x_2 + \beta _3 x_3 + \cdots + \beta _N x_N, \label {Nvariableslinearequationexpected} \end {equation} which is usually indicated as the population regression equation. In (14) \(\beta _1\) is the intercept and \(\beta _2, \ldots , \beta _N\) are the regression slope parameters. Suppose that our sample is composed of \(M\) observations for the explanatory variables \(x_i\). We can write the values in the \(i\)-th observations as: \[y_i, \ x_{2i}, \ x_{3i}, \ldots , x_{Ni}.\] For all \(i=1, \ldots , M\), we have: \[y_i= \beta _1 + \beta _2 x_{2i}+ \cdots + \beta _N x_{Ni}+\epsilon _i,\] or, in simple matrix form: \begin {equation} Y= X \beta + \epsilon , \label {matrixform} \end {equation} where \(Y\), \(\beta \) and \(\epsilon \) are the following vectors: \begin {equation*} Y=\left (\begin {array}{c} y_1 \\ y_2 \\ \vdots \\ \vdots \\ y_M \end {array} \right ), \qquad \beta =\left (\begin {array}{c} \beta _1 \\ \beta _2 \\ \vdots \\ \beta _N \end {array} \right ), \epsilon =\left (\begin {array}{c} \epsilon _1 \\ \epsilon _2 \\ \vdots \\ \vdots \\ \epsilon _M \end {array} \right ). \end {equation*} On the other hand, \(X\) is the following \(M \times N\) matrix: \begin {equation*} X=\left (\begin {array}{ccccc} 1 & x_{21} & x_{31} & \cdots & x_{N1} \\ & & & & \\ 1 & x_{22} & \cdots & \cdots & x_{N2} \\ & & & & \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ 1 & x_{2M} & \cdots & \cdots & x_{NM} \end {array} \right ). \end {equation*} If \(\hat {\beta }_1, \ldots , \hat {\beta }_N\) are estimated values of the regression parameters, then \(\hat {y}\) is the predicted value of \(y\). Also here residuals are \(e_i=y_i-\hat {y}_i\), and \(e\) is the vector collecting all the residuals. We have \[Y=\hat {Y}+e \qquad \Longleftrightarrow \qquad e=Y-X \hat {\beta }.\] Also in this case we use OLS, so we are supposed to minimize the sum of the squares of the residuals \(S=\sum _{i=1}^N e_i^2\). We can employ the standard properties of Linear Algebra to achieve the following form (\(^T\) indicates transpose): \[S=e^T e=\left (Y- X \hat {\beta } \right )^T \left (Y-X \hat {\beta } \right )=\left (Y^T- \hat {\beta }^T X^T \right ) \left (Y-X \hat {\beta } \right )=\] \[=Y^T Y – \hat {\beta }^T X^T Y-Y^T X \hat {\beta }+\hat {\beta }^T X^T X \hat {\beta }=\] \[=Y^T Y – 2 \hat {\beta }^T X^T Y+\hat {\beta }^T X^T X \hat {\beta },\] because they are all scalars, as is simple to check (\(e^T e\) is a scalar product). The \(2\) negative terms have been added because \(\hat {\beta }^T X^T Y=\left (Y^T X \hat {\beta }\right )^T\).

As in the \(2\)-variables case, the next step is the differentiation of \(S\) with respect to \(\hat {\beta }\), i.e. \(N\) distinct FOCs which can be collected in a unique vector of normal equations: \begin {equation} \dfrac {\partial S}{\partial \hat {\beta }}=-2 X^T Y+ 2 X^T X \hat {\beta }=0. \label {NFOCs} \end {equation} The relation (16) can be rearranged to become: \[X^T X \hat {\beta }=X^T Y \qquad \Longrightarrow \qquad \left (X^T X \right )^{-1} X^T X \hat {\beta }= \left ( X^T X \right )^{-1} X^T Y,\] which can be solved for \(\hat {\beta }\) to achieve the formula which is perhaps the most famous identity in Econometrics: \begin {equation} \hat {\beta }= \left ( X^T X \right )^{-1} X^T Y. \label {bestformulainEconometrics} \end {equation} Clearly, the matrix \(X^T X\) must be non-singular, and the determination of its inverse may need a long and computationally costly procedure.

In the next Example, we are going to employ (17) in a simple case where the regression parameters are only \(3\) and the observations are \(3\) as well, to avoid excessive calculations.

2.4 Assumptions for classical regression models

Typically, some assumptions are made on the explanatory variables and on the disturbances in the regression models. Such assumptions are not always the same, as can be seen by comparing different approaches. We are going to refer to the list of assumptions proposed by Greene (see [1], p. \(56\)), augmenting it with a brief explanation on their meaning and importance. The first trivial assumption, which is not listed generally, concerns the values of \(x\) in the sample. We assume that there are some variations in each sample, meaning that for all \(h=2, \ldots , N\), there exist at least \(2\) different values, i.e. \(i \neq j\) such that \(x_{hi} \neq x_{hj}\). If this assumption is not verified, there are some variables which are actually constant.

Assumptions involving the explanatory variables

• (1A) – Linearity: a linear relationship is specified between explained and explanatory variables, i.e. (12) or (13);
• (1B) – Full rank: no exact linear relationship exists among any of the model’s explanatory variables;
• (1C) – Data generation: the data collected in the independent variables can be either constants or random variables or a mixture of both;

Assumption (1A) intends to establish the validity of the regression equation, whereas assumption (1B) means that no further constraints have to be taken into account (clearly, any linear relation among explanatory variables would be equivalent to the redundancy of some variables, so the system should be reduced.

On the other hand, assumption (1C) states that analysis is carried out conditionally on the observed values of \(X\), so hence the outcome will not be influenced by the specific nature of the values (either fixed constants or random draws from a stochastic process).

We also have to consider \(4\) assumptions on all the disturbances, that are listed as follows.

Assumptions involving the disturbances

It is assumed that for all \(i=1, \ldots , N\), the disturbances \(\epsilon _i\):

• (2A) – Exogeneity of the independent variables: \(\mathbb {E}[\epsilon _i]=0\) and \(\mathbb {E}[\epsilon _i \ | \ X]=0\);
• (2B) – Homoscedasticity²: \(Var (\epsilon _i)=\mathbb {E}[\epsilon _i-\mathbb {E}[\epsilon _i]]^2=\sigma ^2=\) constant; moreover \(Var[\epsilon _i | X]=\sigma ^2=\) constant;
• (2C) – Non-autocorrelation: \(Cov(\epsilon _i, \epsilon _j)=\mathbb {E}\left \{[\epsilon _i-\mathbb {E}[\epsilon _i]][\epsilon _j-\mathbb {E}[\epsilon _j]]\right \}=0\) for all \(i \neq j\);
• (2D) – Normal distribution: each \(\epsilon _i\) is normally distributed with zero mean.

\((2A)\) refers to the mean values of the disturbances, either conditional on \(X\) or not. This property denotes exogeneity of \(X\) (in other words, \(X\) is an exogenous variable), which has great importance in economic models, because it corresponds to the fact that \(X\) is really an external variable, so its effect on \(Y\) is ’pure’. On the other hand, assumption \((2B)\) is called Homoscedasticity, and it means that conditional variance is constant. When this assumption does not hold, there is Heteroscedasticity (or Heteroskedasticity), which is a definitely more complex case.

Here we are going to state some results to prove the correctness, or unbiasedness³ of the regression parameters under some of the above assumptions. Basically, why do we use estimators achieved from the OLS method? We will see that estimators \(\alpha \) and \(\beta \) have very relevant properties. Suppose that the following linear equation: \[y=\alpha ^* + \beta ^* x + \epsilon \] contains the best parameters to fit the scatter of points. The following results are stated in the \(2\)-variables case, but they can be easily extended to \(N\) variables.

Clearly, the mean value is not the only important characteristic of the regression parameters: as usually happens with random variables, the variance is crucial as well. This means that in addition to being a correct estimator, parameter \(\beta \) must also have a low variance. We are going to introduce the following result, also known as the Gauss-Markov Theorem, under the homoscedaticity assumption:

Sometimes estimator \(\beta \) in (7) is indicated as the BLUE (Best Linear Unbiased Estimator).

After discussing the properties of \(\beta \), we should examine the distribution of errors. Assumption \((2D)\) establishes that each disturbance \(\epsilon _i\) is normally distributed with \(0\) mean. At the present stage, we do not have any information on \(\sigma ^2\), i.e. the variance of \(\beta \) is still to be estimated. First, we have to come back to the expression of the least squares residuals \(e_i\) in \(N\) variables and apply (2.3.6 ): \[e = Y – X \hat {\beta } = Y – X \left ( X^T X \right )^{-1} X^T Y = (I_M – X \left ( X^T X \right )^{-1} X^T) Y,\] where \(I_M\) is the usual \(M \times M\) identity matrix.

Now, call \(\mathcal {M} = I_M – X \left ( X^T X \right )^{-1} X^T\) the \(M \times M\) residual maker (see [1], p. \(71\)). We have that \(e = \mathcal {M} Y\), furthermore, by construction: \[\mathcal {M} X = (I_M – X \left ( X^T X \right )^{-1} X^T) X = X – X \left ( X^T X \right )^{-1} (X^T X) = X – X I_N= 0,\] i.e the null \(M \times N\) matrix. We know from the above identity that the residual maker is also useful because \[e = \mathcal {M} Y =\mathcal {M} (X \beta + \epsilon ) = \mathcal {M} \epsilon .\] So an estimator of \(\sigma ^2\) can be obtained from the sum of squared residuals: \[e^T e = \left (\mathcal {M} \epsilon \right )^T \mathcal {M} \epsilon = \epsilon ^T \mathcal {M}^T \mathcal {M} \epsilon .\] Before proceeding, we prove another key property of the residual maker: \[\mathcal {M}^T \mathcal {M} = \left ( I_M – X \left ( X^T X \right )^{-1} X^T \right )^T \left (I_M – X \left ( X^T X \right )^{-1} X^T \right ) = \] \[ = \left ( I_M – X \left (\left ( X^T X \right )^{-1}\right )^T X^T \right ) \left (I_M – X \left ( X^T X \right )^{-1} X^T \right ) =\] \[ = I_M – X \left (\left ( X^T X \right )^{-1}\right )^T X^T – X \left ( X^T X \right )^{-1} X^T + \] \[ + X \left (\left ( X^T X \right )^{-1}\right )^T X^T X \left ( X^T X \right )^{-1} X^T = \] \[ = I_M – X \left (\left ( X^T X \right )^{-1}\right )^T X^T – X \left ( X^T X \right )^{-1} X^T + X \left (\left ( X^T X \right )^{-1}\right )^T X^T = \] \[ = I_M – X \left ( X^T X \right )^{-1} X^T = \mathcal {M}.\] Since \(\mathcal {M}^T \mathcal {M} = \mathcal {M}\), we have that: \[e^T e = \epsilon ^T \mathcal {M} \epsilon .\] Borrowing a property of the trace of a matrix from Linear Algebra, we have: \[tr(\epsilon ^T \mathcal {M} \epsilon ) = tr(\mathcal {M} \epsilon \epsilon ^T) \ \ \Longrightarrow \ \ \mathbb {E}[tr(\epsilon ^T \mathcal {M} \epsilon ) \ | \ X]=\mathbb {E}[tr(\mathcal {M} \epsilon \epsilon ^T) \ | \ X].\] Now we note that \(\mathcal {M}\) can be taken out of the expectation, so that: \[\mathbb {E}[tr(\mathcal {M} \epsilon \epsilon ^T) \ | \ X] = tr(\mathcal {M}\mathbb {E}[\epsilon \epsilon ^T \ | \ X] = tr (\mathcal {M} \sigma ^2 I_M) = \sigma ^2 tr(\mathcal {M}).\] The trace of \(\mathcal {M}\) can be calculated easily, using its properties: \[tr(\mathcal {M}) = tr (I_M – X \left ( X^T X \right )^{-1} X^T) = tr (I_M) – tr (\left ( X^T X \right )^{-1} X^T X) = M – N.\] Finally, we obtain that \(\mathbb {E}[e^t e \ | \ X] = (M – N) \sigma ^2,\) and we are able to define an unbiased estimator of \(\sigma ^2\), which is called \(s^2\): \begin {equation} s^2 = \dfrac {e^T e}{M-N}. \label {s2} \end {equation} Note that \(\mathbb {E}[s^2]=\sigma ^2\). The quantity (18) will be very useful in the testing procedures. We will also call \(s\) the standard error of regression.

To conclude this preliminary discussion on parameters, given the previous assumptions and results, we can state that the distribution of \(\beta \) is the following: \begin {equation} \beta | x \ \sim \ N \left (\beta ^*, \sigma ^2 (X^T X)^{-1} \right ), \label {betadistribution} \end {equation} i.e. a multivariate normal distribution, meaning that each component of \(\beta \) is normally distributed: \begin {equation} \beta _k | x \ \sim \ N \left (\beta _k^*, \sigma ^2 (X^T X)_{kk}^{-1} \right ). \label {betakdistribution} \end {equation} Finally, as far as \(s^2\) is concerned, we must remember that \[\mathbb {E}[s^2 \ | \ x]=\mathbb {E}[s^2]=\sigma ^2.\]