Simple Linear Regression Equation

Performing a simple linear regression analysis results in a regression equation of the form:

$$\hat{Y}=b_0 + b_1 \cdot X$$

To calculate the slope coefficient $b_1$ of the regression line, use of the following formula:

$$b_1 =\cfrac{\sum\limits_{i=1}^n{(X_i-\bar{X})(Y_i-\bar{Y})}}{\sum\limits_{i=1}^n{(X_i-\bar{X})^2}}=r_{\small{XY}}\bigg(\cfrac{s_{\small{Y}}}{s_{\small{X}}}\bigg)$$

Once the slope is known, it is possible to calculate the intercept coefficient $b_0$ with the following formula:

$$b_0 = \bar{Y} - b_1 \cdot \bar{X}$$

Residuals

The residual of the $i^{th}$ observation is calculated as follows:

$$e_i = Y_i - \hat{Y}_i$$

where $Y_i$ is the observed value and $\hat{Y_i}$ the predicted value for observation $i$.

Regression Sum of Squares

The total sum of squares, denoted $SS_{total}$, represents all the variation in $Y$ that could possibly be explained by the regression model.

$$SS_{total}=\sum_{i=1}^{n} (Y_i - \bar{Y})^2$$

The model sum of squares, denoted $SS_{model}$, represents the amount of variation in the outcome variable $Y$ that can be explained by the regression model.

$$SS_{model}=\sum_{i=1}^{n} (\hat{Y}_i - \bar{Y})^2$$

The residual sum of squares, denoted $SS_{model}$, represents the amount of variation in the outcome variable $Y$ that cannot be explained by the regression model.

$$SS_{residual}=\sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2=\sum_{i=1}^{n} (e_i)^2$$

Coefficient of Determination

The coefficient of determination, denoted $R^2$, is the proportion of the total variation in the outcome variable $Y$ that can be explained by the regression model.

$$R^2=\cfrac{SS_{model}}{SS_{total}}\,\,\,\,\,\,\,\,\,\, \text{or} \,\,\,\,\,\,\,\,\,\, R^2=1-\cfrac{SS_{residual}}{SS_{total}}$$

Standard Error of the Estimate

The standard deviation of the errors is called the standard error of the estimate and is calculated as follows:

$$s_E = \sqrt{MS_{error}}=\sqrt{\cfrac{SS_{residual}}{n-2}} =\sqrt{ \cfrac{\displaystyle \sum_{i=1}^{n} (e_i)^2}{n-2}}$$

Standard Error of the Slope

The standard error of the slope $s_{b_1}$ is a measure of the amount of error we can reasonably expect when using sample data to estimate the slope $\beta_1$ of a linear regression model and is calculated with the following formula:

$$s_{b_1} = \cfrac{s_E}{\sqrt{SS_X}} = \cfrac{s_E}{ \sqrt{ \sum_{i=1}^{n} (X_i - \bar{X})^2 } }$$

where $s_E$ is the standard error of the estimate.

Confidence Interval for the Slope of a Linear Model

The general formula for computing a $C\%\,CI$ for the slope $\beta_1$ is:

$$CI_{\beta_1}=\bigg(b_1 - t^*\cdot s_{b_1},\,\,\,\, b_1 + t^*\cdot s_{b_1} \bigg)$$

Where $t^*$ is the critical value of the $t_{n-2}$ distribution such that $\mathbb{P}(-t^* \leq t \leq t^*)=\frac{C}{100}$.

Hypothesis Test for the Slope of a Linear Model

The hypotheses of a two-sided test for the slope $\beta_1$ of a linear model are:

$$\begin{array}{rcl} H_0: \beta_1 = 0\\ H_a: \beta_1 \neq 0 \end{array}$$
The relevant test statistic for the null hypothesis $H_0: \beta_1 = 0$ is:

$$t_{b_1} = \cfrac{b_1}{s_{b_1}}$$
Under the null hypothesis of the test, $t_{b_1}$ follows a $t$-distribution with $df = n-2$ degrees of freedom:

$$t_{b_1} \sim t_{n-2}$$