1. Simple linear regression

Regression equation simple linear regression

$$\begin{equation*} \widehat{y}_i=a+bx_i, \end{equation*}$$ where the regression coefficient is estimated by
$$\begin{equation*} b=r_{xy} \left(\frac{s_y}{s_x}\right) \end{equation*}$$ and the intercept is estimated by
$$\begin{equation*} a=\overline{y}-b\overline{x}. \end{equation*}$$

---

Residual

The residual (or prediction error) is

$$\begin{equation*} (y_i -\hat{y_i}), \end{equation*}$$ where $y_i$ is the observed value and $\hat{y_i}$ the predicted value for person $i$.

---

Sums of squares for y

$$\begin{eqnarray*} \sum_{i=1}^n\limits (y_i-\overline{y})^2 = & \sum_{i=1}^n\limits (\widehat{y}_i-\overline{y})^2 & + \sum_{i=1}^n\limits (y_i-\widehat{y}_i)^2, \\ \text{also referred to as:} \\ SS_y = & SS_{\widehat{y}-\overline{y}} & + \text{ } SS_{y-\widehat{y}},\\ \text{or as:} \\ SS_{tot} = & SS_{reg} & + \text{ } SS_{res} \end{eqnarray*}$$ where $SS_{tot}$ is the total sum of squares of $y$, $SS_{reg}$ is the regression sum of squares 'explained' by the model and $SS_{res}$ is the residual sum of squares.

---

Proportion explained variation

The proportion explained variation (also called the proportional reduction in prediction error) is

$$\begin{eqnarray*} r^2_{xy} &=& \frac{\sum_{i=1}^n\limits (y_i-\overline{y})^2 - \sum_{i=1}^n\limits (y_i-\widehat{y}_i)^2}{\sum_{i=1}^n\limits (y_i-\overline{y})^2},\\ &=&\frac{SS_{tot}-SS_{res}}{SS_{tot}},\\ &=& \frac{SS_{reg}}{SS_{tot}} \end{eqnarray*}$$

---

$t$-test for regressin

The test statistic for regression coefficient $b$ assuming $H_0$: $\beta=\beta_0=0$ is

$$\begin{equation*} t=\frac{(b-\beta_0)}{se_b}. \end{equation*}$$ where $se_b$ is calculated by software. The statistic follows a $t$ distribution with $n-2$ degrees of freedom ($df=n-2$), when the assumptions hold.

---

Standardized residual

The standardized residual equals

$$\begin{equation*} \frac{y_i-\widehat{y_i}}{se_{y_i-\widehat{y_i}}}, \end{equation*}$$ where $se_{y_i-\widehat{y_i}}$, the standard error for the residual (also referred to as $se_{res}$) is calculated by software.

---

Residual standard deviation

The residual standard deviation based on $n$ observations equals

$$\begin{equation*} s_{res} = \sqrt\frac{\sum_{i=1}^n\limits{(y_i-\widehat{y_i})^2}}{n-k}, \end{equation*}$$ in other words: $s_{res}=\sqrt{\frac{SS_{res}}{n-k}}$, where $k$ equals the number of parameters in the regression equation ($k=2$ for simple regression).

---

$95\%$ - prediction interval for $\mu_y$

$$\begin{equation*} \widehat{y}_i-2s\le y_i \le\widehat{y}_i+2s, \end{equation*}$$ where $s$ is the residual standard deviation and 2 is an approximation of $t_{\alpha/2}$.

---

$95\%$ - confidence interval for $\mu_y$

$$\begin{equation*} \widehat{y}-2(s/\sqrt{n})\le \mu_y \le\widehat{y}+2(s/\sqrt{n}), \end{equation*}$$ where $s$ is the residual standard deviation, 2 is an approximation of $t_{\alpha/2}$ and $n$ is the number of observations.

---

---

2. Multiple linear regression

Regression equation simple multiple regression

For the independent variables (predictors) $x_{1},x_{2},x_{3},\ldots,x_{j},\ldots$

$$\begin{equation*} \widehat{y}_i=a+b_1x_{i1}+b_2x_{i2}+b_3x_{i3}+\ldots+b_jx_{ij}+\ldots \end{equation*}$$

---

Proportion explained variation

The proportion explained variation (proportional reduction in prediction error), or squared multiple correlation coefficient is

$$\begin{eqnarray*} R^2&=& \frac{\sum_{i=1}^n\limits (y_i-\overline{y})^2 - \sum_{i=1}^n\limits (y_i-\widehat{y}_i)^2} {\sum_{i=1}^n\limits (y_i-\overline{y})^2}, \end{eqnarray*}$$
In other words: $R^2 = \frac{SS_{tot}-SS_{res}}{SS_{tot}}=\frac{SS_{reg}}{SS_{tot}}$.

---

Multiple correlation coefficient

$$\begin{equation*} R= \sqrt{R^2} \end{equation*}$$

---

$F$-test statistic regression analysis

The null hypothesis that all regression coefficients equal zero is tested using

$$\begin{equation*} F = \frac{\displaystyle\frac{\sum_{i=1}^n\limits (\widehat{y}_i- \overline{y})^2}{k-1}}{\displaystyle\frac{\sum_{i=1}^n\limits (y_i-\widehat{y}_i)^2}{n-k}} = \frac{\displaystyle\frac{SS_{reg}}{df_{reg}}}{\displaystyle\frac{SS_{res}}{df_{res}}} = \frac{MS_{reg}}{MS_{res}}, \end{equation*}$$ where $k$ equals the number of parameters in the regression equation and $n$ the number of observations. The degrees of freedom are $df_{reg}=k-1$ en $df_{res}=n-k$. $df_{reg}$ and $df_{res}$ are often referred to as $df_1$ and $df_2$.

$MS$ denotes mean squares. $MS_{res}$ denotes the residual variance.

---

Test statistic for $b_j$

The test statistic for regression coefficient $b$ assuming $H_0$: $\beta=\beta_0=0$ is

$$\begin{equation*} t_{b_j}=\frac{(b_j-\beta_0)}{se_{b_j}}, \end{equation*}$$ where $se_{b_j}$ is calculated by software. The statistic follows a $t$ distribution with $n-k$ degrees of freedom, where $k$ equals the number of parameters in the regression equation ($k=2$ for simple regression), when the assumptions hold.

---

$100(1-\alpha)\%$ - confidence interval for $\beta_j$

$$\begin{equation*} b_j-t_{\alpha/2}\cdot se_{b_j}\le \beta_j \le b_j+t_{\alpha/2}\cdot se_{b_j}. \end{equation*}$$

---

---

3. Exponential regression

Regression equation simple exponential regression

$$\begin{equation*} \mu_y = \alpha\beta^x, \end{equation*}$$ where $\beta >0$ must hold. This populatie level equation provides the predicted value for the population mean of $y$ for a given value of $x$.

---

---

4. Logistic regression

Log-odds (logit)

If $y$ is a dichotomous (binary) variable taking on values 0 or 1 and we denote $p(y=1)$ as $p$, then:

$$\begin{equation*} \begin{split} \mbox{odds}&=\frac{p}{1-p} \\ \mbox{log-odds} =\mbox{logit}(p)&=\ln\Bigg(\frac{p}{1-p}\Bigg)\\ p&=\frac{\mbox{odds}}{1+\mbox{odds}} \end{split} \end{equation*}$$ To calculate the probability $p$ from a log-odds value requires the following rule: $e^{ln(x)}=x$.

---

Regression equation simple logistic regression

$$\begin{equation*} p(Y=1)=\frac{e^{\alpha+\beta x}}{1+e^{\alpha+\beta x}}, \end{equation*}$$ for which the following holds:
$$\begin{equation*} \mbox{logit}\big(P(Y=1)\big) = \alpha+\beta x. \end{equation*}$$