#general linear model

Simplifying Freedman-Lane, the permutation method used in randomise and PALM.

Doing a permutation test with the general linear model (GLM) in the presence of nuisance variables can be challenging. Let the model be:

where is a matrix of observed variables, is a matrix of predictors of interest, is a matrix of covariates (of no interest), and is a matrix of the same size as with the residuals.

Because the interest is in testing the relationship between and ,…

A generalized additive model (GAM) is much more flexible than a general linear model  A GAM is of the form

g(E(Y)) = β0 + f1(x1) + f2(x2) +...+ fn(xn)

  Where the functions fi’s  and g functions can be parametric , nonparametric or splines.

A generalized linear model (GLM), there is a g function but no f functions. So, the model is of the form:

g(E(Y)) = β0 + f1(x1) + f2(x2) +...+ fn(xn)

 A general linear model (also called GLM, hence create confusion), there is no g function and f functions are scalar multiplication by numbers. So, the model is of the form:

Y = β0 + β0 x1 + β0 x2 +...+ β0 xn

 Generalized linear model Vs general linear models:

For general linear models the distribution of residuals is assumed to be Gaussian. If it is not the case, it turns out that the relationship between Y and the model parameters is no longer linear. But if the distribution of residuals is one from the exponential family such as binomial, Poisson, negative binomial, or gamma distributions, there exists some functions of mean of Y, which has linear relationship with model parameters. This function is called link function.

For example, a binomial residual can use a logit or a probit link function. A Poisson residual uses a log link function.

Commonly used link functions for different types of response variables:

Where μi denotes the mean response of the ith row and Φ−1(·) denotes the inverse cumulative distribution function of the normal distribution

Read more about link functions: https://en.wikipedia.org/wiki/Generalized_linear_model

 The basic difference between generalized linear model & general linear model can be summarized as follows:

  Examples:

Multiple linear regressions (general linear model):

Example_1:

House price = β0 + β1 * number of rooms + β2  * size of house + β3  * covered parking available (yes/ no) + ... + βn * average income in the respective neighborhood according to census data + white noise

Logistic Regression     (Generalized linear model):

Example_1:

Log (probability the Donald Trump will win US presidential election/ He will lose)

= β0 + β1 * amount of money spent on his campaign+ β2  * amount of time spent campaigning negatively+ β3  * his popularity index + ... + βn * other factors + white noise

Here, Log (probability the Donald Trump will win US presidential election/ He will lose) is called logit, which can be interpreted as log(odd ratio). Interpret regression coefficients is different from that of in case of linear regression.

Example_2:

Log (student will get admission in graduate school/ He/she will not get admission)          

= β0 + β1 * Graduate Record Exam scores + β2 * grade point average+ β3 * prestige of the undergraduate institution+ ... + βn * other factors + white noise

Generalized additive model:

Log (customer will make a purchase on festive sale/ He or she will not make a purchase)

= β0 + f1(number of accounts) + f2(active account types) + f3(credit limits) + ... + βn * (age of the individual) + white noise

 Software implementation:

Reference:

Generalized Additive Models were developed by Hastie and Tibshirani (1986)1.

1http://web.stanford.edu/~hastie/Papers/gam.pdf

Simple linear regression and multiple regression are subsets of general linear models. A typical GLM has K+1 parameters and looks something like this

y=B0+B1x1+B2x2+B3x3...BkXk+epsilon

and "linear" here means linear in the parameters (x's can be of higher order and the model can be called linear). B0 is still the slope (value when all Xi=0), but now B1---Bk are partial slopes, meaning each Bi they represent the change in y per unit change in Xi, holding all other Xj constant.

GLMs an be additive (y=B0+B1x1i+B2x2i+error) which indicates the belief that the effect of the explanatory variables are additive or GLMS can include an interaction term (y=B0+B1x1+B2x2+B3X1X2), meaning the explanatory variables affect each other in an increasing or decreasing way.

Dummy variables are sometimes used when using categorical variables, like groups. The use of dummy variables get around the following problem: if we have 4 regions and code them 1,2,3,4 then we are forcing the difference between groups 2 and 3 to be the same as the difference between 1 and 4.