#least square

Introduction to Least Squares Regression<\p>

Suppose we wish in contemplation of predict the values of the variable Y from the value of the variable X.<\p>

Imperative of least square states that the deviate relating to best turn the scale for the given pair speaking of observation is that curve that makes the recount of the squares relative to the differences between observed value and the estimated value called residuals a plenty.<\p>

The straight delineation are surfeited by the principle of least squares to the pair of observations (cross, y) plotted on the scatter diagram are called comedown lines. The points on the scatter cluster themselves along these lines.<\p>

Let us take the regression equation as y=a+bx<\p>

Thus, if xi is an observed value of X, then the predicted idolize of Y in furtherance of the specificative reckon of DECADE will continue a+bxi.We are for that cause restricting ourselves to the ceremony of linear regression function.The benefit referring to a and b can be identified by the method of differential calculus cause obtaining the maximum and minimum with respect to functions.<\p>

Once the values speaking of a and b is decisive, the retroversion equation takes the simple form<\p>

` y - bary = (Cov(CRUX CAPITATA,Y))\sigma^2 (crux - craze x)`<\p>

where `sigma^2 `<\p>

denotes the departure of X.<\p>

This coextension is the equilibrium of the least puffy line of passage of Y on DECEMVIR.<\p>

The of long duration `(Cov(ENIGMA, Y))\sigma^2 `<\p>

is called the regression coefficient of Y on X and is denoted passing by bxy.<\p>

Similarly we can obtain the the minority square line of climbing of X on Y and draw<\p>

` x - bar x = (Cov(X,Y))\sigma^2 (y - bar y)`<\p>

where `sigma^2 `<\p>

denotes the dissonance of Y.<\p>

This tangent is the equation of the least square line of oblique motion of X prevailing Y.<\p>

The constant `(Cov(X, Y))\sigma^2 `<\p>

is called the regression coefficient of MARK on Y and is denoted by byx.<\p>

The lines of reflowing are called lowliest square lines of regression because they have been obtained abeam minimising the sums of squares.<\p>

Motive: Least-squares Falling back<\p>

Consider the observation <\p>

(1,2), (2,4), (3, 8), (4, 7), (5, 10), (6,5), (7,14), (8, 16), (9, 2), (10,20)<\p>

then we have `Sigma x = 55`<\p>

`sigma y = 88`<\p>

`Sigma x^2 = 385` <\p>

`sigma y^2 = 1114`<\p>

`sigma xy = 586`<\p>

`bar decemvir = 5.5`<\p>

`bar y = 8.8`<\p>

Off `byx = (586 -((55)(88))\10)\(385 - ((55) (55))\10) = 102\82.5 = 1.24`<\p>

and `bxy = (586 -((55)(88))\10)\(1114 - ((88) (88))\10) = 102\339.6=0.30`<\p>

The regression line of Y along X is<\p>

` y - 8.8 = 1.24 (tau - 5.5)`<\p>

` y = 1.24x +1.98`<\p>

The regression draftsmanship of X on Y is<\p>

` x - 5.5 = 0.3 (y - 8.8)`<\p>

In the previous post of this series an important assumption has been made: 

The Errors, it is to say the Point-Line Distance, are IID Variables following a Gaussian Distribution 

So the corresponding Normalized PDF is the following one 

Setting the Normalization Constant properly to 

and the variable 

the proper PDF is obtained 

Let's remember that the Likelihood has been defined as 

and the factorization has been obtained supposing independence among all of the random variables 

When dealing with factorized Likelihood Function it's usually more convenient to define a Log-Likelihood this way 

Thus leading to 

Obviously maximizing the Log-Likelihood is equivalent to minimizing the negative term 

but it's now obvious that $ \frac{1}{2 \sigma^2} $ is just a constant, even if it's unknown, and so it doesn't influence the minimum that depends only on the squre of the errors so it follows that 

The function to be minimized is convex so it could be solved with descent direction method 

So it has been formally demonstrated the Optimality for the Least Square Fitting Method under certain assumption

Now it should be more clear the assumptions under which this method provides an optimum, it is to say 

No A Priori knowledge requires the Maximum Likelihood 

Errors are assumed to be Random Variables IID following a Gaussian Distribution  

The first assumption is quite clear and no further comment is needed. 

The second assumption needs a couple of final observations: 

in real world problem, the Independence Assumption could be a rather strong one