Advanced Lecture on Neural Information Processing Systems (Lecture 02)

Advanced Lecture on

Neural Information Processing Systems (Lecture 02)

Ichiro Takeuchi

Nagoya Institute of Technology

Computer programs learned by themselves

▶

Learn a computer program

double func(double x1, double x2) {

double y = ???;

return y;

}

that satisfies the following input-output relations:

x

= 2, x

= 4 ⇒ y = 5

x

= 1, x

= 8 ⇒ y = 2

x

= 6, x

= 9 ⇒ y = 7

x

= 3, x

= 3 ⇒ y = 4

Input output relations

Linear models

▶

Consider linear input-output relations:

double func(double x1, double x2) {

double y = w1*x1 + w2*x2;

return y;

}

▶

Linear model for d-dimensional input x ∈ R

:

y = f(x

, x

, . . . , x

) = w

x

+ w

x

+ . . . + w

x

Simple linear regression: 1D input case

f(x) = wx

Examples

▶

Find a function

y = f(x) = wx

that satisfies the following input-output relations:

Example 1

x = 2 ⇒ y = 1 x = 6 ⇒ y = 3 x = 8 ⇒ y = 4 x = 4 ⇒ y = 2

Example 2

x = 2 ⇒ y = 1

x = 6 ⇒ y = 2

x = 8 ⇒ y = 4

x = 4 ⇒ y = 3

Plots

Minimizing errors

▶

Training data

input x output y x

y

x

y

.. . .. . x

y

▶

Minimizing the sum of squared errors:

w

= arg min

w∈R

∑

(y

− wx

)

Exercise

▶

Compute the optimal w ∈ R that minimizes the sum of squared errors:

E :=

∑

(y

− wx

)

where the training set is given as

x = 2 ⇒ y = 1

x = 6 ⇒ y = 2

x = 8 ⇒ y = 4

x = 4 ⇒ y = 3

Multiple linear regression

f(x

, x

, . . . , x

) = w

x

+ w

x

+ . . . + w

x

Caution: notations of the training data

▶

The training data is represented by X ∈ R

and y ∈ R

:

n

X

:=



 

 

x

x

· · · x

x

x

· · · x

.. . .. . . .. .. . x

x

· · · x



 

  =



 

  x

x

.. . x



 

  , y

n×1

:=



 

  y

y

.. . y



 

 

▶

x

∈ R : the j

input variable of the i

training instance

▶

x

∈ R

: the input vector of the i

training instance

Least square linear regression

▶

Training data:

X

n×d

:=



 

 

x

x

· · · x

x

x

· · · x

.. . .. . . .. .. . x

x

· · · x



 

  , y

n×1

:=



 

  y

y

.. . y



 

 

▶

Linear model estimation by LS method:

w

= arg min

w∈R^d

∑

(y

− (w

x

+ . . . + w

x

))

LS linear regression in matrix vector form

▶

The sum of squared error is written as

E :=

∑

(y

− (w

x

+ . . . + w

x

))

=

∑

( y

− x

w )

= (y − Xw)

(y − Xw)

= w

(X

X)w − 2(X

y)

w + y

y

Optimality conditions

Minimizing the sum of squared errors

▶

The sum of squared error:

E = w

(X

X)w − 2(X

y)

w + y

y

▶

The optimality condition:

∂E

∂w = 2(X

X)w − 2(X

y) = 0

▶

Normal equations:

(X

X)w = X

y

Example

Example

Final exersize

▶

Consider linear regression problem with a constant term:

▶ Training data: {(x_i, y_i)∈R×R}ⁿ_i=1

▶ Model: f(xi) =w0+w1xi, i= 1, . . . , n

▶ Problem: min_w0,w1∈R ∑_n

i=1(y_i−(w₀+w₁x_i))²

▶

Show that the solution of the problem is formulated as the following system of linear equations:

nw

+ (

∑

i=1

x

)

w

=

∑

y

(

∑

i=1

x

)

w

+ (

∑

i=1

x

)

w

=

∑

x

y