Machine Learning Garden

Powered by 🌱Roam Garden

Supervised Learning

References

Tags: concept

Sources:

Related notes:

Machine Learning

Deep Learning

Updates:

April 19th, 2021: add lecture notes from CS 182: Lecture 2, Part 2: Machine Learning Basics.

April 18th, 2021: created note.

Notes {{word-count}}

Summary:

Key points:

In Supervised Learning, given \mathcal{D}=\left{\left(x_{1}, y_{1}\right),\left(x_{2}, y_{2}\right), \ldots,\left(x_{n}, y_{n}\right)\right}, the objective is to learn fθ(x)≈yf_{\theta}(x) \approx y.

Generally, our goal is to predict yy given some xx.

However, prediction itself is very difficult because there are many boundary cases in the real world.

Predicting probabilities

So we use probabilities to represent the likelihood of a prediction falling into a certain category.

Predicting probabilities instead of labels can make training easier, which is due to smoothness,

Intuitively, discrete labels cannot be changed by a bit. It's either all or nothing.

Given \mathcal{D}=\left{\left(x_{1}, y_{1}\right),\left(x_{2}, y_{2}\right), \ldots,\left(x_{n}, y_{n}\right)\right}, the objective is to learn pθ(y∣x)p_{\theta}(y \mid x).

xx is a Random Variable representing the input.

xx is a random variable because we do not know what xx we will get. There is some true underlying process in the real world that gives rise to different xx's.

yy is a Random Variable representing the output.

p(x,y)=p(x)p(y∣x)p(x, y)=p(x) p(y \mid x) by Chain Rule (Probability).

p(y∣x)=p(x,y)p(x)\displaystyle p(y \mid x)=\frac{p(x, y)}{p(x)} by the definition of Conditional Probability.

Models that learn p(y∣x)p(y \mid x) are called Discriminative Model because the goal is to discriminate between different yy's.

Models that learn p(x,y)p(x, y) are called Generative Model because such a model can learn to generate xx.

If we can learn p(x,y)p(x, y), we can recover p(y∣x)p(y \mid x) from the definition of Conditional Probability.

When predicting probabilities, instead of representing the output by object labels, we do it using objective probability: what is the likelihood of this object falling into this category?

Softmax

How to make a number zz postive?

exp⁡(z)\exp(z) is especially convenient because it is one to one and onto and it maps the entire real number line to an entire set of positive real numbers.

How to make a bunch of numbers sum to 1?

We do this using normalization by z1∑i=1nzi\displaystyle \frac{z_1}{\sum^n_{i=1} z_i}.

softmax⁡dog⁡(fdog⁡(x),fcat(x))=exp⁡(fdog⁡(x))exp⁡(fdog⁡(x))+exp⁡(fcat(x))\displaystyle \operatorname{softmax}_{\operatorname{dog}}\left(f_{\operatorname{dog}}(x), f_{\mathrm{cat}}(x)\right)=\frac{\exp \left(f_{\operatorname{dog}}(x)\right)}{\exp \left(f_{\operatorname{dog}}(x)\right)+\exp \left(f_{\mathrm{cat}}(x)\right)}

For NN possible labels, p(y∣x)p(y \mid x) is a vector of NN elements, and fθ(x)f_\theta (x) is a vector -valued function with NN outputs.

p(y=i∣x)=softmax⁡(fθ(x))[i]=exp⁡(fθ,i(x))∑j=1Nexp⁡(fθ,j(x))\displaystyle p(y=i \mid x)=\operatorname{softmax}\left(f_{\theta}(x)\right)[i]=\frac{\exp \left(f_{\theta, i}(x)\right)}{\sum_{j=1}^{N} \exp \left(f_{\theta, j}(x)\right)}

Referenced in

Supervised Learning

In Supervised Learning, given \mathcal{D}=\left{\left(x_{1}, y_{1}\right),\left(x_{2}, y_{2}\right), \ldots,\left(x_{n}, y_{n}\right)\right}, the objective is to learn fθ(x)≈yf_{\theta}(x) \approx y.

Supervised Learning