Machine Learning Garden

Softmax

References

Tags: concept

Sources:

Related notes:

Updates:

Notes {{word-count}}

Summary:

Key points:

How to make a number $z$ postive?

$\exp(z)$ is especially convenient because it is one to one and onto and it maps the entire real number line to an entire set of positive real numbers.

How to make a bunch of numbers sum to 1?

We do this using normalization by $\displaystyle \frac{z_1}{\sum^n_{i=1} z_i}$ .

$\displaystyle \operatorname{softmax}_{\operatorname{dog}}\left(f_{\operatorname{dog}}(x), f_{\mathrm{cat}}(x)\right)=\frac{\exp \left(f_{\operatorname{dog}}(x)\right)}{\exp \left(f_{\operatorname{dog}}(x)\right)+\exp \left(f_{\mathrm{cat}}(x)\right)}$

For $N$ possible labels, $p(y \mid x)$ is a vector of $N$ elements, and $f_\theta (x)$ is a vector -valued function with $N$ outputs.

$\displaystyle p(y=i \mid x)=\operatorname{softmax}\left(f_{\theta}(x)\right)[i]=\frac{\exp \left(f_{\theta, i}(x)\right)}{\sum_{j=1}^{N} \exp \left(f_{\theta, j}(x)\right)}$

Why is it called a Softmax?

Referenced in

Supervised Learning

Softmax

Machine Learning Concepts

Softmax

Why is it called a Softmax?

Softmax