References
Tags: concept
Sources:
Related notes:
Negative Log-Likelihood (NLL)
Updates:
April 20th, 2021: created note.
Notes {{word-count}}
Summary:
Key points:
Cross-Entropy measures how similar two distributions pθp_\thetapθ and ppp are.
H(p,pθ)=−∑yp(y∣xi)logpθ(y∣xi)H\left(p, p_{\theta}\right)=-\sum_{y} p\left(y \mid x_{i}\right) \log p_{\theta}\left(y \mid x_{i}\right)H(p,pθ)=−∑yp(y∣xi)logpθ(y∣xi)
If we assume yi∼p(y∣xi)y_{i} \sim p\left(y \mid x_{i}\right)yi∼p(y∣xi), meaning the label is sampled from the true distribution, then H(p,pθ)≈−logpθ(yi∣xi)H\left(p, p_{\theta}\right) \approx-\log p_{\theta}\left(y_{i} \mid x_{i}\right)H(p,pθ)≈−logpθ(yi∣xi).
This is also called Cross-Entropy.
Cross-Entropy
Negative Log-Likelihood (NLL) is sometimes also called as Cross-Entropy.