Softmax Function cheatsheet

• It is used to convert a real number to a number between 0 and 1.
• It is used in classification problems.
• https://en.wikipedia.org/wiki/Softmax_function Wikipedia

• Neural Networks
• Deep Learning

$$\sigma(x) = \frac{e^{{x}}{\sum_{i=1}}e^{x_{i}}}$$

• x is the input.

• The softmax function is always positive.
• The softmax function is always less than or equal to 1.
• The softmax function is equal to 0 when the input is less than 0.
• The softmax function is equal to the input when the input is greater than or equal to 0.

• Advantages

  • The softmax function is differentiable.
  • The softmax function is monotonic.
  • The softmax function is bounded.

• Disadvantages ## looks into this

  • The softmax function is not differentiable at 0.
  • The softmax function is not differentiable at negative values.
  • The softmax function is not bounded above.
  • The softmax function is not bounded below.
  • The softmax function is not symmetric.
  • The softmax function is not centered around 0.

import numpy as np

def softmax(x):
    return np.exp(x) / np.sum(np.exp(x), axis=0)

softmax <- function(x) {
    exp(x) / sum(exp(x))
}

function softmax(x)
    exp.(x) ./ sum(exp.(x))
end

import tensorflow as tf

def softmax(x):
    return tf.nn.softmax(x)

import torch.nn.functional as F

def softmax(x):
    return F.softmax(x, dim=0)