Derivative of sigmoid function $\sigma (x) = \frac{1}{1+e^{-x}}$

In my AI textbook there is this paragraph, without any explanation.

The sigmoid function is defined as follows

$$\sigma (x) = \frac{1}{1+e^{-x}}.$$

This function is easy to differentiate because

$$\frac{d\sigma (x)}{d(x)} = \sigma (x)\cdot (1-\sigma(x)).$$

It has been a long time since I’ve taken differential equations, so could anyone tell me how they got from the first equation to the second?

Answer

Let’s denote the sigmoid function as $\sigma(x) = \dfrac{1}{1 + e^{-x}}$.

The derivative of the sigmoid is $\dfrac{d}{dx}\sigma(x) = \sigma(x)(1 – \sigma(x))$.

Here’s a detailed derivation:

$$
\begin{align}
\dfrac{d}{dx} \sigma(x) &= \dfrac{d}{dx} \left[ \dfrac{1}{1 + e^{-x}} \right] \\
&= \dfrac{d}{dx} \left( 1 + \mathrm{e}^{-x} \right)^{-1} \\
&= -(1 + e^{-x})^{-2}(-e^{-x}) \\
&= \dfrac{e^{-x}}{\left(1 + e^{-x}\right)^2} \\
&= \dfrac{1}{1 + e^{-x}\ } \cdot \dfrac{e^{-x}}{1 + e^{-x}} \\
&= \dfrac{1}{1 + e^{-x}\ } \cdot \dfrac{(1 + e^{-x}) – 1}{1 + e^{-x}} \\
&= \dfrac{1}{1 + e^{-x}\ } \cdot \left( \dfrac{1 + e^{-x}}{1 + e^{-x}} – \dfrac{1}{1 + e^{-x}} \right) \\
&= \dfrac{1}{1 + e^{-x}\ } \cdot \left( 1 – \dfrac{1}{1 + e^{-x}} \right) \\
&= \sigma(x) \cdot (1 – \sigma(x))
\end{align}
$$

Attribution
Source : Link , Question Author : Bryan Glazer , Answer Author : Michael Percy

Leave a Comment