Figure 6
The graph plots f of x on the vertical axis and x on the horizontal axis from minus 5 to 5. A solid curve represents Softplus of x, which increases smoothly and rises steeply for positive x. A dashed curve shows the first derivative of Softplus of x, which increases from near 0 and approaches 1. A dotted curve shows the second derivative of Softplus of x, which forms a peak near x equals 0 and decreases on both sides. A dash dot curve shows the third derivative of Softplus of x, which crosses 0 and changes sign around x equals 0. A straight line represents Re L U of x, which is 0 for negative x and increases linearly for positive x.Softplus activation function and its 1st, 2nd and 3rd derivatives, alongside ReLU activation function