Notes on Backpropagation

← Back to document page

Notes on BackpropagationPeter SadowskiDepartment of Computer ScienceUniversity of California IrvineIrvine, CA document derives Backpropagation for some common neural Cross Entropy Error with Logistic ActivationIn a classification task with two classes, it is standard to use a neural network architecture witha single logistic output unit and the cross-entropy loss function (as opposed to, for example, thesum-of-squared loss function). With this combination, the output prediction is always between zeroand one, and is interpreted as a probability. Training corresponds to maximizing the conditionallog-likelihood of the data, and as we will see, the gradient calculation simplifies nicely with can generalize this slightly to the case where we have multiple, independent, two-class classi-fication tasks.

@s i @s i @w ji (4) Examining each factor in turn, @E @y i = t i y i + 1 t i 1 y i; (5) = y i t i y i(1 y i); (6) @y i @s i = y i(1 y i) (7) @s i @w ji = h j (8) where x j is the activation of the j node in the hidden layer. Combining things back together, @E @s i = y i t i (9) and @E @w ji = (y i t i)h j (10). The above gives us the gradients ...

  Notes, Back, Notes on backpropagation, Backpropagation