Lecture 12 Introduction to Neural Networks

1 Lecture 12 Introduction to Neural Networks29 February 2016 Taylor B. ArnoldYale StatisticsSTAT 365/6651/37 Notes: Problem set 4 is due this Friday (SVM implementation) Problem set 5 will be posted prior to class tomorrow ( Neural net implementation )2/37mis classification Introducing Neural network architecture high level description of how to learn Neural Networks and specific challenges indoing so simulation example of SGD4/37 There are a large set of introductions to Neural Networks online. Popular ones that I likeare: Andrej Karpathy s Hacker s guide to Neural Networks : Andrej Karpathy s Lecture notes: Geoffrey E. Hinton, Yann LeCun, and Yoshua Bengio (video; NIPS 2015): Michael Nielsen s Neural Networks and Deep Learning: think these are all worthwhile, and approach the subject from slightly different anglesand with different learning outcomes.

2 I am going to (very) closely follow MichaelNielsen s notes for the next two lectures, as I think they work the best in Lecture formatand for the purposes of this course. We will then switch gears and start followingKarpathy s Lecture notes in the following simple decisionSay you want to decide whether you are going to attend a cheese festival this upcomingweekend. There are three variables that go into your the weather good? your friend want to go with you? it near public transportation?We ll assume that answers to these questions are the only factors that go into simple decision, will write the answers to these question as binary variablesxi, with zero being theanswer no and one being the answer yes the weather good?

3 Your friend want to go with you? it near public transportation?x3 Now, what is an easy way to describe the decision statement resulting from these simple decision, could determine weightswiindicating how important each feature is to whether youwould like to attend. We can then see if:x1 w1+x2 w2+x3 w3 thresholdFor some pre-determined threshold. If this statement is true, we would attend thefestival, and otherwise we would simple decision, example, if we really hated bad weather but care less about going with our friend andpublic transit, we could pick the weights6, a threshold of5, this causes us to go if and only if the weather is happens if the threshold is decreased to3? What about if it is decreased to1?

4 9/37A simple decision, example, if we really hated bad weather but care less about going with our friend andpublic transit, we could pick the weights6, a threshold of5, this causes us to go if and only if the weather is happens if the threshold is decreased to3? What about if it is decreased to1?9/37A simple decision, example, if we really hated bad weather but care less about going with our friend andpublic transit, we could pick the weights6, a threshold of5, this causes us to go if and only if the weather is happens if the threshold is decreased to3? What about if it is decreased to1?9/37A simple decision, we define a new binary variableythat represents whether we go to the festival, we canwrite this variable as:y={0;x1 w1+x2 w2+x3 w3<threshold1;x1 w1+x2 w2+x3 w3 thresholdIs this starting to look familiar yet?}

5 10/37A simple decision, , if I rewrite this in terms of a dot product between the vector of of all binary inputs(x), a vector of weights (w), and change the threshold to the negative bias (b), we have:y={0;x w+b<01;x w+b 0So we are really just finding separating hyperplanes again, much as we did with logisticregression and support vector machines!11/37A perceptronWe can graphically represent this decision algorithm as an object that takes3binaryinputs and produces a single binary output:This object is called aperceptronwhen using the type of weighting scheme we network of perceptronsA perceptron takes a number of binary inputs and emits a binary output. Therefore it iseasy to build a network of such perceptrons, where the output from some perceptronsare used in the inputs of other perceptrons:Notice that some perceptrons seem to have multiple output arrows, even though we havedefined them as having only one output.}

6 This is only meant to indicate that a singleoutput is being sent to multiple new network of perceptrons, input and outputs are typically represented as their own neurons, with the otherneurons namedhidden layers14/37A network of perceptrons, biological interpretation of a perceptron is this: when it emits a1this is equivalentto firing an electrical pulse, and when it is0this is when it is not firing. The biasindicates how difficult it is for this particular node to send out a network of perceptrons, that the network of nodes I have shown only sends signals in one direction. Thisis called afeed-forward network. These are by far the most well-studied types ofnetworks, though we will (hopefully) have a chance to talk about recurrent neuralnetworks (RNNs) that allow for loops in the network.

7 The one-directional nature offeed-forward Networks is probably the biggest difference between artificial neuralnetworks and their biological neuronAn important shortcoming of a perceptron is that a small change in the input values cancause a large change the output because each node (or neuron) only has two possiblestates:0or1. A better solution would be to output a continuum of values, say anynumber tutorials spend a significant amount of time describing the conceptual leap frombinary outputs to a continuous output. For us, however this should be neuron, one option, we could simply have the neuron emit the value: (x w+b) =11 +e (x w+b)For a particularly positive or negative value ofx w+b, the result will be nearly the sameas with the perceptron ( , near0or1).

8 For values close to the boundary of theseparating hyperplane, values near0:5will be neuron, perfectly mimics logistic regression, and in fact uses the logit function to do so. Inthe Neural network literature, the logit function is called thesigmoidfunction, thusleading to the namesigmoid neuronfor a neuron that uses it s that the previous restriction to binaryinputswas not at all needed, and can beeasily replaces with continuous input without an changes needed to the functionsIn the sigmoid neuron example, the choice of what function to use to go fromx w+btoan output is called theactivation function. Using a logistic, or sigmoid, activationfunction has some benefits in being able to easily take derivatives and the interpret themusing logistic choices have certain benefits that have recently grown in popularity.

9 Some ofthese tan:tanh(z) = 2 (2x) linear unit:ReLU(z) =max(0;z) rectified linear will explore the pros and cons of these in upcoming example23/37 MNIST example, classificationTo determine which class to put a particular input into, we look at which of the outputneurons have the largest Neural networksWe now have an architecture that describes a Neural network, but how do we learn theweights and bias terms in the model given a set of training data?As an important side note, notice that with just one node, we could define a learningalgorithm which perfectly replicates a support vector machine or logistic functionThe primary set-up for learning Neural Networks is to define acost function(also knownas aloss function) that measures how well the network predicts outputs on the test goal is to then find a set of weights and biases that minimizes the function, example of a cost function is just squared error loss:C(w;b) =12n i(yi by(xi))2Or, for classification, the hinge loss:C(w.)

10 B) = i[1 yi by(xi)]+As with the activation functions, we ll explore the different cost functions over the thenext several problemHow does one actually do the optimization required in fitting Neural Networks ? Withvery few exceptions, every technique is somehow related togradient descent. That is, wecalculate the gradient function, move a small amount in the opposite direction of thegradient (because we are minimizing), and then recalculate the gradient on the new descentMathematically, we can describe these updates as:wk+1=wk wCbk+1=bk bCFor some value >0. This tuning parameter, as in gradient boosted trees, is called thelearning rate. Too low, and learning takes a very long time. Too small, and it is likely tohave trouble finding the true minimum (as it will keep overshooting it).