CS7015 (Deep Learning) : Lecture 2 - McCulloch Pitts ...

1 CS7015 (Deep Learning) : Lecture 2 McCulloch Pitts Neuron, thresholding Logic, Perceptrons, Perceptron LearningAlgorithm and Convergence, Multilayer Perceptrons (MLPs), RepresentationPower of MLPsMitesh M. KhapraDepartment of Computer Science and EngineeringIndian Institute of Technology MadrasModule : Biological Neurons1x1x2x3 yw1w2w3 Artificial Neuron The most fundamental unit of a deepneural network is called anartificialneuron Why is it called a neuron ? Where doesthe inspiration come from ? The inspiration comes from biology(more specifically, from thebrain) biological neurons = neural cells = neuralprocessing units We will first see what a biological neuronlooks like ..2x1x2x3 yw1w2w3 Artificial Neuron The most fundamental unit of a deepneural network is called anartificialneuron Why is it called a neuron ?

2 Where doesthe inspiration come from ? The inspiration comes from biology(more specifically, from thebrain) biological neurons = neural cells = neuralprocessing units We will first see what a biological neuronlooks like ..2x1x2x3 yw1w2w3 Artificial Neuron The most fundamental unit of a deepneural network is called anartificialneuron Why is it called a neuron ? Where doesthe inspiration come from ? The inspiration comes from biology(more specifically, from thebrain) biological neurons = neural cells = neuralprocessing units We will first see what a biological neuronlooks like ..2x1x2x3 yw1w2w3 Artificial Neuron The most fundamental unit of a deepneural network is called anartificialneuron Why is it called a neuron ? Where doesthe inspiration come from ? The inspiration comes from biology(more specifically, from thebrain) biological neurons = neural cells = neuralprocessing units We will first see what a biological neuronlooks like.

3 2x1x2x3 yw1w2w3 Artificial Neuron The most fundamental unit of a deepneural network is called anartificialneuron Why is it called a neuron ? Where doesthe inspiration come from ? The inspiration comes from biology(more specifically, from thebrain) biological neurons = neural cells = neuralprocessing units We will first see what a biological neuronlooks like ..2 Biological Neurons Image adapted ,25 dendrite:receives signals from otherneurons synapse:point of connection to otherneurons soma:processes the information axon:transmits the output of thisneuron3 Biological Neurons Image adapted ,25 dendrite:receives signals from otherneurons synapse:point of connection to otherneurons soma:processes the information axon:transmits the output of thisneuron3 Biological Neurons Image adapted ,25 dendrite:receives signals from otherneurons synapse:point of connection to otherneurons soma:processes the information axon:transmits the output of thisneuron3 Biological Neurons Image adapted ,25 dendrite:receives signals from otherneurons synapse:point of connection to otherneurons soma:processes the information axon.

4 Transmits the output of thisneuron3 Biological Neurons Image adapted ,25 dendrite:receives signals from otherneurons synapse:point of connection to otherneurons soma:processes the information axon:transmits the output of thisneuron3 Let us see a very cartoonish illustrationof how a neuron works Our sense organs interact with the out-side world They relay information to the neurons The neurons (may) get activated and pro-duces a response (laughter in this case)4 Let us see a very cartoonish illustrationof how a neuron works Our sense organs interact with the out-side world They relay information to the neurons The neurons (may) get activated and pro-duces a response (laughter in this case)4 Let us see a very cartoonish illustrationof how a neuron works Our sense organs interact with the out-side world They relay information to the neurons The neurons (may) get activated and pro-duces a response (laughter in this case)

5 4 Let us see a very cartoonish illustrationof how a neuron works Our sense organs interact with the out-side world They relay information to the neurons The neurons (may) get activated and pro-duces a response (laughter in this case)4 Of course, in reality, it is not just a single neuronwhich does all this There is a massively parallel interconnected net-work of neurons The sense organs relay information to the lowestlayer of neurons Some of these neurons may fire (in red) in re-sponsetothisinformationandinturnrelay inform-ation to other neurons they are connected to These neurons may also fire (again, in red) andthe process continueseventually resulting in a re-sponse (laughter in this case) An average human brain has around1011(100 bil-lion) neurons!5 Of course, in reality, it is not just a single neuronwhich does all this There is a massively parallel interconnected net-work of neurons The sense organs relay information to the lowestlayer of neurons Some of these neurons may fire (in red) in re-sponsetothisinformationandinturnrelay inform-ation to other neurons they are connected to These neurons may also fire (again, in red) andthe process continueseventually resulting in a re-sponse (laughter in this case) An average human brain has around1011(100 bil-lion) neurons!

6 5 Of course, in reality, it is not just a single neuronwhich does all this There is a massively parallel interconnected net-work of neurons The sense organs relay information to the lowestlayer of neurons Some of these neurons may fire (in red) in re-sponsetothisinformationandinturnrelay inform-ation to other neurons they are connected to These neurons may also fire (again, in red) andthe process continueseventually resulting in a re-sponse (laughter in this case) An average human brain has around1011(100 bil-lion) neurons!5 Of course, in reality, it is not just a single neuronwhich does all this There is a massively parallel interconnected net-work of neurons The sense organs relay information to the lowestlayer of neurons Some of these neurons may fire (in red) in re-sponsetothisinformationandinturnrelay inform-ation to other neurons they are connected to These neurons may also fire (again, in red) andthe process continueseventually resulting in a re-sponse (laughter in this case) An average human brain has around1011(100 bil-lion) neurons!

7 5 Of course, in reality, it is not just a single neuronwhich does all this There is a massively parallel interconnected net-work of neurons The sense organs relay information to the lowestlayer of neurons Some of these neurons may fire (in red) in re-sponsetothisinformationandinturnrelay inform-ation to other neurons they are connected to These neurons may also fire (again, in red) andthe process continueseventually resulting in a re-sponse (laughter in this case) An average human brain has around1011(100 bil-lion) neurons!5 Of course, in reality, it is not just a single neuronwhich does all this There is a massively parallel interconnected net-work of neurons The sense organs relay information to the lowestlayer of neurons Some of these neurons may fire (in red) in re-sponsetothisinformationandinturnrelay inform-ation to other neurons they are connected to These neurons may also fire (again, in red) andthe process continueseventually resulting in a re-sponse (laughter in this case) An average human brain has around1011(100 bil-lion) neurons!

8 5 Of course, in reality, it is not just a single neuronwhich does all this There is a massively parallel interconnected net-work of neurons The sense organs relay information to the lowestlayer of neurons Some of these neurons may fire (in red) in re-sponsetothisinformationandinturnrelay inform-ation to other neurons they are connected to These neurons may also fire (again, in red) andthe process continueseventually resulting in a re-sponse (laughter in this case) An average human brain has around1011(100 bil-lion) neurons!5 This massively parallel network also ensures thatthere is division of work Each neuron may perform a certain role or respondto a certain stimulus6 This massively parallel network also ensures thatthere is division of work Each neuron may perform a certain role or respondto a certain stimulus6A simplified illustration This massively parallel network also ensures thatthere is division of work Each neuron may perform a certain role or respondto a certain stimulus6A simplified illustration This massively parallel network also ensures thatthere is division of work Each neuron may perform a certain role or respondto a certain stimulus6A simplified illustration This massively parallel network also ensures thatthere is division of work Each neuron may perform a certain

9 Role or respondto a certain stimulus6A simplified illustration This massively parallel network also ensures thatthere is division of work Each neuron may perform a certain role or respondto a certain stimulus6A simplified illustration This massively parallel network also ensures thatthere is division of work Each neuron may perform a certain role or respondto a certain stimulus6 The neurons in the brain are arranged ina hierarchy We illustrate this with the help of visualcortex (part of the brain) which dealswith processing visual information Starting from the retina, the informationis relayed to several layers (follow the ar-rows) We observe that the layersV1,V2toAITform a hierarchy (from identifyingsimple visual forms to high level objects)7 The neurons in the brain are arranged ina hierarchy We illustrate this with the help of visualcortex (part of the brain) which dealswith processing visual information Starting from the retina, the informationis relayed to several layers (follow the ar-rows) We observe that the layersV1,V2toAITform a hierarchy (from identifyingsimple visual forms to high level objects)7 The neurons in the brain are arranged ina hierarchy We illustrate this with the help of visualcortex (part of the brain) which dealswith processing visual information Starting from the retina, the informationis relayed to several layers (follow the ar-rows) We observe that the layersV1,V2toAITform a hierarchy (from identifyingsimple visual forms to high level objects)

10 7 The neurons in the brain are arranged ina hierarchy We illustrate this with the help of visualcortex (part of the brain) which dealswith processing visual information Starting from the retina, the informationis relayed to several layers (follow the ar-rows) We observe that the layersV1,V2toAITform a hierarchy (from identifyingsimple visual forms to high level objects)7 Sample illustration of hierarchicalprocessing Idea borrowed from Hugo Larochelle s Lecture slides8 Sample illustration of hierarchicalprocessing Idea borrowed from Hugo Larochelle s Lecture slides8 Sample illustration of hierarchicalprocessing Idea borrowed from Hugo Larochelle s Lecture slides8 Disclaimer I understand very little about how the brain works! What you saw so far is an overly simplified explanation of how the brain works!