13 The Hopfield Model - fu-berlin.de

1 13 The Hopfield ModelOne of the milestones for the current renaissance in the field of neural networkswas the associative Model proposed by Hopfield at the beginning of the s approach illustrates the way theoretical physicists like to thinkabout ensembles of computing units. No synchronization is required, eachunit behaving as a kind of elementary system in complex interaction with therest of the ensemble. An energy function must be introduced to harness thetheoretical complexities posed by such an approach. The next two sectionsdeal with the structure of Hopfield networks. We then proceed to show thatthe Model converges to a stable state and that two kinds of learning rules canbe used to find appropriate network Synchronous and asynchronous networksA relevant issue for the correct design of recurrent neural networks is the ad-equate synchronization of the computing elements.

2 In the case of McCulloch-Pitts networks we solved this difficulty by assuming that the activation of eachcomputing element consumes a unit of time. The network is built taking thisdelay into account and by arranging the elements and their connections in thenecessary pattern. When the arrangement becomes too contrived, additionalunits can be included which serve as delay elements. What happens whenthis assumption is lifted, that is, when the synchronization of the computingelements is eliminated? Recursive networkswith stochastic dynamicsWe discussed the design and operation of associative networks in the previouschapter. The synchronization of the output was achieved by requiring that allcomputing elements evaluate their inputs and compute their output simulta-neously.

3 Under this assumption the operation of the associative memory canR. Rojas: neural Networks, Springer-Verlag, Berlin, 1996R. Rojas: neural Networks, Springer-Verlag, Berlin, 1996R. Rojas: neural Networks, Springer-Verlag, Berlin, 199633813 The Hopfield Modelbe described with simple linear algebraic methods. The excitation of the out-put units is computed using vector-matrix multiplication and evaluating thesign function at each methods we have used before to avoid dealing explicitly with thesynchronization problem have the disadvantage, from the point of view of bothbiology and physics, that global information is needed, namely a global in conventional computers synchronization of the digital buildingblocks is achieved using a clock signal, there is no such global clock in biologicalsystems.

4 In a more biologically oriented simulation, global synchronizationshould thus be avoided. In this chapter we deal with the problem of identifyingthe properties of neural networks lacking global in which the computing units are activated at different timesand which provide a computation after a variable amount of time are stochas-tic automata. Networks built from this kind of units behave likestochasticdynamical The bidirectional associative memoryBefore we start analyzing asynchronous networks we will examine anotherkind of synchronous associative Model with bidirectional edges. We will arriveat the concept of theenergy functionin a very natural have already discussed recurrent associative networks in which theoutput of the network is fed back to the input units using additional feed-back connections (Figure ).

5 In this way we designed recurrent dynamicalsystems and tried to determine their fixpoints. However, there is another wayto define a recurrent associative memory made up of two layers which sendinformation recursively between them. The input layer contains units whichreceive the input to the network and send the result of their computationto the output layer. The output of the first layer is transported by bidirec-tional edges to the second layer of units, which then return the result of theircomputation back to the first layer using the same edges. As in the case ofassociative memory models, we can askwhether the network achieves a stablestate in which the information being sent back and forth does not change aftera few iterations [258]. Such a network (shown in Figure ) is known as aresonance network orbidirectional associative memory(BAM).

6 The activa-tion function of the units is the sign function and information is coded usingbipolar network in Figure maps ann-dimensional row vectorx0to ak-dimensional row kweight matrix of the networkbyWso that the mapping computed in the first step can be written asy0=sgn(x0W).In the feedback stepy0is treated as the input and the new computation isxT1=sgn(WyT0).R. Rojas: neural Networks, Springer-Verlag, Berlin, 1996R. Rojas: neural Networks, Springer-Verlag, Berlin, 1996R. Rojas: neural Networks, Springer-Verlag, Berlin, Synchronous and asynchronous of a resonance network (BAM)A new computation from left to right producesy1=sgn(x1W).Aftermiterations the system has computed a set ofm+ 1 vector pairs(x0,y0),..,(xm,ym) which fulfill the conditionsyi=sgn(xiW)( )andxTi+1=sgn(WyTi).

7 ( )The question is whether after some iterations a fixpoint (x,y) is found. Thisis the case when bothy=sgn(xW)andxT=sgn(WyT)( )hold. The BAM is thus a generalization of a unidirectional associative input vector, the key , can be presented to the network from the left orfrom the right and, after some iterations, the BAM finds the correspondingcomplementary vector. As can be seen, no external feedbackconnections arenecessary. The same edges are used for the transmission of information backand can be immediately deduced from ( ) that if a vector pair (x,y)isgiven and we want to condition a BAM to accept this pair as a fixed point,Hebbian learning can be used to compute an adequate asW=xTy, as prescribed by Hebbian learning, theny=sgn(xW)=sgn(xxTy)=sgn( x 2y)=yand alsoxT=sgn(WyT)=sgn(xTyyT)=sgn(xT y 2)= Rojas: neural Networks, Springer-Verlag, Berlin, 1996R.

8 Rojas: neural Networks, Springer-Verlag, Berlin, 1996R. Rojas: neural Networks, Springer-Verlag, Berlin, 199634013 The Hopfield ModelIf we want to store several vector pairs (x1,y1),..,(xm,ym)inaBAM,thenHebbian learning works better if the vectorsx1,..,xmandy1,..,ymarepairwise orthogonal within their respective groups, because in that case theperturbation term becomes negligible (refer to Chap. 12).For a set ofmvector pairs the matrixWis set toW=xT1y1+xT2y2+ + can be used to build autoassociative networks because the matricesproduced by the Hebb rule or by computing the pseudoinverse are see this, defineXas the matrix, each of whosemrows is ann-dimensionalvector, so that ifWdenotes the connection matrix of an autoassociativememory for thosemvectors, then it is true thatX=XWandXT=WXT,becauseWis symmetric.

9 This is just another way of writing the type ofcomputation performed by a The energy functionWith the BAM we can motivate and explore the concept of anenergy functionin a simple setting. Assume that a BAM is given for which the vector pair(x,y) is a stable state. If the initial vector presented to the network fromthe left isx0, the network will converge to (x,y) after some iterations. Thevectory0is computed according toy0=sgn(x0W). Ify0is now used for anew iteration from the right, excitation of the units in the left layer can besummarized in an excitation vectorecomputed according toeT= vector pair (x0,y0) is a stable state of the network if sgn(e)= enough tox0fulfill this condition. These vectors differ fromx0by a small angle and therefore the productx0eTis larger than for othervectors of the same length but further away fromx0.

10 The productE= x0eT= x0 WyT0is therefore smaller (because of the minus sign) if the vectorWyT0lies closertox0. The scalar valueEcan be used as a kind of index of convergence tothe stable states of an associative memory. We callEtheenergy functionofthe energy functionEof a BAM with weight matrixW,inwhich the outputyiof the right layer of units is computed in thei-th iterationaccording to equation( )and the outputxiof the left layer is computedaccording to( )is given byE(xi,yi)= 12xiWyTi.( )R. Rojas: neural Networks, Springer-Verlag, Berlin, 1996R. Rojas: neural Networks, Springer-Verlag, Berlin, 1996R. Rojas: neural Networks, Springer-Verlag, Berlin, Definition of Hopfield networks341 The factor 1/2 will be useful later and is just a scaling constant for theenergy function.