Neural Network Structures - IEEE

1 3 Neural Network StructuresThis chapter describes various types of Neural Network Structures that are usefulfor RF and microwave applications. The most commonly used Neural networkconfigurations, known as multilayer perceptrons (MLP), are described first,together with the concept of basic backpropagation training, and the universalapproximation theorem. Other Structures discussed in this chapter includeradial basis function (RBF) Network , wavelet Neural Network , and self-organizingmaps (SOM). Brief reviews of arbitrary Structures for ANNs and recurrentneural networks are also IntroductionA Neural Network has at least two physical components, namely, the processingelements and the connections between them. The processing elements arecalled neurons, and the connections between the neurons are known as link has a weight parameter associated with it. Each neuron receivesstimulus from the neighboring neurons connected to it, processes the informa-tion, and produces an output.

2 Neurons that receive stimuli from outside thenetwork ( , not from neurons of the Network ) are called input whose outputs are used externally are called output neurons. Neuronsthat receive stimuli from other neurons and whose output is a stimulus forother neurons in the Neural Network are known as hidden neurons. There aredifferent ways in which information can be processed by a neuron, and differentways of connecting the neurons to one another. Different Neural networkstructures can be constructed by using different processing elements and bythe specific manner in which they are Networks for RF and Microwave DesignA variety of Neural Network Structures have been developed for signalprocessing, pattern recognition, control, and so on. In this chapter, we describeseveral Neural Network Structures that are commonly used for microwave model-ing and design [1, 2]. The Neural Network Structures covered in this chapterinclude multilayer perceptrons (MLP), radial basis function networks (RBF),wavelet Neural networks, arbitrary Structures , self-organizing maps (SOM), andrecurrent NotationLetnandmrepresent the number of input and output neurons of the neuralnetwork.

3 Letxbe ann-vector containing the external inputs (stimuli) to theneural Network ,ybe anm-vector containing the outputs from the outputneurons, andwbe a vector containing all the weight parameters representingthe connections in the Neural Network . The functiony=y(x,w) mathemati-cally represents a Neural Network . The definition ofwand the manner in whichyis computed fromxandw, determine the structure of the Neural illustrate the notation, we consider the Neural Network model of anFET shown in Figure The inputs and outputs of the FET Neural modelare given by,x=[LWaNdVGSVDS freq]T( )y=[MS11PS11MS12PS12MS21PS21MS22PS22]T( )wherefreqis frequency, andMSijandPSijrepresent the magnitude andphase of the S-parameterSij. The input vectorxcontains physical/process/Figure physics-based Network Structuresbias parameters of the FET. The original physics-based FET problem can beexpressed asy=f(x)( )The Neural Network model for the problem isy=y(x,w)( ) of the Neural Network Modeling ApproachIn the FET example above, the Neural Network will represent the FET behavioronly after learning the originalx yrelationship through a process calledtraining.

4 Samples of (x,y) data, calledtraining data, should first be generatedfrom original device physics simulators or from device measurements. Trainingis done to determine Neural Network internal weightswsuch that the neuralmodel output best matches the training data. A trained Neural Network modelcan then be used during microwave design providing answers to the task itlearned. In the FET example, the trained model can be used to provideS-parameters from device physical/geometrical and bias values during further highlight features of the Neural Network modeling approach,we contrast it with two broad types of conventional microwave modelingapproaches. The first type is the detailed modeling approach such as EM-based models for passive components and physics-based models for activecomponents. The overall model, ideally, is defined by well-established theoryand no experimental data is needed for model determination. However, suchdetailed models are usually computationally expensive.

5 The second type ofconventional modeling uses empirical or equivalent circuit-based models forpassive and active components. The models are typically developed using amixture of simplified component theory, heuristic interpretation and represen-tations, and fitting of experimental data. The evaluation of these models isusually much faster than that of the detailed models. However, the empiricaland equivalent circuit models are often developed under certain assumptionsin theory, range of parameters, or type of components. The models have limitedaccuracy especially when used beyond original assumptions. The Neural networkapproach is a new type of modeling approach where the model can be developedby learning from accurate data of the original component. After training, theneural Network becomes a fast and accurate model of the original problem itlearned. A summary of these aspects is given in Table in-depth description of Neural Network training, its applications inmodeling passive and active components and in circuit optimization will be64 Neural Networks for RF and Microwave DesignTable Comparison of Modeling Approaches for RF/Microwave ApplicationsEmpirical andBasis forEquivalent CircuitPure Neural NetworkComparisonEM/Physics ModelsModelsModelsSpeedSlowFastFastAccur acyHighLimitedCould be close toEM/physics modelsNumber of training0A fewSufficient trainingdatadata is required, whichcould be large forhigh-dimensionalproblemsCircuit/EM theoryMaxwell, orPartially involvedNot involvedof the problemsemiconductorequationsdescribed in subsequent chapters.

6 In the present chapter, we describe structuresof Neural networks, that is, the various ways of realizingy=y(x,w). Thestructural issues have an impact on model accuracy and cost of model multilayer Perceptrons (MLP) multilayer perceptrons (MLP) are the most popular type of Neural networksin use today. They belong to a general class of Structures called feedforwardneural networks, a basic type of Neural Network capable of approximatinggeneric classes of functions, including continuous and integrable functions [3].MLP Neural networks have been used in a variety of microwave modeling andoptimization StructureIn the MLP structure , the neurons are grouped into layers. The first and lastlayers are called input and output layers respectively, because they representinputs and outputs of the overall Network . The remaining layers are calledhidden layers. Typically, an MLP Neural Network consists of an input layer,one or more hidden layers, and an output layer, as shown in Figure Network StructuresFigure perceptrons (MLP) the total number of layers isL.

7 The1st layer is the input layer, theLth layer is the output layer, and layers 2 toL 1 are hidden layers. Let thenumber of neurons inlth layer beNl,l=1,2,.., the weight of the link betweenjth neuron ofl 1thlayer andith neuron oflth layer, 1 j Nl 1,1 i Nl. Letxirepresenttheith external input to the MLP, andzlibe the output ofith neuron oflthlayer. We introduce an extra weight parameter for each neuron,wli0, represent-ing the bias forith neuron oflth layer. As such,wof MLP includeswlij,j=0,1,..,Nl 1,i=1,2,..,Nl,l=2,3,..,L, that is,w=[ ,wLNLNL 1]T( ) Processing by a NeuronIn a Neural Network , each neuron with the exception of neurons at theinput layer receives and processes stimuli (inputs) from other neurons. The66 Neural Networks for RF and Microwave Designprocessed information is available at the output end of the neuron. Figure the way in which each neuron in an MLP processes the an example, a neuron of thelth layer receives stimuli from the neurons ofl 1th layer, that is,zl 11,zl 12.

8 ,zl 1Nl 1. Each input is first multiplied bythe corresponding weight parameter, and the resulting products are added toproduce a weighted sumg. This weighted sum is passed through a neuronactivation functions(?) to produce the final output of the neuron. This outputzlican, in turn, become the stimulus for neurons in the next FunctionsThe most commonly-used hidden neuron activation function is the sigmoidfunction given bys(g)=1(1 +e g)( )As shown in Figure , the sigmoid function is a smooth switch functionhaving the property ofs(g) H1asg + 0asg Figure processing byith neuron oflth Network StructuresFigure possible hidden neuron activation functions are the arc-tangentfunction shown in Figure and given bys(g)=S2pDarctan(g)( )and the hyperbolic-tangent function shown in Figure and given bys(g)=(eg e g)(eg+e g)( )All these logistic functions are bounded, continuous, monotonic, andcontinuously Networks for RF and Microwave DesignFigure input neurons simply relay the external stimuli to the hidden layerneurons; that is, the input neuron activation function is a relay function,z1i=xi,i=1,2.

9 ,n, andn=N1. As such, some researchers only countthe hidden and output layer neurons as part of the MLP. In this book, wefollow a convention, wherein the input layer neurons are also considered aspart of the overall structure . The activation functions for output neurons caneither be logistic functions ( , sigmoid), or simple linear functions thatcompute the weighted sum of the stimuli. For RF and microwave modelingproblems, where the purpose is to model continuous electrical parameters,linear activation functions are more suitable for output neurons. The linearactivation function is defined ass(g)=g= NL 1j=0wLijzL 1j( )The use of linear activation functions in the output neurons could helpto improve the numerical conditioning of the Neural Network training processdescribed in Chapter of BiasThe weighted sum expressed asgli=wli1zl 11+wli2zl 12+..+wliNl 1zl 1Nl 1( )69 Neural Network Structuresis zero, if all the previous hidden layer neuron responses (outputs)zl 11,zl 12.

10 ,zl 1Nl 1are zero. In order to create a bias, we assume a fictitiousneuron whose output iszl 10= 1( )and add a weight parameterwl 1i0called bias. The weighted sum can then bewritten asgli= Nl 1j=0wlijzl 1j( )The effect of adding the bias is that the weighted sum is equal to thebias when all the previous hidden layer neuron responses are zero, that is,gli=wli0,ifzl 11=zl 12=..=zl 1Nl 1= 0( )The parameterwli0is the bias value forith neuron inlth layer as shownin Figure typicalith hidden neuron oflth layer with an additional weight parametercalled Networks for RF and Microwave Network FeedforwardGiven the inputsx=[ ]Tand the weightsw, Neural networkfeedforward is used to compute the outputsy=[ ]Tfrom a MLPneural Network . In the feedforward process, the external inputs are first fed tothe input neurons (1st layer), the outputs from the input neurons are fed tothe hidden neurons of the2nd layer, and so on, and finally the outputs ofL 1th layer are fed to the output neurons (Lth layer).