Neural Networks - D. Kriesel

1 A Brief Introduction to Neural Networks David Kriesel Download location: for the programmers: Scalable and efficient NN framework, written in JAVA remembrance ofDr. Peter Kemp, Notary (ret.), Bonn, Kriesel A Brief Introduction to Neural Networks (ZETA2-EN)iiiA small preface"Originally, this work has been prepared in the framework of a seminar of theUniversity of Bonn in Germany, but it has been and will be extended (afterbeing presented and published online ). First and foremost, to provide a comprehensive overview of thesubject of Neural Networks and, second, just to acquire more and moreknowledge about LATEX . And who knows maybe one day this summary willbecome a real preface!"Abstract of this work, end of 2005 The above abstract has not yet become apreface but at least alittle preface, eversince the extended text (then 40 pageslong) has turned out to be a and intention of thismanuscriptThe entire text is written and laid outmore effectively and with more illustra-tions than before.

2 I did all the illustra-tions myself, most of them directly inLATEX by using XYpic. They reflect whatI would have liked to see when becomingacquainted with the subject: Text and il-lustrations should be memorable and easyto understand to offer as many people aspossible access to the field of Neural , the mathematically and for-mally skilled readers will be able to under-stand the definitions without reading therunning text, while the opposite holds forreaders only interested in the subject mat-ter; everything is explained in both collo-quial and formal language. Please let meknow if you find out that I have violatedthis sections of this text are mostlyindependent from each otherThe document itself is divided into differ-ent parts, which are again divided intochapters. Although the chapters containcross-references, they are also individuallyaccessible to readers with little previousknowledge.

3 There are larger and smallerchapters: While the larger chapters shouldprovide profound insight into a paradigmof Neural Networks ( the classic neuralnetwork structure: theperceptronand itslearning procedures), the smaller chaptersgive a short overview but this is also in the introduction of each addition to all the definitions and expla-nations I have included some excursusesto provide interesting information not di-rectly related to the , I was not able to find freeGerman sources that are multi-facetedin respect of content (concerning theparadigms of Neural Networks ) and, nev-ertheless, written in coherent style. Theaim of this work is (even if it could notbe fulfilled at first go) to close this gap bitby bit and to provide easy access to to learn not only byreading, but also by coding?Use SNIPE!SNIPE1is a well-documented JAVA li-brary that implements a framework forneural Networks in a speedy, feature-richand usable way.

4 It is available at nocost for non-commercial purposes. It wasoriginally designed for high performancesimulations with lots and lots of neuralnetworks (even large ones) being trainedsimultaneously. Recently, I decided togive it away as a professional reference im-plementation that covers network aspectshandled within this work, while at thesame time being faster and more efficientthan lots of other implementations due to1 Scalable and Generalized Neural Information Pro-cessing Engine, downloadable , online JavaDoc original high-performance simulationdesign goal. Those of you who are up forlearning by doing and/or have to use afast and stable Neural Networks implemen-tation for some reasons, should definetelyhave a look at , the aspects covered by Snipe arenot entirely congruent with those coveredby this manuscript. Some of the kindsof Neural Networks are not supported bySnipe, while when it comes to other kindsof Neural Networks , Snipe may have lotsand lots more capabilities than may everbe covered in the manuscript in the formof practical hints.

5 Anyway, in my experi-ence almost all of the implementation re-quirements of my readers are covered the Snipe download page, look for thesection "Getting started with Snipe" youwill find an easy step-by-step guide con-cerning Snipe and its documentation, aswell as some :This manuscript frequently incor-porates Snipe. Shaded Snipe-paragraphslike this one are scattered among largeparts of the manuscript, providing infor-mation on how to implement their con-text in also implies thatthose who do not want to use Snipe,just have to skip the shaded Snipe-paragraphs!The Snipe-paragraphs as-sume the reader has had a close look atthe "Getting started with Snipe" , class names are used. As Snipe con-sists of only a few different packages, I omit-ted the package names within the qualifiedclass names for the sake of Kriesel A Brief Introduction to Neural Networks (ZETA2-EN) s easy to print thismanuscriptThis text is completely illustrated incolor, but it can also be printed as is inmonochrome: The colors of figures, tablesand text are well-chosen so that in addi-tion to an appealing design the colors arestill easy to distinguish when printed are many tools directlyintegrated into the textDifferent aids are directly integrated in thedocument to make reading more flexible:However, anyone (like me) who prefersreading words on paper rather than onscreen can also enjoy some the table of contents, differenttypes of chapters are markedDifferent types of chapters are directlymarked within the table of contents.

6 Chap-ters, that are marked as "fundamental"are definitely ones to read because almostall subsequent chapters heavily depend onthem. Other chapters additionally dependon information given in other (preceding)chapters, which then is marked in the ta-ble of contents, headlines throughout thetext, short ones in the table ofcontentsThe whole manuscript is now pervaded bysuch headlines. Speaking headlines arenot just title-like ("Reinforcement Learn-ing"), but centralize the information givenin the associated section to a single sen-tence. In the named instance, an appro-priate headline would be "Reinforcementlearning methods provide feedback to thenetwork, whether it behaves good or bad".However, such long headlines would bloatthe table of contents in an unacceptableway. So I used short titles like the first onein the table of contents, and speaking ones,like the latter, throughout the notes are a navigationalaidThe entire document contains marginalnotes in colloquial language (see the exam-Hypertexton paper:-)ple in the margin), allowing you to "scan"the document quickly to find a certain pas-sage in the text (including the titles).

7 New mathematical symbols are marked byspecific marginal notes for easy findingJx(see the example forxin the margin).There are several kinds of indexingThis document contains different types ofindexing: If you have found a word inthe index and opened the correspondingpage, you can easily find it by searchingD. Kriesel A Brief Introduction to Neural Networks (ZETA2-EN) text all indexed wordsare highlighted like symbols appearing in sev-eral chapters of this document ( foran output neuron; I tried to maintain aconsistent nomenclature for regularly re-curring elements) are separately indexedunder "Mathematical Symbols", so theycan easily be assigned to the correspond-ing of persons written insmall capsare indexed in the category "Persons" andordered by the last of use and licenseBeginning with the epsilon edition, thetext is licensed under theCreative Com-mons Attribution-No Derivative Unported License2, except for somelittle portions of the work licensed undermore liberal licenses as mentioned (mainlysome figures from Wikimedia Commons).

8 A quick license are free to redistribute this docu-ment (even though it is a much betteridea to just distribute the URL of myhomepage, for it always contains themost recent version of the text). may not modify, transform, orbuild upon the document except forpersonal must maintain the author s attri-bution of the document at all may not use the attribution toimply that the author endorses youor your document I m no lawyer, the above bullet-pointsummary is just informational: if there isany conflict in interpretation between thesummary and the actual license, the actuallicense always takes precedence. Note thatthis license does not extend to the sourcefiles used to produce the document. Thoseare still to cite this manuscriptThere s no official publisher, so you needto be careful with your citation. Pleasefind more information in English andGerman language on my homepage, re-spectively the subpage concerning I would like to express my grati-tude to all the people who contributed, inwhatever manner, to the success of thiswork, since a work like this needs manyhelpers.

9 First of all, I want to thankthe proofreaders of this text, who helpedme and my readers very much. In al-phabetical order: Wolfgang Apolinarski,Kathrin Gr ve, Paul Imhoff, Thomas3 Kriesel A Brief Introduction to Neural Networks (ZETA2-EN) hn, Christoph Kunze, Malte Lohmeyer,Joachim Nock, Daniel Plohmann, DanielRosenthal, Christian Schulz and , I want to thank the readersDietmar Berger, Igor Buchm ller, MarieChrist, Julia Damaschek, Jochen D ll,Maximilian Ernestus, Hardy Falk, AnneFeldmeier, Sascha Fink, Andreas Fried-mann, Jan Gassen, Markus Gerhards, Se-bastian Hirsch, Andreas Hochrath, NicoH ft, Thomas Ihme, Boris Jentsch, TimHussein, Thilo Keller, Mario Krenn, MirkoKunze, Maikel Linke, Adam Maciak,Benjamin Meier, David M ller, AndreasM ller, Rainer Penninger, Lena Reichel,Alexander Schier, Matthias Siegmund,Mathias Tirtasana, Oliver Tischler, Max-imilian Voit, Igor Wall, Achim Weber,Frank Weinreis, Gideon Maillette de BuijWenniger, Philipp Woock and many oth-ers for their feedback, suggestions and , I d like to thank SebastianMerzbach.

10 Who examined this work in avery conscientious way finding inconsisten-cies and errors. In particular, he clearedlots and lots of language clumsiness fromthe English , I would like to thank BeateKuhl for translating the entire text fromGerman to English, and for her questionswhich made me think of changing thephrasing of some would particularly like to thank Eckmiller and Dr. Nils Goerke aswell as the entire Division of Neuroinfor-matics, Department of Computer Scienceof the University of Bonn they all madesure that I always learned (and also hadto learn) something new about Neural net-works and related subjects. Especially has always been willing to respondto any questions I was not able to answermyself during the writing process. Conver-sations with Prof. Eckmiller made me stepback from the whiteboard to get a betteroverall view on what I was doing and whatI should do , and not only in the context ofthis work, I want to thank my parents whonever get tired to buy me specialized andtherefore expensive books and who havealways supported me in my many "remarks" and the very specialand cordial atmosphere ;-) I want to thankAndreas Huber and Tobias Treutler.