Natural Language Processing

1 Natural Language ProcessingJacob EisensteinNovember 13, 2018 ContentsContents1 PrefaceiBackground ..iHow to use this book ..ii1 Language Processing and its neighbors .. themes in Natural Language Processing .. and knowledge .. and learning .. , compositional, and distributional perspectives ..9I Learning112 Linear text bag of words .. ve Bayes .. and tokens .. hyperparameters .. learning .. perceptron .. functions and large-margin classification .. large margin classification .. *Derivation of the online support vector machine .. regression .. optimization .. optimization .. *Additional topics in classification .. selection by regularization.

2 Views of logistic regression .. of learning algorithms ..433 Nonlinear neural networks .. neural networks .. functions .. structure .. and loss functions .. and lookup layers .. neural networks .. and dropout .. *Learning theory .. neural networks ..624 Linguistic applications of and opinion analysis .. problems .. approaches to sentiment analysis .. sense disambiguation .. many word senses? .. sense disambiguation as classification .. decisions for text classification .. is a word? .. many words? .. or binary? .. classifiers .. , recall, andF-MEASURE.. metrics .. comparison and statistical significance .. *Multiple comparisons.

3 Datasets ..88 Jacob Eisenstein. Draft of November 13, as labels .. data ..885 Learning without learning .. clustering .. (EM) .. as an optimization algorithm .. many clusters? .. of expectation-maximization .. sense induction .. learning .. modeling .. learning .. learning .. algorithms .. adaptation .. domain adaptation .. domain adaptation .. *Other approaches to learning with latent variables .. learning .. 117II Sequences and trees1236 Language Language models .. and discounting .. and backoff .. *Interpolation .. *Kneser-Ney smoothing .. neural network Language models.

4 Through time .. recurrent neural networks .. Language models .. likelihood .. words .. 141 Under contract with MIT Press, shared under CC-BY-NC-ND Sequence labeling as classification .. labeling as structure prediction .. Viterbi algorithm .. features .. Markov Models .. sequence labeling with features .. perceptron .. support vector machines .. random fields .. sequence labeling .. neural networks .. models .. Neural Networks for Sequence Labeling .. *Unsupervised sequence labeling .. dynamical systems .. unsupervised learning methods .. notation and the generalized viterbi algorithm.

5 1728 Applications of sequence tagging .. part-of-speech tagging .. Attributes .. Entity Recognition .. switching .. acts .. 1879 Formal Language languages .. state acceptors .. as a regular Language .. finite state acceptors .. state transducers .. *Learning weighted finite state automata .. languages .. grammars .. 208 Jacob Eisenstein. Draft of November 13, Language syntax as a context-free Language .. phrase-structure grammar for English .. ambiguity .. *Mildly context-sensitive languages .. phenomena in Natural Language .. categorial grammar .. 22010 Context-free Deterministic bottom-up parsing.

6 Recovering the parse tree .. Non-binary productions .. Complexity .. Ambiguity .. Parser evaluation .. Local solutions .. Weighted Context-Free Grammars .. Parsing with weighted context-free grammars .. Probabilistic context-free grammars .. *Semiring weighted context-free grammars .. Learning weighted context-free grammars .. Probabilistic context-free grammars .. Feature-based parsing .. *Conditional random field parsing .. Neural context-free grammars .. Grammar refinement .. Parent annotations and other tree transformations .. Lexicalized context-free grammars .. *Refinement grammars.

7 Beyond context-free parsing .. Reranking .. Transition-based parsing .. 25111 Dependency Dependency grammar .. Heads and dependents .. Labeled dependencies .. Dependency subtrees and constituents .. Graph-based dependency parsing .. Graph-based parsing algorithms .. Computing scores for dependency arcs .. Learning .. 267 Under contract with MIT Press, shared under CC-BY-NC-ND Transition-based dependency parsing .. Transition systems for dependency parsing .. Scoring functions for transition-based parsers .. Learning to parse .. Applications .. 277 III Meaning28312 Logical Meaning and denotation.

8 Logical representations of meaning .. Propositional logic .. First-order logic .. Semantic parsing and the lambda calculus .. The lambda calculus .. Quantification .. Learning semantic parsers .. Learning from derivations .. Learning from logical forms .. Learning from denotations .. 30113 Predicate-argument Semantic roles .. VerbNet .. Proto-roles and PropBank .. FrameNet .. Semantic role labeling .. Semantic role labeling as classification .. Semantic role labeling as constrained optimization .. Neural semantic role labeling .. Abstract Meaning Representation .. AMR Parsing.

9 32114 Distributional and distributed The distributional hypothesis .. Design decisions for word representations .. Representation .. Context .. Estimation .. Latent semantic analysis .. 329 Jacob Eisenstein. Draft of November 13, Brown clusters .. Neural word embeddings .. Continuous bag-of-words (CBOW) .. Skipgrams .. Computational complexity .. Word embeddings as matrix factorization .. Evaluating word embeddings .. Intrinsic evaluations .. Extrinsic evaluations .. Fairness and bias .. Distributed representations beyond distributional statistics .. Word-internal structure .. Lexical semantic resources.

10 Distributed representations of multiword units .. Purely distributional methods .. Distributional-compositional hybrids .. Supervised compositional methods .. Hybrid distributed-symbolic representations .. 34615 Reference Forms of referring expressions .. Pronouns .. Proper Nouns .. Nominals .. Algorithms for coreference resolution .. Mention-pair models .. Mention-ranking models .. Transitive closure in mention-based models .. Entity-based models .. Representations for coreference resolution .. Features .. Distributed representations of mentions and entities .. Evaluating coreference resolution.