CHAPTER Regular Expressions, Text Normalization, Edit …

2 CHAPTER 2•REGULAR EXPRESSIONS, TEXT NORMALIZATION, EDIT DISTANCE Some languages, like Japanese, don’t have spaces between words, so word tokeniza-tion becomes more difficult. lemmatization Another part of text normalization is lemmatization, the task of determining that two words have the same root, despite their surface differences.

  Texts, Expression, Regular, Normalization, Regular expression, Text normalization