Using regular expressions for data management in Stata

1 Introduction to regular ExpressionsExamplesWhere can I go from here? Using regular expressions for datamanagement in StataRose Anne Consulting GroupAcademic Technology ServicesUniversity of California, Los Angeles2007 West Coast Stata Users Group meetingMedeirosRegular expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?Outline1 Introduction to regular ExpressionsWhat are regular expressions ?What do regular expressions look like?2 ExamplesExample 1: Extracting zip codesExample 2: Cleaning Data3 Where can I go from here?MedeirosRegular expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?What are regular expressions ?What do regular expressions look like?What are regular expressions ?A relatively easy, flexible method of searching strings.

2 Youcan use them to search any string ( variables, macros).In Stata , there are threefunctionsthat use expressions arenotthe solution to every probleminvolving strings. In most cases the built in string functionsin Stata will do at least as good a job, with less effort, and alower probability of expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?What are regular expressions ?What do regular expressions look like?What are regular expressions ?A relatively easy, flexible method of searching strings. Youcan use them to search any string ( variables, macros).In Stata , there are threefunctionsthat use expressions arenotthe solution to every probleminvolving strings. In most cases the built in string functionsin Stata will do at least as good a job, with less effort, and alower probability of expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?

3 What are regular expressions ?What do regular expressions look like?regexm(s,re)allows you to search for the string describedin your regular expressions . It evaluates to 1 if the stringmatches the (n)returns the nthsubstring within an expressionmatched by regexm (hence, regexm must always be runbefore regexs).regexr(s1,re,s2)searches forrewithin the string (s1) andreplaces the matching portion with a new string (s2).MedeirosRegular expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?What are regular expressions ?What do regular expressions look like?regexm(s,re)allows you to search for the string describedin your regular expressions . It evaluates to 1 if the stringmatches the (n)returns the nthsubstring within an expressionmatched by regexm (hence, regexm must always be runbefore regexs).

4 Regexr(s1,re,s2)searches forrewithin the string (s1) andreplaces the matching portion with a new string (s2).MedeirosRegular expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?What are regular expressions ?What do regular expressions look like?regexm(s,re)allows you to search for the string describedin your regular expressions . It evaluates to 1 if the stringmatches the (n)returns the nthsubstring within an expressionmatched by regexm (hence, regexm must always be runbefore regexs).regexr(s1,re,s2)searches forrewithin the string (s1) andreplaces the matching portion with a new string (s2).MedeirosRegular expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?What are regular expressions ?What do regular expressions look like?What do regular expressions look like?

5 In Stata they are always enclosed in quotation can include both strings you wish to match exactly,and more flexible descriptions of what to look typed directly are matched exactly (literals), "a"only matches "a".Operators are characters that appear in square brackets( [ and ] ), they are matched more flexibly, or are othercharacters that describe how they should be *+ ? ^ $ | ( ) [ ] \Values inside brackets may include ranges, 0-9, a-z,A-Z, f-x, example if we wanted to find the area codes in a list ofphone numbers we could use:"^[\(]?[0-9][0-9][0-9]"MedeirosRegul ar expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?What are regular expressions ?What do regular expressions look like?What do regular expressions look like?In Stata they are always enclosed in quotation can include both strings you wish to match exactly,and more flexible descriptions of what to look typed directly are matched exactly (literals), "a"only matches "a".)

6 Operators are characters that appear in square brackets( [ and ] ), they are matched more flexibly, or are othercharacters that describe how they should be *+ ? ^ $ | ( ) [ ] \Values inside brackets may include ranges, 0-9, a-z,A-Z, f-x, example if we wanted to find the area codes in a list ofphone numbers we could use:"^[\(]?[0-9][0-9][0-9]"MedeirosRegul ar expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?What are regular expressions ?What do regular expressions look like?What do regular expressions look like?In Stata they are always enclosed in quotation can include both strings you wish to match exactly,and more flexible descriptions of what to look typed directly are matched exactly (literals), "a"only matches "a".Operators are characters that appear in square brackets( [ and ] ), they are matched more flexibly, or are othercharacters that describe how they should be *+ ?)

7 ^ $ | ( ) [ ] \Values inside brackets may include ranges, 0-9, a-z,A-Z, f-x, example if we wanted to find the area codes in a list ofphone numbers we could use:"^[\(]?[0-9][0-9][0-9]"MedeirosRegul ar expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?What are regular expressions ?What do regular expressions look like?What do regular expressions look like?In Stata they are always enclosed in quotation can include both strings you wish to match exactly,and more flexible descriptions of what to look typed directly are matched exactly (literals), "a"only matches "a".Operators are characters that appear in square brackets( [ and ] ), they are matched more flexibly, or are othercharacters that describe how they should be *+ ? ^ $ | ( ) [ ] \Values inside brackets may include ranges, 0-9, a-z,A-Z, f-x, example if we wanted to find the area codes in a list ofphone numbers we could use:"^[\(]?))

8 [0-9][0-9][0-9]"MedeirosRegular expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?What are regular expressions ?What do regular expressions look like?What do regular expressions look like?In Stata they are always enclosed in quotation can include both strings you wish to match exactly,and more flexible descriptions of what to look typed directly are matched exactly (literals), "a"only matches "a".Operators are characters that appear in square brackets( [ and ] ), they are matched more flexibly, or are othercharacters that describe how they should be *+ ? ^ $ | ( ) [ ] \Values inside brackets may include ranges, 0-9, a-z,A-Z, f-x, example if we wanted to find the area codes in a list ofphone numbers we could use:"^[\(]?[0-9][0-9][0-9]"MedeirosRegul ar expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?)

9 Example 1: Extracting zip codesExample 2: Cleaning DataExample 1: Extracting zip codesWe have a list of addresses stored in a string variable, andwe want to extract the zip do we want to search for?A five-digit number ([0-9][0-9][0-9][0-9][0-9])Are there any complications? (Of course there are!)Some addresses include zip+4 Some addresses include the countrySome addresses have five-digit street numbersMedeirosRegular expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?Example 1: Extracting zip codesExample 2: Cleaning DataExample 1: Extracting zip codesWe have a list of addresses stored in a string variable, andwe want to extract the zip do we want to search for?A five-digit number ([0-9][0-9][0-9][0-9][0-9])Are there any complications? (Of course there are!)Some addresses include zip+4 Some addresses include the countrySome addresses have five-digit street numbersMedeirosRegular expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?

10 Example 1: Extracting zip codesExample 2: Cleaning DataExample 1: Extracting zip codesWe have a list of addresses stored in a string variable, andwe want to extract the zip do we want to search for?A five-digit number ([0-9][0-9][0-9][0-9][0-9])Are there any complications? (Of course there are!)Some addresses include zip+4 Some addresses include the countrySome addresses have five-digit street numbersMedeirosRegular expressions in StataIntroduction to regular ExpressionsExamplesWhere can I go from here?Example 1: Extracting zip codesExample 2: Cleaning DataThe dataMedeirosRegular expressions in Stata "([0-9][0-9][0-9][0-9][0-9])[\-]*[0 -9]*[ a-zA-Z]*$"+----------------------------- -------------------------+| address ||-------------------------------------- ----------------|| 4905 Lakeway Drive, College Station, Texas 77845 USA || 673 Jasmine Street, Los Angeles, CA 90024 || 2376 First street, San Diego, CA 90126 || 66666 West Central St, Tempe AZ 80068 || 12345 Main St.