Any Way You Slice It: Evaluating the Penn Phonetics Lab ...

1 Methods Any Way You Slice It: Evaluating the Penn Phonetics Lab Forced Aligner Nicolaus Schrum Gwendolyn Rehrig, and Karin Stromswold Psychology Department and Rutgers University Center for Cognitive Science Rutgers University - New Brunswick Background Results Discussion Previous Phonetics studies used multiple coders creating morpheme boundaries using Praat acoustic analysis software by hand (Rehrig et al., 2015) Inter-rater reliability is extremely high, after a short training session The Penn Phonetics Lab Forced Aligner (p2fa) is an auto-aligner based on American English, and is the most commonly used aligner in the field (Yuan and Lieberman, 2008) Participants: 2 adult monolingual English-speakers Production Task: Read sentences aloud Stimuli: The/That x Object/Relative Clause/Reduced Relative Clause x 2 Primary NPs x 11 Verbs 22 of each type of six sentences: lawyer believed the bystander from the street.

2 Lawyer believed that bystander from the alley. lawyer believed the bystander saw the crime. lawyer believed that bystander saw the accused. lawyer believed that the bystander saw the prisoner. lawyer believed that that bystander saw the car. Only the first two noun phrases ( The lawyer & the/that bystander ) and the conjunction that were used for this comparison Coding: Word and silence boundaries marked by hand and p2fa Morpheme duration extracted Correlation Between P2fa and a Human Coder for 2 Subjects Future Directions Further Study: Are there other problem patterns? Possibly consecutive sibilants? Retraining: Can coders make judgements similar to the p2fa, or is the program using measurements beyond what the eye and ear can do?

3 Is the p2fa worth changing our procedures? The P2fa gives similar boundaries as a hand coder Appropriate for more general, large scale work Could be dangerous when used in tandem with hand coded data in smaller projects Some patterns are identified differently Sentences with two similar adjacent consonants ([d] on the verb ending ed and [ ] at the beginning of complementizer that) were consistently analyzed differently by p2fa/manual Are there other problem patterns? Is a .80 correlation acceptable for use? Can P2fa be used on its own, or should there be a second pass of manual correction? Research Questions How concordant with manual slicing is p2fa slicing? Are there any quirks to p2fa slicing?

4 Acknowledgments This work was supported by the National Science Foundation (REU CNS-1062735, IGERT DGE 0549115). Special Thanks to Paul de Lacy, Katie Aveni, Aldo Mayro, and Megan Kenny. Word-by-Word Breakdown That Examples Participant B Hand Coded Participant B P2fa Coded Participant Ignoring Silences Separate Silences A r(556) = .858, p < .0005 r(557) = .799, p < .0005 B r(564) = .840, p < .0005 r(564) = .817, p < .0005 P2fa is frequently significantly longer, because the manual slicing did not include silences The p2fa/manual differences were greatest on complementizer that References Jiahong Yuan and Mark Liberman (2008): Speaker identification on the SCOTUS corpus. Proceedings of Acoustics '08.

5 Paul Boersma & David Weenink (2013): Praat: doing Phonetics by computer [Computer program]. Version Rehrig, G., Beier, E., Chalmers, E., Schrum, N., Stromswold, K. (2015). Robust acoustic cues indicate upcoming structure in active and passive sentences. Poster presented March 19th 2015, The 28th Annual CUNY Conference on Human Sentence Processing, Midtown Los Angeles Radisson Hotel, Los Angeles, CA. 1 Noun 2 Noun 2 Duration (s) Participant A Breakdown Manual does not include silences 1 Noun 2 Noun 2 Duration (s) Participant B Breakdown Manual does not include silences Coding Manual Coding ignoring Silences Participant A: Manual ignoring Silences vs. P2fa Coding Manual Coding Ignoring Silences Participant B: Manual ignorng Silences vs.

6 P2fa * * [F(1, 88) = , p < .001] [F(1, 90) = , p = .011] * * * * *