SuperGLUE: A Stickier Benchmark for General-Purpose ...

superglue : A Stickier Benchmark forGeneral-Purpose Language Understanding SystemsAlex Wang New York UniversityYada Pruksachatkun New York UniversityNikita Nangia New York UniversityAmanpreet Singh Facebook AI ResearchJulian MichaelUniversity of WashingtonFelix HillDeepMindOmer LevyFacebook AI ResearchSamuel R. BowmanNew York UniversityAbstractIn the last year, new models and methods for pretraining and transfer learning havedriven striking performance improvements across a range of language understand-ing tasks. The GLUE Benchmark , introduced a little over one year ago, offersa single-number metric that summarizes progress on a diverse set of such tasks,but performance on the Benchmark has recently surpassed the level of non-experthumans, suggesting limited headroom for further research.

we identify the following desiderata of tasks in the benchmark: Task substance: Tasks should test a system’s ability to understand and reason about texts written in English. Task difficulty: Tasks should be beyond the scope of current state-of-the-art systems, but solvable by most college-educated English speakers. We exclude tasks that require

Loading..

Tags:

  Desiderata, Superglue

Information

Related search queries