Prefix-Tuning: Optimizing Continuous Prompts for Generation

← Back to document page

Proceedings of the 59th Annual Meeting of the Association for Computational Linguisticsand the 11th International Joint Conference on Natural Language Processing, pages 4582 4597August 1 6, 2021. 2021 Association for Computational Linguistics4582Prefix-Tuning: Optimizing Continuous Prompts for GenerationXiang Lisa LiStanford LiangStanford is the de facto way of leveraginglarge pretrained language models for down-stream tasks. However, fine-tuning modifiesall the language model parameters and there-fore necessitates storing a full copy for eachtask. In this paper, we propose prefix-tuning, alightweight alternative to fine-tuning for natu-ral language Generation tasks, which keeps lan-guage model parameters frozen and instead op-timizes a sequence ofcontinuous task-specificvectors, which we call theprefix. Prefix-tuningdraws inspiration from prompting for languagemodels, allowing subsequent tokens to attendto this prefix as if it were virtual tokens.

to apply these controllable generation techniques to enforce fine-grained control over generated con-tents, as demanded by tasks like table-to-text and summarization. P*-tuning. Prefix tuning is an instance of a new class of methods that has emerged, which we call p*-tuning (since the other prominent instances, p-

  Generation, Methods, Technique, Generation techniques