WebGPT: Browser-assisted question-answering with human ...

1 WebGPT: Browser-assisted question -answering withhuman feedbackReiichiro Nakano Jacob Hilton Suchir Balaji Jeff WuLong OuyangChristina KimChristopher HesseShantanu JainVineet KosarajuWilliam Saunders Xu Jiang Karl Cobbe Tyna Eloundou Gretchen KruegerKevin ButtonMatthew KnightBenjamin ChessJohn SchulmanOpenAIAbstractWe fine-tune GPT-3 to answer long-form questions using a text-based web-browsing environment, which allows the model to search and navigate the setting up the task so that it can be performed by humans, we are able to trainmodels on the task using imitation learning, and then optimize answer quality withhuman feedback. To make human evaluation of factual accuracy easier, modelsmust collect references while browsing in support of their answers.

2 We train andevaluate our models on ELI5, a dataset of questions asked by Reddit users. Ourbest model is obtained by fine-tuning GPT-3 using behavior cloning, and thenperforming rejection sampling against a reward model trained to predict humanpreferences. This model s answers are preferred by humans 56% of the time tothose of our human demonstrators, and 69% of the time to the highest-voted answerfrom IntroductionA rising challenge in NLP is long-form question -answering (LFQA), in which a paragraph-lengthanswer is generated in response to an open-ended question . LFQA systems have the potentialto become one of the main ways people learn about the world, but currently lag behind humanperformance [Krishna et al.]

3 , 2021]. Existing work tends to focus on two core components of the task,information retrieval and this work we leverage existing solutions to these components: we outsource document retrieval tothe Microsoft Bing Web Search API,2and utilize unsupervised pre-training to achieve high-qualitysynthesis by fine-tuning GPT-3 [Brown et al., 2020]. Instead of trying to improve these ingredients,we focus on combining them using more faithful training objectives. Following Stiennon et al. [2020],we use human feedback to directly optimize answer quality, allowing us to achieve performancecompetitive with make two key contributions: Equal contribution, order randomized2 (a) Screenshot from the demonstration interface.

4 QuestionHow can I train the crows in my neighborhood to bring me gifts? QuotesFrom Gifts From Crows | Outside My Window ( )> Many animals give gifts to members of their own species but crows andother corvids are the only ones known to give gifts to humans. Past actionsSearch how to train crows to bring you giftsClick Gifts From Crows | Outside My Window TitleSearch results for: how to train crows to bring you gifts Scrollbar: 0 - 11 Text 0 How to Make Friends With Crows - PetHelpful If you did this a few times, your crows would learn your new place, butas I said, I m not sure if they will follow or visit you there since it sprobably not in their territory.

5 The other option is simply to make newcrow friends with the crows that live in your new neighborhood. 1 Gifts From Crows | Outside My Window The partial piece of apple may have been left behind when the crow wasstartled rather than as a gift. If the crows bring bright objects you llknow for sure that it s a gift because it s not something they Williams says: May 28, 2020 at 7:19 am. Actions left: 96 Next action(b) Corresponding text given to the 1: An observation from our text-based web-browsing environment, as shown to humandemonstrators (left) and models (right). The web page text has been abridged for illustrative purposes. We create a text-based web-browsing environment that a fine-tuned language model caninteract with.

6 This allows us to improve both retrieval and synthesis in an end-to-end fashionusing general methods such as imitation learning and reinforcement learning. We generate answerswith references: passages extracted by the model from web pageswhile browsing. This is crucial for allowing labelers to judge the factual accuracy of answers,without engaging in a difficult and subjective process of independent models are trained primarily to answer questions from ELI5 [Fan et al., 2019], a dataset ofquestions taken from the Explain Like I m Five subreddit. We collect two additional kinds ofdata:demonstrationsof humans using our web-browsing environment to answer questions, andcomparisonsbetween two model-generated answers to the same question (each with their own set ofreferences).

7 Answers are judged for their factual accuracy, coherence, and overall use this data in four main ways: behavior cloning ( , supervised fine-tuning) using the demon-strations, reward modeling using the comparisons, reinforcement learning against the reward model,and rejection sampling against the reward model. Our best model uses a combination of behaviorcloning and rejection sampling. We also find reinforcement learning to provide some benefit wheninference-time compute is more evaluate our best model in three different ways. First, we compare our model s answers to answerswritten by our human demonstrators on a held-out set of questions. Our model s answers are preferred56% of the time, demonstrating human -level usage of the text-based browser.

8 Second, we compareour model s answers (with references stripped, for fairness) to the highest-voted answer providedby the ELI5 dataset. Our model s answers are preferred 69% of the time. Third, we evaluate ourmodel on TruthfulQA [Lin et al., 2021], an adversarial dataset of short-form questions. Our model sanswers are true 75% of the time, and are both true and informative 54% of the time, outperformingour base model (GPT-3), but falling short of human remainder of the paper is structured as follows: In Section 2, we describe our text-based web-browsing environment and how our modelsinteract with it. In Section 3, we explain our data collection and training methods in more detail.

9 In Section 4, we evaluate our best-performing models (for different inference-time computebudgets) on ELI5 and TruthfulQA. In Section 5, we provide experimental results comparing our different methods and howthey scale with dataset size, parameter count, and inference-time compute. In Section 6, we discuss the implications of our findings for training models to answerquestions truthfully, and broader 1: Actions the model can take. If a model generates any other text, it is considered to be aninvalid action. Invalid actions still count towards the maximum, but are otherwise <query>Send<query>to the Bing API and display a search results pageClicked on link <link ID>Follow the link with the given ID to a new pageFind in page: <text>Find the next occurrence of<text>and scroll to itQuote: <text>If<text>is found in the current page, add it as a referenceScrolled down <1, 2, 3>Scroll down a number of timesScrolled up <1, 2, 3>Scroll up a number of timesTopScroll to the top of the pageBackGo to the previous pageEnd: AnswerEnd browsing and move to answering phaseEnd.

10 <Nonsense, Controversial>End browsing and skip answering phase2 Environment designPrevious work on question -answering such as REALM [Guu et al., 2020] and RAG [Lewis et al.,2020a] has focused on improving document retrieval for a given query. Instead, we use a familiarexisting method for this: a modern search engine (Bing). This has two main advantages. First,modern search engines are already very powerful, and index a large number of up-to-date , it allows us to focus on the higher-level task of using a search engine to answer questions,something that humans can do well, and that a language model can this approach, we designed a text-based web-browsing environment.