Abstract - arxiv.org

1 On-line Active Reward Learning for Policy Optimisationin Spoken Dialogue SystemsPei-Hao Su, Milica Ga si c, Nikola Mrk si c, Lina Rojas-Barahona,Stefan Ultes, David Vandyke, Tsung-Hsien Wen and Steve YoungDepartment of Engineering, University of Cambridge, Cambridge, UK{phs26, mg436, nm480, lmr46, su259, djv27, thw28, ability to compute an accurate re-ward function is essential for optimisinga dialogue policy via reinforcement learn-ing. In real-world applications, using ex-plicit user feedback as the reward sig-nal is often unreliable and costly to col-lect.}

2 This problem can be mitigated ifthe user s intent is known in advance ordata is available to pre-train a task suc-cess predictor off-line. In practice neitherof these apply for most real world applica-tions. Here we propose an on-line learn-ing framework whereby the dialogue pol-icy is jointly trained alongside the rewardmodel via active learning with a Gaussianprocess model. This Gaussian process op-erates on a continuous space dialogue rep-resentation generated in an unsupervisedfashion using a recurrent neural networkencoder-decoder.

3 The experimental resultsdemonstrate that the proposed frameworkis able to significantly reduce data annota-tion costs and mitigate noisy user feedbackin dialogue policy IntroductionSpoken Dialogue Systems (SDS) allow human-computer interaction using natural speech. Theycan be broadly divided into two categories: chat-oriented systems which aim to converse with usersand provide reasonable contextually relevant re-sponses (Vinyals and Le, 2015; Serban et al.,2015), and task-oriented systems designed to as-sist users to achieve specific goals ( find ho-tels, movies or bus schedules) (Daubigney et al.)

4 ,2014; Young et al., 2013). The latter are typi-cally designed according to a structuredontology(or a databaseschema), which defines the domainFigure 1: An example of a task-oriented dialoguewith a pre-defined task and the evaluation the system can talk about. Teaching a systemhow to respond appropriately in a task-orientedSDS is non-trivial. Thisdialogue managementtask is often formulated as a manually defined di-alogue flow that directly determines the quality ofinteraction. More recently, dialogue managementhas been formulated as a reinforcement learning(RL) problem which can be automatically opti-mised (Levin and Pieraccini, 1997; Roy et al.

5 ,2000; Williams and Young, 2007; Young et al.,2013). In this framework, the system learns byatrial and errorprocess governed by a poten-tially delayed learning objective defined by are-ward typical approach to defining the reward func-tion in a task-oriented dialogue system is to ap-ply a small per-turn penalty to encourage shortdialogues and to give a large positive reward atthe end of each successful interaction. Figure 1is an example of a dialogue task which is typi-cally set for users who are being paid to conversewith the system.

6 When users are primed with aspecific task to complete, dialogue success can bedetermined from subjective user ratings (Subj), [ ] 2 Jun 2016an objective measure (Obj) based on whether ornot the pre-specified task was completed (Walkeret al., 1997; Ga si c et al., 2013). However, priorknowledge of the user s goal is not normally avail-able in real situations, making the objective rewardestimation approach , objective ratings are inflexible andoften fail as can be seen from Figure 1, if theuser does not strictly follow the task.

7 This re-sults in a mismatch between theObjandSubjrat-ings. However, relying on subjective ratings aloneis also problematic since crowd-sourced subjectsfrequently give inaccurate responses and real usersare often unwilling to extend the interaction in or-der to give feedback, resulting in unstable learning(Zhao et al., 2011; Ga si c et al., 2011). In orderto filter out incorrect user feedback, Ga si c et al.(2013) used only dialogues for whichObj= , this is inefficient and not feasibleanyway in most real-world tasks where the user sgoal is generally unknown and difficult to light of the above, Su et al.

8 (2015a) pro-posed learning a neural network-basedObjesti-mator from off-line simulated dialogue data. Thisremoves the need for theObjcheck during on-line policy learning and the resulting policy is aseffective as one trained with dialogues using theObj=Subjcheck. However, a user simulatorwill only provide a rough approximation of realuser statistics and developing a user simulator is acostly process (Schatzmann et al., 2006).To deal with the above issues, this paper de-scribes an on-line active learning method in whichusers are asked to provide feedback on whether thedialogue was successful or not.

9 However, activelearning is used to limit requests for feedback toonly those cases where the feedback would be use-ful, and also a noise model is introduced to com-pensate for cases where the user feedback is inac-curate. A Gaussian process classification (GPC)model is utilised to robustly model the uncertaintypresented by the noisy user feedback. Since GPCoperates on a fixed-length observation space anddialogues are of variable-length, a recurrent neu-ral network (RNN)-based embedding function isused to provide fixed-length dialogue representa-tions.

10 In essence, the proposed method learns a di-alogue policy and a reward estimator on-line fromscratch, and is directly applicable to real-world rest of the paper is organised as next section gives an overview of relatedwork. The proposed framework is then describedin 3. This consists of the policy learning al-gorithm, the creation of the dialogue embeddingfunction and the active reward model trained fromreal user ratings. In 4, the proposed approach isevaluated in the context of an application provid-ing restaurant information in Cambridge, UK.