The Experts below are selected from a list of 45507 Experts worldwide ranked by ideXlab platform

Milica Gasic - One of the best experts on this subject based on the ideXlab platform.

  • agentgraph toward universal Dialogue Management with structured deep reinforcement learning
    IEEE Transactions on Audio Speech and Language Processing, 2019
    Co-Authors: Lu Chen, Bowen Tan, Zhi Chen, Sishan Long, Milica Gasic
    Abstract:

    Dialogue policy plays an important role in task-oriented spoken Dialogue systems. It determines how to respond to users. The recently proposed deep reinforcement learning (DRL) approaches have been used for policy optimization. However, these deep models are still challenging for two reasons: first, many DRL-based policies are not sample efficient; and second, most models do not have the capability of policy transfer between different domains. In this paper, we propose a universal framework, AgentGraph , to tackle these two problems. The proposed AgentGraph is the combination of graph neural network (GNN) based architecture and DRL-based algorithm. It can be regarded as one of the multi-agent reinforcement learning approaches. Each agent corresponds to a node in a graph, which is defined according to the Dialogue domain ontology. When making a decision, each agent can communicate with its neighbors on the graph. Under AgentGraph framework, we further propose dual GNN-based Dialogue policy, which implicitly decomposes the decision in each turn into a high-level global decision and a low-level local decision. Experiments show that AgentGraph models significantly outperform traditional reinforcement learning approaches on most of the 18 tasks of the PyDial benchmark. Moreover, when transferred from the source task to a target task, these models not only have acceptable initial performance but also converge much faster on the target task.

  • curiosity driven reinforcement learning for Dialogue Management
    International Conference on Acoustics Speech and Signal Processing, 2019
    Co-Authors: Paula Wesselmann, Milica Gasic
    Abstract:

    In this paper we describe the use of curiosity rewards for Dialogue policy learning of goal oriented Dialogues via reinforcement learning. Using curiosity improves state-action space exploration and helps overcome reward sparsity. Additionally, for goal oriented Dialogues it makes sense to perform inherently curious actions in order to gain knowledge about the user goal. We show that intrinsic curiosity rewards can replace random -greedy exploration and stabilize training. The best results are achieved when curiosity rewards are combined with -greedy exploration.

  • feudal Dialogue Management with jointly learned feature extractors
    Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2018
    Co-Authors: Inigo Casanueva, Pawel Budzianowski, Stefan Ultes, Florian Kreyssig, Bohsiang Tseng, Milica Gasic
    Abstract:

    Reinforcement learning (RL) is a promising Dialogue policy optimisation approach, but traditional RL algorithms fail to scale to large domains. Recently, Feudal Dialogue Management (FDM), has shown to increase the scalability to large domains by decomposing the Dialogue Management decision into two steps, making use of the domain ontology to abstract the Dialogue state in each step. In order to abstract the state space, however, previous work on FDM relies on handcrafted feature functions. In this work, we show that these feature functions can be learned jointly with the policy model while obtaining similar performance, even outperforming the handcrafted features in several environments and domains.

  • feudal reinforcement learning for Dialogue Management in large domains
    North American Chapter of the Association for Computational Linguistics, 2018
    Co-Authors: Inigo Casanueva, Pawel Budzianowski, Stefan Ultes, Bohsiang Tseng, Lina Maria Rojasbarahona, Milica Gasic
    Abstract:

    Reinforcement learning (RL) is a promising approach to solve Dialogue policy optimisation. Traditional RL algorithms, however, fail to scale to large domains due to the curse of dimensionality. We propose a novel Dialogue Management architecture, based on Feudal RL, which decomposes the decision into two steps; a first step where a master policy selects a subset of primitive actions, and a second step where a primitive action is chosen from the selected subset. The structural information included in the domain ontology is used to abstract the Dialogue state space, taking the decisions at each step using different parts of the abstracted state. This, combined with an information sharing mechanism between slots, increases the scalability to large domains. We show that an implementation of this approach, based on Deep-Q Networks, significantly outperforms previous state of the art in several Dialogue domains and environments, without the need of any additional reward signal.

  • a benchmarking environment for reinforcement learning based task oriented Dialogue Management
    arXiv: Machine Learning, 2017
    Co-Authors: Inigo Casanueva, Steve Young, Pawel Budzianowski, Stefan Ultes, Nikola Mrksic, Lina Maria Rojasbarahona, Tsunghsien Wen, Milica Gasic
    Abstract:

    Dialogue assistants are rapidly becoming an indispensable daily aid. To avoid the significant effort needed to hand-craft the required Dialogue flow, the Dialogue Management (DM) module can be cast as a continuous Markov Decision Process (MDP) and trained through Reinforcement Learning (RL). Several RL models have been investigated over recent years. However, the lack of a common benchmarking framework makes it difficult to perform a fair comparison between different models and their capability to generalise to different environments. Therefore, this paper proposes a set of challenging simulated environments for Dialogue model development and evaluation. To provide some baselines, we investigate a number of representative parametric algorithms, namely deep reinforcement learning algorithms - DQN, A2C and Natural Actor-Critic and compare them to a non-parametric model, GP-SARSA. Both the environments and policy models are implemented using the publicly available PyDial toolkit and released on-line, in order to establish a testbed framework for further experiments and to facilitate experimental reproducibility.

Steve Young - One of the best experts on this subject based on the ideXlab platform.

  • a benchmarking environment for reinforcement learning based task oriented Dialogue Management
    arXiv: Machine Learning, 2017
    Co-Authors: Inigo Casanueva, Steve Young, Pawel Budzianowski, Stefan Ultes, Nikola Mrksic, Lina Maria Rojasbarahona, Tsunghsien Wen, Milica Gasic
    Abstract:

    Dialogue assistants are rapidly becoming an indispensable daily aid. To avoid the significant effort needed to hand-craft the required Dialogue flow, the Dialogue Management (DM) module can be cast as a continuous Markov Decision Process (MDP) and trained through Reinforcement Learning (RL). Several RL models have been investigated over recent years. However, the lack of a common benchmarking framework makes it difficult to perform a fair comparison between different models and their capability to generalise to different environments. Therefore, this paper proposes a set of challenging simulated environments for Dialogue model development and evaluation. To provide some baselines, we investigate a number of representative parametric algorithms, namely deep reinforcement learning algorithms - DQN, A2C and Natural Actor-Critic and compare them to a non-parametric model, GP-SARSA. Both the environments and policy models are implemented using the publicly available PyDial toolkit and released on-line, in order to establish a testbed framework for further experiments and to facilitate experimental reproducibility.

  • sample efficient actor critic reinforcement learning with supervised data for Dialogue Management
    arXiv: Computation and Language, 2017
    Co-Authors: Peihao Su, Stefan Ultes, Pawel Budzianowski, Milica Gasic, Steve Young
    Abstract:

    Deep reinforcement learning (RL) methods have significant potential for Dialogue policy optimisation. However, they suffer from a poor performance in the early stages of learning. This is especially problematic for on-line learning with real users. Two approaches are introduced to tackle this problem. Firstly, to speed up the learning process, two sample-efficient neural networks algorithms: trust region actor-critic with experience replay (TRACER) and episodic natural actor-critic with experience replay (eNACER) are presented. For TRACER, the trust region helps to control the learning step size and avoid catastrophic model changes. For eNACER, the natural gradient identifies the steepest ascent direction in policy space to speed up the convergence. Both models employ off-policy learning with experience replay to improve sample-efficiency. Secondly, to mitigate the cold start issue, a corpus of demonstration data is utilised to pre-train the models prior to on-line reinforcement learning. Combining these two approaches, we demonstrate a practical approach to learn deep RL-based Dialogue policies and demonstrate their effectiveness in a task-oriented information seeking domain.

  • continuously learning neural Dialogue Management
    arXiv: Computation and Language, 2016
    Co-Authors: Milica Gasic, Stefan Ultes, Nikola Mrksic, Lina Maria Rojasbarahona, David Vandyke, Tsunghsien Wen, Steve Young
    Abstract:

    We describe a two-step approach for Dialogue Management in task-oriented spoken Dialogue systems. A unified neural network framework is proposed to enable the system to first learn by supervision from a set of Dialogue data and then continuously improve its behaviour via reinforcement learning, all using gradient-based algorithms on one single model. The experiments demonstrate the supervised model's effectiveness in the corpus-based evaluation, with user simulation, and with paid human subjects. The use of reinforcement learning further improves the model's performance in both interactive settings, especially under higher-noise conditions.

  • distributed Dialogue policies for multi domain statistical Dialogue Management
    International Conference on Acoustics Speech and Signal Processing, 2015
    Co-Authors: Milica Gasic, Dongho Kim, Pirros Tsiakoulis, Steve Young
    Abstract:

    Statistical Dialogue systems offer the potential to reduce costs by learning policies automatically on-line, but are not designed to scale to large open-domains. This paper proposes a hierarchical distributed Dialogue architecture in which policies are organised in a class hierarchy aligned to an underlying knowledge graph. This allows a system to be deployed using a modest amount of data to train a small set of generic policies. As further data is collected, generic policies can be adapted to give in-domain performance. Using Gaussian process-based reinforcement learning, it is shown that within this framework generic policies can be constructed which provide acceptable user performance, and better performance than can be obtained using under-trained domain specific policies. It is also shown that as sufficient in-domain data becomes available, it is possible to seamlessly improve performance, without subjecting users to unacceptable behaviour during the adaptation period and without limiting the final performance compared to policies trained from scratch.

  • requirements analysis and theory for statistical learning approaches in automaton based Dialogue Management
    2014
    Co-Authors: Romain Laroche, Steve Young, Ghislain Putois, Philippe Bretier, Oliver Lemon
    Abstract:

    Project ref. no. 216594Project acronym CLASSiCProject full title Computational Learning in Adaptive Systems for SpokenConversationInstrument STREPThematic Priority Cognitive Systems, Interaction, and RoboticsStart date / duration 01 March 2008 / 36 MonthsSecurity PublicContractual date of delivery M06 = 09 2008Actual date of delivery October 24, 2008Deliverable number 1.1.1Deliverable title Requirements Analysis and Theory for Statistical LearningApproaches in Automaton-Based Dialogue ManagementType ReportStatus & version Final 1.0Number of pages 32 (excluding front matter)Contributing WP 1WP/Task responsible WP 1Other contributors Florence Duclaye,Author(s) Romain Laroche,Ghislain Putois, Philippe Bretier, SteveYoung, Oliver LemonEC Project Officer Xavier GrosKeywords Requirements, Dialogue Management, Machine Learning,EBRL, MVDP, POMDP, SDS, MDP

Tatsuya Kawahara - One of the best experts on this subject based on the ideXlab platform.

  • bayes risk based Dialogue Management for document retrieval system with speech interface
    Speech Communication, 2010
    Co-Authors: Teruhisa Misu, Tatsuya Kawahara
    Abstract:

    We propose an efficient technique of Dialogue Management for an information navigation system based on a document knowledge base. The system can use ASR N-best hypotheses and contextual information to perform robustly for fragmental speech input and erroneous output of automatic speech recognition (ASR). It also has several choices in generating responses or confirmations. We formulate the optimization of these choices based on a Bayes risk criterion, which is defined based on a reward for correct information presentation and a penalty for redundant turns. The parameters for the Dialogue Management we propose can be adaptively tuned by online learning. We evaluated this strategy with our spoken Dialogue system called ''Dialogue Navigator for Kyoto City'', which generates responses based on the document retrieval and also has question-answering capability. The effectiveness of the proposed framework was demonstrated by the increased success rate of Dialogue and the reduced number of turns for information access through an experiment with a large number of utterances by real users.

  • bayes risk based Dialogue Management for document retrieval system with speech interface
    International Conference on Computational Linguistics, 2008
    Co-Authors: Teruhisa Misu, Tatsuya Kawahara
    Abstract:

    We propose an efficient Dialogue Management for an information navigation system based on a document knowledge base with a spoken Dialogue interface. In order to perform robustly for fragmental speech input and erroneous output of an automatic speech recognition (ASR), the system should selectively use N-best hypotheses of ASR and contextual information. The system also has several choices in generating responses or confirmations. In this work, we formulate the optimization of the choices based on a unified criterion: Bayes risk, which is defined based on reward for correct information presentation and penalty for redundant turns. We have evaluated this strategy with a spoken Dialogue system which also has questionanswering capability. Effectiveness of the proposed framework was confirmed in the success rate of retrieval and the average number of turns.

  • flexible mixed initiative Dialogue Management using concept level confidence measures of speech recognizer output
    International Conference on Computational Linguistics, 2000
    Co-Authors: Kazunori Komatani, Tatsuya Kawahara
    Abstract:

    We present a method to realize flexible mixed-initiative Dialogue, in which the system can make effective confirmation and guidance using concept-level confidence measures (CMs) derived from speech recognizer output in order to handle speech recognition errors. We define two concept-level CMs, which are on content-words and on semantic-attributes, using 10-best outputs of the speech recognizer and parsing with phrase-level grammars. Content-word CM is useful for selecting plausible interpretations. Less confident interpretations are given to confirmation process. The strategy improved the interpretation accuracy by 11.5%. Moreover, the semantic-attribute CM is used to estimate user's intention and generates system-initiative guidances even when successful interpretation is not obtained.

Stefan Ultes - One of the best experts on this subject based on the ideXlab platform.

  • exploring the applicability of elaborateness and indirectness in Dialogue Management
    IWSDS, 2019
    Co-Authors: Louisa Pragst, Wolfgang Minker, Stefan Ultes
    Abstract:

    In this paper, we investigate the applicability of soft changes to system behaviour, namely changing the amount of elaborateness and indirectness displayed. To this end, we examine the impact of elaborateness and indirectness on the perception of human-computer communication in a user study. Here, we show that elaborateness and indirectness influence the user’s impression of a Dialogue and discuss the implications of our results for adaptive Dialogue Management. We conclude that elaborateness and indirectness offer valuable possibilities for adaptation and should be incorporated in adaptive Dialogue Management.

  • feudal Dialogue Management with jointly learned feature extractors
    Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2018
    Co-Authors: Inigo Casanueva, Pawel Budzianowski, Stefan Ultes, Florian Kreyssig, Bohsiang Tseng, Milica Gasic
    Abstract:

    Reinforcement learning (RL) is a promising Dialogue policy optimisation approach, but traditional RL algorithms fail to scale to large domains. Recently, Feudal Dialogue Management (FDM), has shown to increase the scalability to large domains by decomposing the Dialogue Management decision into two steps, making use of the domain ontology to abstract the Dialogue state in each step. In order to abstract the state space, however, previous work on FDM relies on handcrafted feature functions. In this work, we show that these feature functions can be learned jointly with the policy model while obtaining similar performance, even outperforming the handcrafted features in several environments and domains.

  • feudal reinforcement learning for Dialogue Management in large domains
    North American Chapter of the Association for Computational Linguistics, 2018
    Co-Authors: Inigo Casanueva, Pawel Budzianowski, Stefan Ultes, Bohsiang Tseng, Lina Maria Rojasbarahona, Milica Gasic
    Abstract:

    Reinforcement learning (RL) is a promising approach to solve Dialogue policy optimisation. Traditional RL algorithms, however, fail to scale to large domains due to the curse of dimensionality. We propose a novel Dialogue Management architecture, based on Feudal RL, which decomposes the decision into two steps; a first step where a master policy selects a subset of primitive actions, and a second step where a primitive action is chosen from the selected subset. The structural information included in the domain ontology is used to abstract the Dialogue state space, taking the decisions at each step using different parts of the abstracted state. This, combined with an information sharing mechanism between slots, increases the scalability to large domains. We show that an implementation of this approach, based on Deep-Q Networks, significantly outperforms previous state of the art in several Dialogue domains and environments, without the need of any additional reward signal.

  • a benchmarking environment for reinforcement learning based task oriented Dialogue Management
    arXiv: Machine Learning, 2017
    Co-Authors: Inigo Casanueva, Steve Young, Pawel Budzianowski, Stefan Ultes, Nikola Mrksic, Lina Maria Rojasbarahona, Tsunghsien Wen, Milica Gasic
    Abstract:

    Dialogue assistants are rapidly becoming an indispensable daily aid. To avoid the significant effort needed to hand-craft the required Dialogue flow, the Dialogue Management (DM) module can be cast as a continuous Markov Decision Process (MDP) and trained through Reinforcement Learning (RL). Several RL models have been investigated over recent years. However, the lack of a common benchmarking framework makes it difficult to perform a fair comparison between different models and their capability to generalise to different environments. Therefore, this paper proposes a set of challenging simulated environments for Dialogue model development and evaluation. To provide some baselines, we investigate a number of representative parametric algorithms, namely deep reinforcement learning algorithms - DQN, A2C and Natural Actor-Critic and compare them to a non-parametric model, GP-SARSA. Both the environments and policy models are implemented using the publicly available PyDial toolkit and released on-line, in order to establish a testbed framework for further experiments and to facilitate experimental reproducibility.

  • challenges for adaptive Dialogue Management in the kristina project
    Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents, 2017
    Co-Authors: Louisa Pragst, Wolfgang Minker, Juliana Miehle, Stefan Ultes
    Abstract:

    Access to health care related information can be vital and should be easily accessible. However, immigrants often have difficulties to obtain the relevant information due to language barriers and cultural differences. In the KRISTINA project, we address those difficulties by creating a socially competent multimodal Dialogue system that can assist immigrants in getting information about health care related questions. Dialogue Management, as core component responsible for the system behaviour, has a significant impact on the successful reception of such a system. Hence, this work presents the specific challenges of the KRISTINA project to adaptive Dialogue Management, namely the handling of a large Dialogue domain and the cultural adaptability required by the envisioned Dialogue system, and our approach to handling them.

Lu Chen - One of the best experts on this subject based on the ideXlab platform.

  • distributed structured actor critic reinforcement learning for universal Dialogue Management
    arXiv: Computation and Language, 2020
    Co-Authors: Zhi Chen, Lu Chen, Xiaoyuan Liu
    Abstract:

    The task-oriented spoken Dialogue system (SDS) aims to assist a human user in accomplishing a specific task (e.g., hotel booking). The Dialogue Management is a core part of SDS. There are two main missions in Dialogue Management: Dialogue belief state tracking (summarising conversation history) and Dialogue decision-making (deciding how to reply to the user). In this work, we only focus on devising a policy that chooses which Dialogue action to respond to the user. The sequential system decision-making process can be abstracted into a partially observable Markov decision process (POMDP). Under this framework, reinforcement learning approaches can be used for automated policy optimization. In the past few years, there are many deep reinforcement learning (DRL) algorithms, which use neural networks (NN) as function approximators, investigated for Dialogue policy.

  • Distributed Structured Actor-Critic Reinforcement Learning for Universal Dialogue Management
    'Institute of Electrical and Electronics Engineers (IEEE)', 2020
    Co-Authors: Chen Zhi, Lu Chen, Liu Xiaoyuan, Yu Kai
    Abstract:

    The task-oriented spoken Dialogue system (SDS) aims to assist a human user in accomplishing a specific task (e.g., hotel booking). The Dialogue Management is a core part of SDS. There are two main missions in Dialogue Management: Dialogue belief state tracking (summarising conversation history) and Dialogue decision-making (deciding how to reply to the user). In this work, we only focus on devising a policy that chooses which Dialogue action to respond to the user. The sequential system decision-making process can be abstracted into a partially observable Markov decision process (POMDP). Under this framework, reinforcement learning approaches can be used for automated policy optimization. In the past few years, there are many deep reinforcement learning (DRL) algorithms, which use neural networks (NN) as function approximators, investigated for Dialogue policy.Comment: 12 pages, 7 figure

  • Distributed Structured Actor-Critic Reinforcement Learning for Universal Dialogue Management
    IEEE ACM Transactions on Audio Speech and Language Processing, 2020
    Co-Authors: Zhi Chen, Lu Chen, Xiaoyuan Liu
    Abstract:

    Traditional Dialogue policy needs to be trained independently for each Dialogue task. In this work, we aim to solve a collection of independent Dialogue tasks using a unified Dialogue agent. The unified policy is parallelly trained using the conversation data from all of the distributed Dialogue tasks. However, there are two key challenges:(1) the design of a unified Dialogue model to adapt to different Dialogue tasks; (2) finding a robust reinforcement learning method to keep the efficiency and the stability of the training process. Here we propose a novel structured actor-critic approach to implement structured deep reinforcement learning (DRL), which not only can learn parallelly from data of different Dialogue tasks but also achieves stable and sample-efficient learning. We demonstrate the effectiveness of the proposed approach on 18 tasks of PyDial benchmark. The results show that our method is able to achieve state-of-the-art performance.

  • agentgraph toward universal Dialogue Management with structured deep reinforcement learning
    IEEE Transactions on Audio Speech and Language Processing, 2019
    Co-Authors: Lu Chen, Bowen Tan, Zhi Chen, Sishan Long, Milica Gasic
    Abstract:

    Dialogue policy plays an important role in task-oriented spoken Dialogue systems. It determines how to respond to users. The recently proposed deep reinforcement learning (DRL) approaches have been used for policy optimization. However, these deep models are still challenging for two reasons: first, many DRL-based policies are not sample efficient; and second, most models do not have the capability of policy transfer between different domains. In this paper, we propose a universal framework, AgentGraph , to tackle these two problems. The proposed AgentGraph is the combination of graph neural network (GNN) based architecture and DRL-based algorithm. It can be regarded as one of the multi-agent reinforcement learning approaches. Each agent corresponds to a node in a graph, which is defined according to the Dialogue domain ontology. When making a decision, each agent can communicate with its neighbors on the graph. Under AgentGraph framework, we further propose dual GNN-based Dialogue policy, which implicitly decomposes the decision in each turn into a high-level global decision and a low-level local decision. Experiments show that AgentGraph models significantly outperform traditional reinforcement learning approaches on most of the 18 tasks of the PyDial benchmark. Moreover, when transferred from the source task to a target task, these models not only have acceptable initial performance but also converge much faster on the target task.

  • Policy Adaptation for Deep Reinforcement Learning-Based Dialogue Management
    ICASSP, 2018
    Co-Authors: Lu Chen, Milica Gaˇ, Bowen Tan, Cheng Chang, Kai Yu, Zhi Chen, Shanghai Jiao
    Abstract:

    Policy optimization is the core part of statistical Dialogue Management. Deep reinforcement learning has been suc-cessfully used for Dialogue policy optimization for a static pre-defined domain. However, when the domain changes dynamically, e.g. a new previously unseen concept (or slot) which can be then used as a database search constraint is added, or the policy for one domain is transferred to an-other domain, the Dialogue state space and action sets both will change. Therefore, the model structures for different domains have to be different. This makes Dialogue policy adaptation/transfer challenging. Here a multi-agent Dialogue policy (MADP) is proposed to tackle these problems. MADP consists of some slot-dependent agents (S-Agents) and a slot-independent agent (G-Agent). S-Agents have shared parameters in addition to private parameters for each one. During policy transfer, the shared parameters in S-Agents and the parameters in G-Agent can be directly transferred to the agents in extended/new domain. Simulation experiments showed that MADP can significantly speed up the policy learning and facilitate policy adaptation.