The Experts below are selected from a list of 972 Experts worldwide ranked by ideXlab platform

Whiteson S - One of the best experts on this subject based on the ideXlab platform.

  • UneVEn: Universal value exploration for multi-agent reinforcement learning
    2021
    Co-Authors: Whiteson S, Boehmer W, Gupta T, Mahajan A, Peng B
    Abstract:

    VDN and QMIX are two popular value-based algorithms for cooperative MARL that learn a centralized action value function as a monotonic mixing of per-agent utilities. While this enables easy decentralization of the learned policy, the restricted joint action value function can prevent them from solving tasks that require significant coordination between agents at a given timestep. We show that this problem can be overcome by improving the joint exploration of all agents during training. Specifically, we propose a novel MARL approach called Universal Value Exploration (UneVEn) that learns a set of related tasks simultaneously with a linear decomposition of universal successor features. With the policies of already solved related tasks, the joint exploration process of all agents can be improved to help them achieve better coordination. Empirical results on a set of exploration games, challenging cooperative predator-prey tasks requiring significant coordination among agents, and StarCraft II Micromanagement benchmarks show that UneVEn can solve tasks where other state-of-the-art MARL methods fail

  • Deep coordination graphs
    2020
    Co-Authors: Whiteson S, Boehmer W
    Abstract:

    This paper introduces the deep coordination graph (DCG) for collaborative multi-agent reinforcement learning. DCG strikes a flexible trade-off between representational capacity and generalization by factoring the joint value function of all agents according to a coordination graph into payoffs between pairs of agents. The value can be maximized by local message passing along the graph, which allows training of the value function end-to-end with Q-learning. Payoff functions are approximated with deep neural networks that employ parameter sharing and low-rank approximations to significantly improve sample efficiency. We show that DCG can solve predator-prey tasks that highlight the relative overgeneralization pathology, as well as challenging StarCraft II Micromanagement tasks

  • Growing action spaces
    2020
    Co-Authors: Whiteson S, Gustafson L, Farquhar G, Synnaeve G, Usunier N, Lin Z
    Abstract:

    In complex tasks, such as those with large combinatorial action spaces, random exploration may be too inefficient to achieve meaningful learning progress. In this work, we use a curriculum of progressively growing action spaces to accelerate learning. We assume the environment is out of our control, but that the agent may set an internal curriculum by initially restricting its action space. Our approach uses off-policy reinforcement learning to estimate optimal value functions for multiple action spaces simultaneously and efficiently transfers data, value estimates, and state representations from restricted action spaces to the full task. We show the efficacy of our approach in proof-of-concept control tasks and on challenging large-scale StarCraft Micromanagement tasks with large, multi-agent action spaces

  • Stabilising experience replay for deep multi-agent reinforcement learning
    2017
    Co-Authors: Foerster J, Farquhar G, Nardelli N, Torr Phs, Kohli P, Whiteson S
    Abstract:

    Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems. However, existing multi-agent RL methods typically scale poorly in the problem size. Therefore, a key challenge is to translate the success of deep learning on singleagent RL to the multi-agent setting. A key stumbling block is that independent Q-learning, the most popular multi-agent RL method, introduces nonstationarity that makes it incompatible with the experience replay memory on which deep RL relies. This paper proposes two methods that address this problem: 1) conditioning each agent’s value function on a footprint that disambiguates the age of the data sampled from the replay memory and 2) using a multi-agent variant of importance sampling to naturally decay obsolete data. Results on a challenging decentralised variant of StarCraft unit Micromanagement confirm that these methods enable the successful combination of experience replay with multi-agent RL

Changzheng Chen - One of the best experts on this subject based on the ideXlab platform.

Boehmer W - One of the best experts on this subject based on the ideXlab platform.

  • UneVEn: Universal value exploration for multi-agent reinforcement learning
    2021
    Co-Authors: Whiteson S, Boehmer W, Gupta T, Mahajan A, Peng B
    Abstract:

    VDN and QMIX are two popular value-based algorithms for cooperative MARL that learn a centralized action value function as a monotonic mixing of per-agent utilities. While this enables easy decentralization of the learned policy, the restricted joint action value function can prevent them from solving tasks that require significant coordination between agents at a given timestep. We show that this problem can be overcome by improving the joint exploration of all agents during training. Specifically, we propose a novel MARL approach called Universal Value Exploration (UneVEn) that learns a set of related tasks simultaneously with a linear decomposition of universal successor features. With the policies of already solved related tasks, the joint exploration process of all agents can be improved to help them achieve better coordination. Empirical results on a set of exploration games, challenging cooperative predator-prey tasks requiring significant coordination among agents, and StarCraft II Micromanagement benchmarks show that UneVEn can solve tasks where other state-of-the-art MARL methods fail

  • Deep coordination graphs
    2020
    Co-Authors: Whiteson S, Boehmer W
    Abstract:

    This paper introduces the deep coordination graph (DCG) for collaborative multi-agent reinforcement learning. DCG strikes a flexible trade-off between representational capacity and generalization by factoring the joint value function of all agents according to a coordination graph into payoffs between pairs of agents. The value can be maximized by local message passing along the graph, which allows training of the value function end-to-end with Q-learning. Payoff functions are approximated with deep neural networks that employ parameter sharing and low-rank approximations to significantly improve sample efficiency. We show that DCG can solve predator-prey tasks that highlight the relative overgeneralization pathology, as well as challenging StarCraft II Micromanagement tasks

Harvey F Lodish - One of the best experts on this subject based on the ideXlab platform.

Shimon Whiteson - One of the best experts on this subject based on the ideXlab platform.

  • stabilising experience replay for deep multi agent reinforcement learning
    2017
    Co-Authors: Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip H S Torr, Pushmeet Kohli, Shimon Whiteson
    Abstract:

    Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems. However, existing multi-agent RL methods typically scale poorly in the problem size. Therefore, a key challenge is to translate the success of deep learning on single-agent RL to the multi-agent setting. A major stumbling block is that independent Q-learning, the most popular multi-agent RL method, introduces nonstationarity that makes it incompatible with the experience replay memory on which deep Q-learning relies. This paper proposes two methods that address this problem: 1) using a multi-agent variant of importance sampling to naturally decay obsolete data and 2) conditioning each agent's value function on a fingerprint that disambiguates the age of the data sampled from the replay memory. Results on a challenging decentralised variant of StarCraft unit Micromanagement confirm that these methods enable the successful combination of experience replay with multi-agent RL.