The Experts below are selected from a list of 321 Experts worldwide ranked by ideXlab platform
Tom Everitt - One of the best experts on this subject based on the ideXlab platform.
-
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective.
arXiv: Artificial Intelligence, 2019Co-Authors: Tom Everitt, Marcus HutterAbstract:Can an arbitrarily intelligent reinforcement learning agent be kept under control by a human user? Or do agents with sufficient intelligence inevitably find ways to shortcut their reward signal? This question impacts how far reinforcement learning can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we use an intuitive yet precise graphical model called Causal Influence diagrams to formalize reward tampering problems. We also describe a number of modifications to the reinforcement learning objective that prevent incentives for reward tampering. We verify the solutions using recently developed graphical criteria for inferring agent incentives from Causal Influence diagrams. Along the way, we also compare corrigibility and self-preservation properties of the various solutions, and discuss how they can be combined into a single agent without reward tampering incentives.
-
Modeling AGI Safety Frameworks with Causal Influence Diagrams
arXiv: Artificial Intelligence, 2019Co-Authors: Tom Everitt, Ramana Kumar, Victoria Krakovna, Shane LeggAbstract:Proposals for safe AGI systems are typically made at the level of frameworks, specifying how the components of the proposed system should be trained and interact with each other. In this paper, we model and compare the most promising AGI safety frameworks using Causal Influence diagrams. The diagrams show the optimization objective and Causal assumptions of the framework. The unified representation permits easy comparison of frameworks and their assumptions. We hope that the diagrams will serve as an accessible and visual introduction to the main AGI safety frameworks.
-
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings.
arXiv: Artificial Intelligence, 2019Co-Authors: Tom Everitt, Pedro A. Ortega, Elizabeth Barnes, Shane LeggAbstract:Agents are systems that optimize an objective function in an environment. Together, the goal and the environment induce secondary objectives, incentives. Modeling the agent-environment interaction using Causal Influence diagrams, we can answer two fundamental questions about an agent's incentives directly from the graph: (1) which nodes can the agent have an incentivize to observe, and (2) which nodes can the agent have an incentivize to control? The answers tell us which information and Influence points need extra protection. For example, we may want a classifier for job applications to not use the ethnicity of the candidate, and a reinforcement learning agent not to take direct control of its reward mechanism. Different algorithms and training paradigms can lead to different Causal Influence diagrams, so our method can be used to identify algorithms with problematic incentives and help in designing algorithms with better incentives.
Todd P. Coleman - One of the best experts on this subject based on the ideXlab platform.
-
measuring sample path Causal Influences with relative entropy
IEEE Transactions on Information Theory, 2020Co-Authors: Gabriel Schamberg, Todd P. ColemanAbstract:We present a sample path dependent measure of Causal Influence between time series. The proposed Causal measure is a random sequence, a realization of which enables identification of specific patterns that give rise to high levels of Causal Influence. We show that these patterns cannot be identified by existing measures such as directed information (DI). We demonstrate how sequential prediction theory may be leveraged to estimate the proposed Causal measure and introduce a notion of regret for assessing the performance of such estimators. We prove a finite sample bound on this regret that is determined by the worst case regret of the sequential predictors used in the estimator. Justification for the proposed measure is provided through a series of examples, simulations, and application to stock market data. Within the context of estimating DI, we show that, because joint Markovicity of a pair of processes does not imply the marginal Markovicity of individual processes, commonly used plug-in estimators of DI will be biased for a large subset of jointly Markov processes. We introduce a notion of DI with “stale history”, which can be combined with a plug-in estimator to upper and lower bound the DI when marginal Markovicity does not hold.
-
A Sample Path Measure of Causal Influence.
2018 IEEE International Symposium on Information Theory (ISIT), 2018Co-Authors: Gabriel Schamberg, Todd P. ColemanAbstract:We present a sample path dependent measure of Causal Influence between two time series. The proposed measure is a random variable whose expected sum is the directed information. A realization of the proposed measure may be used to identify the specific patterns in the data that yield a greater flow of information from one process to another, even in stationary processes. We demonstrate how sequential prediction theory may be leveraged to obtain accurate estimates of the Causal measure at each point in time and introduce a notion of regret for assessing the performance of estimators of the measure. We prove a finite sample bound on this regret that is determined by the regret of the sequential predictors used in obtaining the estimate. We estimate the Causal measure for a simulated collection of binary Markov processes using a Bayesian updating approach. Finally, given that the measure is a function of time, we demonstrate how estimators of the Causal measure may be extended to effectively capture Causality in time-varying scenarios.
Marcus Hutter - One of the best experts on this subject based on the ideXlab platform.
-
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective.
arXiv: Artificial Intelligence, 2019Co-Authors: Tom Everitt, Marcus HutterAbstract:Can an arbitrarily intelligent reinforcement learning agent be kept under control by a human user? Or do agents with sufficient intelligence inevitably find ways to shortcut their reward signal? This question impacts how far reinforcement learning can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we use an intuitive yet precise graphical model called Causal Influence diagrams to formalize reward tampering problems. We also describe a number of modifications to the reinforcement learning objective that prevent incentives for reward tampering. We verify the solutions using recently developed graphical criteria for inferring agent incentives from Causal Influence diagrams. Along the way, we also compare corrigibility and self-preservation properties of the various solutions, and discuss how they can be combined into a single agent without reward tampering incentives.
Gabriel Schamberg - One of the best experts on this subject based on the ideXlab platform.
-
measuring sample path Causal Influences with relative entropy
IEEE Transactions on Information Theory, 2020Co-Authors: Gabriel Schamberg, Todd P. ColemanAbstract:We present a sample path dependent measure of Causal Influence between time series. The proposed Causal measure is a random sequence, a realization of which enables identification of specific patterns that give rise to high levels of Causal Influence. We show that these patterns cannot be identified by existing measures such as directed information (DI). We demonstrate how sequential prediction theory may be leveraged to estimate the proposed Causal measure and introduce a notion of regret for assessing the performance of such estimators. We prove a finite sample bound on this regret that is determined by the worst case regret of the sequential predictors used in the estimator. Justification for the proposed measure is provided through a series of examples, simulations, and application to stock market data. Within the context of estimating DI, we show that, because joint Markovicity of a pair of processes does not imply the marginal Markovicity of individual processes, commonly used plug-in estimators of DI will be biased for a large subset of jointly Markov processes. We introduce a notion of DI with “stale history”, which can be combined with a plug-in estimator to upper and lower bound the DI when marginal Markovicity does not hold.
-
Information Theoretic Measures and Estimators of Specific Causal Influences
2019Co-Authors: Gabriel SchambergAbstract:Author(s): Schamberg, Gabriel | Advisor(s): Coleman, Todd P; Kim, Young-Han | Abstract: The need to measure Causal Influences between random variables or processes in complex networks arises throughout academic disciplines. In four parts, we here develop techniques for measuring and estimating Causal Influences using tools from information theory, with the explicit goal of providing context for how information theoretic perspectives on Causal Influence fit within the vast and interdisciplinary body of work studying Causality. Throughout the dissertation, we demonstrate the utility of the proposed methods with applications to physiologic, economic, and climatological datasets. Beginning with a focus on time series, we present a modularized approach to finding the maximum a posteriori estimate of a latent time series that obeys a dynamic stochastic model and is observed through noisy measurements. We specifically consider modern signal processing problems with non-Markov signal dynamics (e.g., group sparsity) and/or non-Gaussian measurement models (e.g., point process observation models used in neuroscience). Importantly, this framework can be leveraged in the estimation of the latent parameters specifying the probability distribution of a time series, which is a fundamental step in the estimation of Causal Influences between time series. Second, we study the conditions under which directed information, a popular information theoretic notion of Causal Influence between time series, can be estimated without bias. While the assumptions made by estimators of directed information are often presented explicitly, a characterization of when we can expect these assumptions to hold is lacking. Using the concept of d-separation from Bayesian networks, we present sufficient and almost everywhere necessary conditions for which proposed estimators can be implemented without bias. We further introduce a notion of partial directed information, which can be used to bound the bias under a milder set of assumptions. Third, we present a sample path dependent measure of Causal Influence between time series. The proposed measure is a random sequence, a realization of which enables identification of specific patterns that give rise to high levels of Causal Influence. We demonstrate how sequential prediction theory may be leveraged to estimate the proposed Causal measure and introduce a notion of regret for assessing the performance of such estimators which we subsequently bound. Finally, we extend our focus to general Causal graphs and show that information theoretic measures of Causal Influence are fundamentally different from mainstream (e.g. statistical) notions in that they (1) compare distributions over the effect rather than values of the effect and (2) are defined with respect to random variables representing a cause rather than specific values of a cause. We leverage perspectives from the statistical Causality literature to present a novel information theoretic framework for measuring direct, indirect, and total Causal effects in natural complex networks. In addition to endowing information theoretic approaches with an enhanced "resolution," the proposed framework uniquely elucidates the relationship between the information theoretic and statistical perspectives on Causality.
-
A Sample Path Measure of Causal Influence.
2018 IEEE International Symposium on Information Theory (ISIT), 2018Co-Authors: Gabriel Schamberg, Todd P. ColemanAbstract:We present a sample path dependent measure of Causal Influence between two time series. The proposed measure is a random variable whose expected sum is the directed information. A realization of the proposed measure may be used to identify the specific patterns in the data that yield a greater flow of information from one process to another, even in stationary processes. We demonstrate how sequential prediction theory may be leveraged to obtain accurate estimates of the Causal measure at each point in time and introduce a notion of regret for assessing the performance of estimators of the measure. We prove a finite sample bound on this regret that is determined by the regret of the sequential predictors used in obtaining the estimate. We estimate the Causal measure for a simulated collection of binary Markov processes using a Bayesian updating approach. Finally, given that the measure is a function of time, we demonstrate how estimators of the Causal measure may be extended to effectively capture Causality in time-varying scenarios.
Mingzhou Ding - One of the best experts on this subject based on the ideXlab platform.
-
granger Causal Influence predicts bold activity levels in the default mode network
Human Brain Mapping, 2011Co-Authors: Qing Jiao, Guangming Lu, Zhiqiang Zhang, Yuan Zhong, Zhengge Wang, Kai Li, Mingzhou DingAbstract:r Abstract: Although the brain areas in the default-mode network (DMN) act in a coordinated way dur- ing rest, the activity levels in the individual areas of the DMN are highly heterogeneous. The relation between the activity levels and the pattern of Causal interaction among the DMN areas remains unknown. In the present fMRI study, seven nodes of the DMN were identified and their activity levels were rank-ordered based on a power spectral analysis of the resting blood oxygenation level-depend- ent (BOLD) signals. Furthermore, the direction of information flow among these DMN nodes was determined using Granger Causality analysis and graph-theoretic methods. We found that the activity levels in these seven DMN nodes had a highly consistent hierarchical distribution, with the highest ac- tivity level in the posterior cingulate/precuneus cortices, followed by ventral medial prefrontal cortex and dorsal medial prefrontal cortex, and with the lowest level in the left inferior temporal gyrus. Importantly, a significant correlation was found between the activity levels and the In-Out degrees of information flow across the DMN nodes, suggesting that Granger Causal Influences can be used to pre- dict BOLD activity levels. These findings shed light on the dynamical organization of cortical neuronal networks and may provide the basis for characterizing network disruption by brain disorders. Hum Brain Mapp 32:154-161, 2011. V C 2010 Wiley-Liss, Inc.
-
Granger Causal Influence predicts BOLD activity levels in the default mode network.
Human brain mapping, 2010Co-Authors: Qing Jiao, Zhiqiang Zhang, Yuan Zhong, Zhengge Wang, Mingzhou Ding, Yongxin Guo, Yijun LiuAbstract:Although the brain areas in the default-mode network (DMN) act in a coordinated way during rest, the activity levels in the individual areas of the DMN are highly heterogeneous. The relation between the activity levels and the pattern of Causal interaction among the DMN areas remains unknown. In the present fMRI study, seven nodes of the DMN were identified and their activity levels were rank-ordered based on a power spectral analysis of the resting blood oxygenation level-dependent (BOLD) signals. Furthermore, the direction of information flow among these DMN nodes was determined using Granger Causality analysis and graph-theoretic methods. We found that the activity levels in these seven DMN nodes had a highly consistent hierarchical distribution, with the highest activity level in the posterior cingulate/precuneus cortices, followed by ventral medial prefrontal cortex and dorsal medial prefrontal cortex, and with the lowest level in the left inferior temporal gyrus. Importantly, a significant correlation was found between the activity levels and the In-Out degrees of information flow across the DMN nodes, suggesting that Granger Causal Influences can be used to predict BOLD activity levels. These findings shed light on the dynamical organization of cortical neuronal networks and may provide the basis for characterizing network disruption by brain disorders.