âAIXIjs: A Software Demo for General Reinforcement Learningâ, 2017-05-22 (; backlinks; similar)â :
Reinforcement learning is a general and powerful framework with which to study and implement artificial intelligence. Recent advances in deep learning have enabled RL algorithms to achieve impressive performance in restricted domains such as playing Atari video games ( et al 2015) and, recently, the board game Go ( et al 2016). However, we are still far from constructing a generally intelligent agent. Many of the obstacles and open questions are conceptual: What does it mean to be intelligent? How does one explore and learn optimally in general, unknown environments? What, in fact, does it mean to be optimal in the general sense? The universal Bayesian agent AIXI (Hutter, 2005) is a model of a maximally intelligent agent, and plays a central role in the sub-field of general reinforcement learning (GRL).
Recently, AIXI has been shown to be flawed in important ways; it doesnât explore enough to be asymptotically optimal (2010), and it can perform poorly with certain priors (2015). Several variants of AIXI have been proposed to attempt to address these shortfalls: among them are entropy-seeking agents (2011), knowledge-seeking agents (Orseau et al 201311ya), Bayes with bursts of exploration (2013), MDL agents (2016a), Thompson sampling ( et al 2016), and optimism (2015).
We present AIXIjs, a JavaScript implementation of these GRL agents. This implementation is accompanied by a framework for running experiments against various environments, similar to OpenAI Gym ( et al 2016), and a suite of interactive demos that explore different properties of the agents, similar to REINFORCEjs (Karpathy, 2015). We use AIXIjs to present numerous experiments illustrating fundamental properties of, and differences between, these agents.