A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking

Matthew Stephenson; Damien Anderson; Ahmed Khalifa; John Levine; Jochen Renz; Julian Togelius; Christoph Salge

doi:10.1109/CEC48606.2020.9185834

A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking

Matthew Stephenson, Damien Anderson, Ahmed Khalifa, John Levine, Jochen Renz, Julian Togelius, Christoph Salge

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper introduces an information-theoretic method for selecting a subset of problems which gives the most information about a group of problem-solving algorithms. This method was tested on the games in the General Video Game AI (GVGAI) framework, allowing us to identify a smaller set of games that still gives a large amount of information about the abilities of different game-playing agents. This approach can be used to make agent testing more efficient. We can achieve almost as good discriminatory accuracy when testing on only a handful of games as when testing on more than a hundred games, something which is often computationally infeasible. Furthermore, this method can be extended to study the dimensions of the effective variance in game design between these games, allowing us to identify which games differentiate between agents in the most complementary ways.

Original language	English
Title of host publication	2020 IEEE Congress on Evolutionary Computation (CEC)
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	1-8
Number of pages	8
ISBN (Print)	978-1-7281-6930-9
DOIs	https://doi.org/10.1109/CEC48606.2020.9185834
Publication status	Published - 24 Jul 2020
Event	2020 IEEE Congress on Evolutionary Computation (CEC) - Glasgow, UK Duration: 19 Jul 2020 → 24 Jul 2020

Conference

Conference	2020 IEEE Congress on Evolutionary Computation (CEC)
Period	19/07/20 → 24/07/20

Keywords

Games
Correlation
Artificial intelligence
Testing
Task analysis
Noise measurement
Technological innovation

Access to Document

10.1109/CEC48606.2020.9185834

https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9185834

Cite this

Stephenson, M., Anderson, D., Khalifa, A., Levine, J., Renz, J., Togelius, J., & Salge, C. (2020). A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking. In 2020 IEEE Congress on Evolutionary Computation (CEC) (pp. 1-8). Article 9185834 Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/CEC48606.2020.9185834

@inproceedings{119565be4c8a461b8973c04162da3e51,

title = "A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking",

abstract = "This paper introduces an information-theoretic method for selecting a subset of problems which gives the most information about a group of problem-solving algorithms. This method was tested on the games in the General Video Game AI (GVGAI) framework, allowing us to identify a smaller set of games that still gives a large amount of information about the abilities of different game-playing agents. This approach can be used to make agent testing more efficient. We can achieve almost as good discriminatory accuracy when testing on only a handful of games as when testing on more than a hundred games, something which is often computationally infeasible. Furthermore, this method can be extended to study the dimensions of the effective variance in game design between these games, allowing us to identify which games differentiate between agents in the most complementary ways.",

keywords = "Games, Correlation, Artificial intelligence, Testing, Task analysis, Noise measurement, Technological innovation",

author = "Matthew Stephenson and Damien Anderson and Ahmed Khalifa and John Levine and Jochen Renz and Julian Togelius and Christoph Salge",

year = "2020",

month = jul,

day = "24",

doi = "10.1109/CEC48606.2020.9185834",

language = "English",

isbn = "978-1-7281-6930-9",

pages = "1--8",

booktitle = "2020 IEEE Congress on Evolutionary Computation (CEC)",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

address = "United States",

note = "2020 IEEE Congress on Evolutionary Computation (CEC) ; Conference date: 19-07-2020 Through 24-07-2020",

}

Stephenson, M, Anderson, D, Khalifa, A, Levine, J, Renz, J, Togelius, J & Salge, C 2020, A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking. in 2020 IEEE Congress on Evolutionary Computation (CEC)., 9185834, Institute of Electrical and Electronics Engineers (IEEE), pp. 1-8, 2020 IEEE Congress on Evolutionary Computation (CEC), 19/07/20. https://doi.org/10.1109/CEC48606.2020.9185834

A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking. / Stephenson, Matthew; Anderson, Damien; Khalifa, Ahmed et al.
2020 IEEE Congress on Evolutionary Computation (CEC). Institute of Electrical and Electronics Engineers (IEEE), 2020. p. 1-8 9185834.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking

AU - Stephenson, Matthew

AU - Anderson, Damien

AU - Khalifa, Ahmed

AU - Levine, John

AU - Renz, Jochen

AU - Togelius, Julian

AU - Salge, Christoph

PY - 2020/7/24

Y1 - 2020/7/24

N2 - This paper introduces an information-theoretic method for selecting a subset of problems which gives the most information about a group of problem-solving algorithms. This method was tested on the games in the General Video Game AI (GVGAI) framework, allowing us to identify a smaller set of games that still gives a large amount of information about the abilities of different game-playing agents. This approach can be used to make agent testing more efficient. We can achieve almost as good discriminatory accuracy when testing on only a handful of games as when testing on more than a hundred games, something which is often computationally infeasible. Furthermore, this method can be extended to study the dimensions of the effective variance in game design between these games, allowing us to identify which games differentiate between agents in the most complementary ways.

AB - This paper introduces an information-theoretic method for selecting a subset of problems which gives the most information about a group of problem-solving algorithms. This method was tested on the games in the General Video Game AI (GVGAI) framework, allowing us to identify a smaller set of games that still gives a large amount of information about the abilities of different game-playing agents. This approach can be used to make agent testing more efficient. We can achieve almost as good discriminatory accuracy when testing on only a handful of games as when testing on more than a hundred games, something which is often computationally infeasible. Furthermore, this method can be extended to study the dimensions of the effective variance in game design between these games, allowing us to identify which games differentiate between agents in the most complementary ways.

KW - Games

KW - Correlation

KW - Artificial intelligence

KW - Testing

KW - Task analysis

KW - Noise measurement

KW - Technological innovation

UR - https://ieeexplore.ieee.org/document/9185834/

U2 - 10.1109/CEC48606.2020.9185834

DO - 10.1109/CEC48606.2020.9185834

M3 - Conference contribution

SN - 978-1-7281-6930-9

SP - 1

EP - 8

BT - 2020 IEEE Congress on Evolutionary Computation (CEC)

PB - Institute of Electrical and Electronics Engineers (IEEE)

T2 - 2020 IEEE Congress on Evolutionary Computation (CEC)

Y2 - 19 July 2020 through 24 July 2020

ER -

A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this