NEAT OpenAI Atari Games

In this tutorial, we are going to create a genetic algorithm to play and Atari game. The algorithm is called NEAT, and mimics how evolution works. There is a starting population which slowly adapts to its environments and evolves. This approach will be applied to a simple game of pinball.

Step 1: Install Dependencies

Assuming you already have the latest version of Python 3 installed, many dependencies are needed.

Start by installing these dependencies:

pip install numpy

 pip install opencv-python

 pip install neat-python

 pip install pickle

pip install gym pip install opencv-python<br>

Now in order to be able to run the actual Atari emulator using OpenAI, separate dependencies must be installed based on your operating system.

Windows:

Since windows does not technically support Atari, run this piece of code in the command line:

pip install --no-index -f https://github.com/Kojoley/atari-py/releases atari_py 

Source

Linux:

This dependency is native to Linux & MacOS, so no 3rd party download is neccessary

pip install atari-py

Step 2: Set Up the Genetic Variables

If you know anything about Deep Learning, you know that there is a huge amount of parameters when training the model. Since the NEAT algorithm is independent, in the sense that it develops on its own, us as the developers need to fine tune the model as much as possible. Although some of the parameters go beyond the scope of this tutorial, most of the important ones will be explained using basic English. The actual definitions can be found here.

fitness_criterion: The function used to compute the termination criterion from the set of genome fitnesses. Allowable values are: min, max, and mean

fitness_threshold: When the agent should stop evolving

pop_size: number of entities per generation

reset_on_extinction: a new random population will be created if a certain species is doing poor

activation_default: the mathematical function between nodes in the neural network

activation_mutate_rate: the probably that an activation function is replaced with one from the activation_options

activation_options: the mathematical function that act as mutations

bias_mutate_rate: the probability that the bias will change in a node

bias_replace_rate: the probability that the bias will be randomized in a node

conn_add_prob: the probability that a connection between nodes will be created

conn_delete_prob: the probability that an existing node connection will be deleted

feed_forward = True: the neural network will be feed-forward strictly. This means no recurrent connections.

node_add_prob: the probability that a node will be created

node_delete_prob: the probability that an existing node will be deleted

num_hidden = starting amount of hidden nodes per population

num_inputs = size of input variable (x), or number of input nodes

num_outputs = size of output variable (y), or number of output nodes

species_fitness_func = method to determine how good a species is (a species fitness)

max_stagnation = how many generations with no improvement until a species is declared extinct

species_elitism = number of species to keep, in order to prevent a total extinction.

elitism = number of species whose geneotype will be remain static on to the next generation (applied to n of the most fit individuals)

survival_threshold = the fraction representing who is allowed to reproduce every generation

This is the basic definition for most of the important variables that you may (or may not want to) play around with. If you want to see the default settings used in this tutorial, click here and go to config-feedforward.txt.

Step 3: Introduction to the Environment

This is an intro to what exactly we are doing

Step 4: Noise Screen

A simple noise screen will be added. This will add variability to the input data and make the model overall better. More information on that here

Step 5: Gym Setup

Here we run a quick example in OpenAI gym.

Step 6: Coding Walkthrough

This video is the full coding lesson

Step 7: Emulation Display

This just shows what the actual emulator looks like

Step 8: Watch the Agent Run!

After this, you can run the code, and watch the agent play out the game. It's pretty interesting to see how exactly the agent learns, and what steps it takes to reach the max reward status.

What Next?

Now you can apply this to a new game. There are many different Atari games that OpenAI Gym supports, so have at it!