Training A Lunar Lander AI With Genetic Algorithms And C++

Training a lunar lander AI with genetic algorithms and c++

Beyond manual control, the Python Lunar Lander project features an AI autopilot capable of learning to land the craft. This is achieved using a Neural Network (NN) trained not with typical backpropagation, but with a Genetic Algorithm (GA), leveraging a C++ backend for performance.

The neural network wrapper

The NeuralNetwork class in mod_nn_train.py acts as the bridge between the Python game logic and the C++ NN/GA implementation.

• Initialization/Loading: It can either initialize a new GA population or load a previously saved state from an .hd5 file. The network structure (input/hidden/output layers) is defined in mod_config.py (nn_config.hlayers). The input layer size (6) matches the state vector provided by GameLogic, and the output layer size (4) corresponds to the possible actions (Noop, Up, Left, Right).
• Saving: Periodically saves the GA state (save method) using the C++ object’s Serialize function.
• Getting Actions: The get_action method takes the current game state, passes it to the C++ feedforward function (using the best individual, member 0), and returns the index of the highest output neuron, representing the chosen action.

# Getting an action from the NN in mod_nn_train.py
def get_action(self, current_state: np.ndarray) -> int:
    if self._net is None: # Check if loaded/initialized
        # ... error handling ...
        return 0

    num_outputs = self._nnsize[-1] # Should be 4
    inputs = np.array(current_state, dtype=np.float64)
    outputs = np.zeros(num_outputs, dtype=np.float64)

    try:
        # Call C++ feedforward via pybind11 wrapper
        # Use member_id 0 (assumed best) for prediction
        self._net.feedforward(inputs, outputs, 0, False)
        action = np.argmax(outputs) # Choose action with highest activation
        return action
    except Exception as e:
        # ... error handling ...
        return 0

State representation

The GameLogic class provides the crucial get_state method, which prepares the input vector for the NN. This vector needs to contain enough information for the NN to make informed decisions.

# State vector creation in GameLogic.get_state
def get_state(self) -> np.ndarray:
    # Calculate distances to the target landing pad
    dist_target_x = self.x - self.landing_pad_center_x
    dist_target_y = self.y - self.landing_pad_y

    # Normalize values for the NN
    state = np.array([
        self.vx / gcfg.max_vx,      # Normalized Horizontal Velocity
        self.vy / gcfg.max_vy,      # Normalized Vertical Velocity
        dist_target_x / cfg.width,  # Normalized X distance to pad center
        dist_target_y / cfg.height, # Normalized Y distance to pad top
        self.fuel / lcfg.max_fuel,  # Normalized Fuel
    ], dtype=float)

    # Clip values to prevent extremes
    state[0] = np.clip(state[0], -2, 2)
    state[1] = np.clip(state[1], -2, 2)
    state[2] = np.clip(state[2], -5, 5)
    state[3] = np.clip(state[3], -5, 5)

    return state

The state includes normalized velocities, normalized distances to the landing pad center and remaining fuel. Normalization helps the NN process these potentially wide-ranging values more effectively.

Genetic algorithm training

The train method implements the GA loop:

1. Evaluation: For each individual (NN) in the current population:
   • A full game episode is simulated using GameLogic.
   • At each step, the individual’s NN determines the action via feedforward.
   • Step penalties are accumulated (e.g., based on distance from the pad).
   • Once the episode ends (landed, crashed, or max steps reached), a final _calculate_terminal_penalty is computed. This heavily rewards successful landings (especially with fuel left) and penalizes crashes or running out of fuel.
   • The total fitness score (lower is better) is the sum of step penalties and the terminal penalty.

# Fitness calculation snippet from NeuralNetwork._calculate_terminal_penalty
def _calculate_terminal_penalty(self, game_sim: GameLogic, steps_taken: int) -> float:
    terminal_penalty = 0.0
    terminal_penalty += steps_taken * 0.1 # Penalty for time

    if game_sim.landed_successfully:
        terminal_penalty -= 10000.0 # Big reward
        terminal_penalty -= game_sim.fuel * 2.0 # Bonus for fuel
    elif game_sim.crashed:
        terminal_penalty += 10000.0 # Big penalty
        final_v_mag = np.sqrt(game_sim.vx**2 + game_sim.vy**2)
        terminal_penalty += final_v_mag * 10.0 # Penalty for speed
    elif game_sim.fuel <= 0 and not game_sim.landed:
        terminal_penalty += 7000.0 # Penalty for running out of fuel

    # ... small penalty for final velocity if not landed successfully ...
    return terminal_penalty

2. Selection & Evolution:

   • The fitness scores for the entire population are collected.
   • Indices are sorted based on fitness (ascending).
   • The C++ backend’s UpdateWeightsAndBiases method is called with the sorted indices. This likely performs selection (e.g., taking the top nn_config.top_individuals) and potentially crossover/mutation internally to generate the basis for the next generation.
   • CreatePopulation is called (with elitism flag from nn_config) to generate the new population, possibly preserving the best individual(s) directly.

3. Repeat: The process repeats for the configured number of epochs (generations). Pad positions can be reset periodically (nn_config.nb_batches) or if fitness stagnates (nn_config.fit_min, nn_config.fit_streak) to encourage exploration.

C++/Python integration

The magic of calling C++ code from Python happens via pybind11. The ann_mlp_ga_py_interface.h file defines the Python module cpp_nn_py.

// Snippet from ann_mlp_ga_py_interface.h exposing ANN_MLP_GA<double>
PYBIND11_MODULE(cpp_nn_py, m)
{
    py::class_<nn::ANN_MLP_GA<double>>(m, "ANN_MLP_GA_double")
        // Expose constructor, Serialize, Deserialize, UpdateWeightsAndBiases, etc.
        .def(py::init<std::vector<size_t>, int, size_t, size_t, size_t, bool>())
        .def("Serialize", &nn::ANN_MLP_GA<double>::Serialize)
        .def("Deserialize", &nn::ANN_MLP_GA<double>::Deserialize)
        .def("UpdateWeightsAndBiases", &nn::ANN_MLP_GA<double>::UpdateWeightsAndBiases)
        .def("CreatePopulation", &nn::ANN_MLP_GA<double>::CreatePopulation)
        // Expose feedforward, handling numpy arrays conversion
        .def("feedforward",
             [](nn::ANN_MLP_GA<double>& self, py::array_t<double> inputs, py::array_t<double> outputs, size_t memberid, bool singleReturn) {
                // ... C++ code to get pointers and call the actual feedforward ...
                self.feedforward(pInputs, inputsSize, pOutputs, outputsSize, memberid, singleReturn);
             },
             py::arg("inputs"), py::arg("outputs"), py::arg("memberid"), py::arg("singleReturn"))
        // ... other bindings ...
        ;
}

This binding code makes the C++ ANN_MLP_GA<double> class usable in Python as cpp_nn_py.ANN_MLP_GA_double. It handles the conversion between Python types (like NumPy arrays for feedforward) and C++ types. This allows the computationally intensive GA operations and network evaluations to run as compiled C++ code, while the game logic and overall control remain in Python.

Running the AI

To train the network:

python main.py --mode=nn_train
# To continue training from the last save:
python main.py --mode=nn_train --continue
# To continue from a specific generation (e.g., 50):
python main.py --mode=nn_train --continue --step=50

To watch the trained AI play:

python main.py --mode=nn_play

This mode loads the lunar_lander_last.hd5 file and uses the best network from the last saved generation to control the lander in the Pygame window.

By combining Pygame for the frontend, Python for high-level logic, and a C++ backend for the heavy lifting of NN training via a Genetic Algorithm, this project demonstrates a powerful approach to building game AI.

For more insights into this topic, you can find the details here.

The code for this implementation is available on Github here.