After last month, this month feels considerably more positive, despite many set-backs!

Restructuring the Game

By far and away the most difficult part of making Push a game that is capable of being used by ML-Agents ("AI Brains") for training, has been butting up against a lot of the core designs I made when creating the multi-player version. For example, the original intention when playing games is that you would start a game manually, enter a lobby with an opponent (where a lot of verification is done), then lock-step into the game world to play the game in sync.

For ML-Agent training I needed to revisit ALL of the areas that prevented me from live-resetting the game inside the game world, which obviously resulted in a LOT of rewriting of some pretty fundamental code. What was once the "Game Controller", "Player Controller" and "Board Controller" for the whole environment, has now become it's own logical unit that can be created and destroyed anywhere, be reset or started at any time. This means the ML-Agents can take control of these units, and play on them, and we can spin up as many as feasible to accelerate the training (though I haven't quite gotten there yet).

There were a lot of other issues, where previously we would expect a solid order-of-operations to occur (as we needed because we had to sync over the network), but now things can somewhat happen out-of-sync, and independently in any of these units. Things as simple as Undo-Redo were causing issues where resetting the game would result in actions from the last game being played out in the initial phases of the new games! It's like turning over an ancient rock, and all these bugs come scurrying out.

The silver-lining here, is that these changes might actually cover a request that some have made, which is to stay in-game and replay the person you are already connected with. With this re-design, that may be possible now, and I will definitely explore it.

Understanding Unity ML-Agents

While I may understand a lot of how my own game is implemented, whenever you introduce a "Black Box" like ML-Agents to your system, and try to integrate it, you open yourself up to all sorts of issues where you may simply not understand what it's doing, and even worse, understand how to fix a problem. A lot of the initial work restructuring the game was spent working on, reworking on, and re-doing over and again the ML-Agent controllers, as my assumptions were challenged, my knowledge increased, and much documentation was read.

It also didn't help in the slightest that a LOT of the examples out there are already woefully out of date, and not applicable. This is especially bad when you are learning a new system, as you can't tell what is correct, and what you should truly be doing.

A side-track for Upgrades

The good news is, I realized I was circling the drain a tad, so I bit the bullet and decided to upgrade the whole project to the latest Unity Build (6+), which then allowed me to also utilize the latest ML-Agents package (3.0), which was thankfully well documented enough, with enough examples, for me to intuit the remainder of what I was missing.

What kind of things were missing, you ask? Well, one example is how to make a turn-based game where you take TWO actions per-turn (out of a possible three), which was not clear how to do from a training standpoint. 99% of the tutorials were for 3D games, teaching Agents how to move around an environment, walk, play 3D puzzles, and only one was even close to a "board game".

It took a while to figure out the desired pattern, but the documentation was good enough to lead me there.

Training in Progress!

Enough words! How about a demonstration? I'm VERY pleased to say that I have training underway, which will no doubt be an iterative process, as I iron out the appropriate rewards, restrict illegal moves, and cover any edge cases, but already, the system appears to be working for the game!

Check out a snippet of the latest training session, where two Agents are learning to play against each other in an adversarial manner (i.e. they are both trying to learn to win, despite the other trying to thwart their attempts, very like human players!)

Exciting!

We shall continue into April, and while I was hoping to get a training set in March, I at least managed to suffer through most of the pain, and it seems like mostly clear skies ahead. April it is!

Goals for April

The carry over from last-month is to complete a training set, and test it out, human vs computer! If I get something even remotely interesting, I'll consider packaging it up for release.

Naturally, I changed a ton of code, so I suspect most of the remaining time is likely to be spent regression testing everything, making sure the multiplayer game still works, and that I haven't prevented normal play in any way.

My robot buddies and I are looking forward to challenging you soon!

I'm really feeling Captain Picard's quote this month, and as a solo-developer also working a full-time job, some months are just really difficult to find the time to dig into a really large problem in the tiny bites of free time one gets.

Algorithmic Complexity

I started out the month working on a Computer Opponent algorithm which fit more traditional algorithms that chess programs and the like use. However, while I got it working relatively quickly, getting it to be performant is hideously difficult. For example, in most chess algorithms, you can usually constrain the number of possible moves one can make down to something that results in an opponent putting up a good challenge by being able to "see" approximately 10 turns ahead, a number which is seen as equivalent to how a really good chess player could see developing strategies almost presciently. In Push, I could only get about 4 turns ahead before I couldn't generate a move within the typical minute-default time limit. The issue with Push is that almost the whole board is "free" for a decent chunk of the game, meaning the number of possibilities per turn is vast, as compared to chess where the initial moves are limited to pawn movement and the knights, followed by limited moving as those shift around. As much as I tried to optimize, traversing that possibility space was mind-bending.

As the month drew on, and the code became more and more unreadable and indecipherable (akin to a text designed to summon some ancient unknowable mechanical god), I pushed my chair back, and decided to take a step away and re-assess the possibilities. This typical chess-style algorithm wasn't working.

Machine Learning Agents

In the back of my mind, I was passively aware of the recent trend in machine learning, and the successes one could find with it when "Training" it to perform specific tasks. Even more success can be found in environments where there are a known set of discrete actions one can take, much like this game. While I am incredibly wary of training sources for machine learning data sets, and care a lot about ethically training these "machine learning agents", I decided to at least educate myself more in the space enough to make an informed decision.

For the latter half of the month, I have been learning as much as I can about Unity's Machine Learning Agents, and training something that would not only learn to play this game, but end up quite fearsome by the end. Most importantly, it would likely be incredibly performant, and be able to make moves within the same time constraints that players have.

So far, I've noodled on a tic-tac-toe game, and pored over some machine-learning version of "Match-3" style games, and I've even taught cubes to balance a ball on their top surface!

The Direction for March

It's obvious, therefore, that as I am learning a whole new avenue of modern computer science, this will take more time. In March, I hope to have arranged the game in such a way as to allow for training of Machine Learning Agents, a process of which I have already started.

I would like to get something trained, watch it play (or even play against it), and then get a release out, as it has been a while since an update.

Thanks, dear reader, for your patience, and your interest in this game! I hope by the time this new direction has borne fruit, we will really have something super fun.

Here's to continuing to committing no more mistakes!