In 2023 we played a total of 7294 games. As a community, we played every day except for Thursday the 23nd November of 2023. Overall, we played about twice as many games as last year.

For the scope of this article, I count games with at least 5 moves played, regardless of whether the game was actually finished or not.

And if we plot this by month we can see that we were very active in spring, this is even clearer than in the heat map at the top of this article.

I hope we'll see many more games played next year and many people joining tournaments as well.

Additions to Paco Play

The website mostly looks the same as it did in January 2023, but it got some additions over the year.

Configuration Options

Since September you can chose the color of the board. We have a new menu for quick settings like that. The quick settings got started in August, because playing sounds would interrupt music playback on iPhone and players asked for a way to disable that.

I added additional color options to the arrows in October.

Tutorial for SPIEL 2023

October also brought an improved and translatable tutorial of the game, because I wanted to have this ready for SPIEL 2023.

Paco in 2

Since November 2023, the replay view for games tells you if you missed a "Paco in 2" or correctly identified the opportunity. This is described in more detail the Paco Ŝako Mate in 2 article.

Accounts to track your Games

Several people now have accounts on the website and you can log in to track your games. Right now there is no open sign-up for accounts, but you can just ask me on Discord to make a login for you.

Even our AI client can play under its own name - though it still isn't ready for general use.

To help players review their games, the Paco Play website has provided replay analysis for a while now. It tells you which moves put a player in Ŝako and crucially, when they missed a chance to win. The complex chains in a game of Paco Ŝako are easy to overlook, but the replay analysis catches them all.

Users quickly began sharing these missed opportunities in the #puzzles channel of our Discord. We now have a stable supply of puzzles from played games keeping us practiced. Here is a Paco I missed and shared. Can you find the chain for white to win in a single move?

You can open this puzzle on pacoplay.com. Reload the page when you get stuck. If you can't find it don't worry. I didn't find it either and subsequently lost the game a few moves later 🙈

With "Paco in 1" sorted out, some users started posting "Paco in 2" in the #puzzles channel. This adds a new level of difficulty and can help us learn more about playing Paco Ŝako as a community. But these positions are hard to build and it is easy to miss a defense.

To help with this, the replay analysis will now show (missed) Paco in 2 when the game had one.

What is a Paco in 2?

To teach the computer about Paco in 2, I first had to precisely understand what that means. I had an intuition before, but you can't code an intuition.

Definition: A Paco Ŝako position is called "Paco White in 2" if and only if

• there exists an "attack move" for White which puts Black in Ŝako such that for any "defense move" Black can respond with, Black remains in Ŝako anyway, and

• there exists no "attack move" for White that directly wins the game.

Implicitly hidden here is also, that Black uniting with the White king would be a valid defense. And of course, the definition works the other way around for Paco Black in 2 as well.

To turn this into a program, I do the following (Checking Paco White in 2):

• Get a list of all possible attack moves White can do.

  • If any of them wins the game, that's Paco White in 1, not in 2.

• For each attack move, look at all possible defense moves.

  • If any of them gets the Black player out of Ŝako, discard the attack move and try the next attack move.

• If there is no defense, store this attack move and look for additional attack moves. This means I get a list of attacks, not just a single one. Getting a list is important to understand if the replay contains a found Paco in 2 or a missed Paco in 2.

🐌 Oh no, it's Slow! 🐌

After I implemented it and played around with it in the UI, I saw that replays open extremely slowly now. Sometimes they don't open at all. Why is that?

In a complicated position with chains, there can be a few hundred possible moves that White can attack with. And for each one, Black may have a few hundred possible defense moves. The replay has to do that over and over again for all the moves in a game. That are a lot of positions for the Computer or Smartphone to look at! No wonder it is so slow.

To make analysis faster, I can't go by "this feels slow". I need some numbers to compare to each other. Let's build a Benchmark:

Loaded 2967 games in 326.602129ms
2333 games have at least 12 actions
Analyzed 2333 games in 524.385653535s

Plotting this into a histogram shows, that most of the time is used for analyzing a small minority of games.

I have added a few vertical lines here for different "percentiles":

• 50% of the games finish in under ~0.05 seconds. (p50)

• 90% of the games finish in ~0.3 seconds. (p90)

• Only 5% of the games take longer than ~0.7 seconds. (p95)

• An additional 33 games are taking more than 10 seconds each, some more than a minute, and some don't finish at all. I have excluded those 33 from my benchmark.

Getting faster

The first step in making this faster is to take off some guardrails. Our function to find Ŝako sequences was using both the traditional algorithm and the Amazon algorithm at the same time. This helped me identify and fix an issue with the Amazon algorithm, but now that it is working it just takes extra time. So let's go for the faster algorithm only:

Loaded 2967 games in 300.372085ms
2333 games have at least 12 actions
Analyzed 2333 games in 345.071743861s

We'll use this as a baseline.

As the first step, I replace vectors with arrays and place them into a separate struct. I also restrict access to use a trait.

// Old Version
struct DenseBoard {
    white: Vec<Option<PieceType>>,
    black: Vec<Option<PieceType>>,
    // ...
}

// New Version
struct DenseBoard {
    substrate: DenseSubstrate,
    // ...
}

struct DenseSubstrate {
    white: [Option<PieceType>; 64],
    black: [Option<PieceType>; 64],
}

trait Substrate {}
impl Substrate for DenseSubstrate {}

This saves some allocations and should also put the relevant data closer to each other in memory. This makes it easier for modern CPUs to work with it. Here are the updated benchmark results:

• p50: 29ms (-9% over baseline)

• p90: 150ms (-14% over baseline)

• p95: 293ms (-13% over baseline)

We already get some performance benefits just from shuffling around the memory. But the main reason we are doing this is to separate the code of the Substrate from the rest of the game logic. (Naming is hard. Substrate isn't a great name. Unfortunately the name Board is already used for the board state + rules + other stuff in combination.)

Faster Hashing

Now we can implement "Zobrist Hashing" in Substrate. It's about time, given that I already mentioned Zobrist Hashing in July 2022 (Optimizing Paco Ŝako with Flamegraphs). And indeed, this gives us quite some performance!

• p50: 9ms (-72% over baseline, -69% over last)

• p90: 61ms (-65% over baseline, -59% over last)

• p95: 128ms (-62% over baseline, -56% over last)

Re-running the benchmarks from the July 2022 article also shows improvements of over 50%. So things get faster across the board. Let's get a Flamegraph of our benchmark to understand where we can get some additional improvements! I'm basing this on games 1 through 100 only, because flamegraphs are slow to build.

cargo flamegraph --unit-test pacosako-tool-server -- test::paco_2_performance

The areas for improvement seem to be:

1. Finding place targets takes a lot of time when deciding if lifting a piece is allowed. That likely involves a lot of wasted work.

2. Finding place targets also takes a lot of time when we are looking to place a piece. That work isn't getting wasted, but maybe we can make it faster.

3. The Amazon algorithm for Ŝako detection uses a SetU32 . It can store numbers up to 4,294,967,295 while I only need to store numbers up to 64.

4. Wrangling hash maps takes about 13% of the time, and most of it is spent on reserve_rehash. I can probably reduce that by allocating a single hash map beforehand and reusing it.

Let's start with number 3. If you need a set that can store values between 0 and 63 you can use a single positive 64-bit integer called a "Bitboard". So far I have avoided working with Bitboards, as the Magic Bitboards for move generation are fast, but not intuitive. I still won't use any magic for now, but Bitboards still make great sets. Here are the new timings I get:

• p50: 8ms (-75% over baseline, -11% over last)

• p90: 53ms (-70% over baseline, -13% over last)

• p95: 108ms (-68% over baseline, -16% over last)

BitBoards for Lists of Board Positions

Several methods return a Vec<BoardPosition> to indicate which moves are allowed in a situation. I have exchanged those with a BitBoard as well to save allocations. Turns out, this changes the order in which moves are evaluated.

---- regression_run stdout ----
Testing the whole regression database...
Regression in game 1
First difference in legal actions on index 2.
Action taken: Lift(e7)
Expected: [Place(e6), Place(e5)]
Actual: [Place(e5), Place(e6)]

My Ŝako detection algorithms now find different, but equally valid solutions to unite with the King. And my big regression suite of over three thousand games no longer matches what the engine produces exactly. We can solve this by sorting the expected actions and the actual actions. And after sorting them the tests pass again.

• p50: 6ms (-81% over baseline, -25% over last)

• p90: 39ms (-78% over baseline, -26% over last)

• p95: 86ms (-75% over baseline, -20% over last)

Interning Boards to Reduce Cloning

Interning replaces an object (in our case a Paco Ŝako game board) with a number and then uses this number to refer to it. We can now hold on to two "copies" of the object just by remembering the number twice. I am using this when generating a list of all possible moves. This juggles a bunch of intermediate board states and builds a graph of them to avoid getting stuck in infinite chains.

I would have expected this to help in complicated situations with many loops, but it turns out this helps more in simple situations. We still get some performance in the complicated cases, so I'm keeping it in.

• p50: 4ms (-88% over baseline, -33% over last)

• p90: 35ms (-80% over baseline, -10% over last)

• p95: 80ms (-76% over baseline, -7% over last)

I have also tried replacing the HashMap with a BTreeMap, but that didn't improve performance for my benchmark.

Avoiding Impact on the Players 🚧

I am quite happy that analysis is now fast enough that most replays open in a reasonable time. We still have outliers to consider. Smartphones are slower than my PC, the code runs slower in the Browser than in my benchmark suite and some of the "problematic" replays won't open on my phone, even after waiting for over five minutes.

To solve this, I have to cut the replay analysis code into pieces.

This makes the replay page load faster for everyone as they no longer wait for Ŝako determination before getting to see the page. While it is loading, I am showing a small loading bar. At this point, the replay is already fully operational and additional labels will flow in one by one. You can click around even before it is fully analyzed.

As normal for progress bars, this will get stuck and jump ahead instead of smoothly increasing. I don't have a way to predict how complex each position is going to be, so I'm okay with keeping it as it is.

When looking at your replays look out for found and missed Paco in two - and please post all interesting positions to #puzzles on Discord!

The last game was https://pacoplay.com/game/6558 played between Yorick and Raimond. No moves were possible anymore and the timer broke. Yorick quickly reported that over Discord so I was able to take a look at the issue half an hour later.

After taking a look, it turned out that all the games were broken and you were no longer able to play at all. Restarting the Server would help, but every time anyone opened that game the server would crash again.

By removing the offending game from the database I was able to restart the server and this time you were able to play without issue. That was at 20:00, so we had an outage of one hour. For now, I was able to leave and play some Ticket to Ride with friends.

What was causing the issue?

To make sure we won't be hitting that same issue again, I made a copy of the database with the "evil" game inside to try things out in my development environment. Opening the game thankfully reproduced the issue and showed that the timer was indeed at fault:

thread '' panicked at 'called Result::unwrap() on an Err value: OutOfRangeError(())', src/ws/wake_up_queue.rs:27:64

The wake-up queue is the part of the server that allows it to wake up when the timer runs out and this forcefully ends the game.

The offending function is a helper method that allows me to pass a chrono::DateTime<Utc> into a function where I would usually expect a std::time::Instant:

pub fn put_utc(key: impl Into<String>, wake_up: chrono::DateTime<Utc>) {
    let wake_up = wake_up - Utc::now();
    let wake_up = std::time::Instant::now()
        + wake_up.to_std().unwrap();

    put(key, wake_up);
}

pub fn put(key: impl Into<String>, wake_up: Instant) { .. }

I am calling .unwrap() in here, so it is no surprise that this crashes. I should have handled that better. So what is the condition under which .to_std() doesn't return a proper value? Let's look at the documentation:

Creates a std::time::Duration object from time::Duration

This function errors when duration is less than zero. As standard library implementation is limited to non-negative values.

This method call asks the timer to please ring in the past. That we only see this happening now is probably the result of some rare timing condition. We try to set a timer a little bit in the future - but then wait too long and it is already in the past.

A fix to this is avoiding the unwrap. For example by just setting a 0-second timer instead.

pub fn put_utc(key: impl Into<String>, wake_up: chrono::DateTime<Utc>) {
  let wake_up = wake_up - Utc::now();
  let wake_up = std::time::Instant::now()
      + wake_up.to_std().unwrap_or(Duration::from_secs(0));

  put(key, wake_up);
}

I restarted the local server another time and lo and behold:

Since the "Timer sometimes goes into negative" issue is still unresolved, I can also see that it took me 3 hours and 45 minutes this time to locate the bug.

So to figure out if you are currently in Ŝako you need to look at the graph of all possible actions in a chain. This is quite slow to do and we want to do this pretty often for some kinds of analysis, so I want to have a faster way to do it. For this, I designed the "Reverse Amazon Algorithm" to skip this search completely when this is possible to detect. What is the algorithm and why the name?

It is unrelated to online shopping and also unrelated to rivers. Instead, it refers to the Amazon fairy chess piece which can move like a queen and a knight combined. Pieces move symmetrically, so I can also start at the king and look "in reverse". Any square that is within "amazon reach" of the king is a potential threat to him. Here are the squares that we see with the first look:

The squares which have either a pair on them or a single piece are the potential threats. We then focus on them and look at any squares that are within amazon reach of those. That is repeated until there are no more involved squares to check.

I can then run the graph search algorithm on a reduced graph. We only consider involved free pieces for the start and we only chain into pairs that are also involved.

If there is no free piece involved, then there is nothing to start and we can skip the complete graph search. I believe this "skipping" is where we save most of the time.

How much faster are we?

I didn't include any benchmarks until now, so how much did this optimization help compared to just running a full graph search on every position? The benchmark here is "training data generated for an AI we are training". For me, this went from "11 samples per second" to "120 samples per second". My friend had a similar improvement: going from 30 to 260 on his better CPU. This is a big win and it already allowed us to train a first model "Ludwig" that significantly outperforms our first attempt "Luna" which was a non-learning AI.

Notes on what makes this more difficult

En passant

While there is no piece on the en passant square, it can still be involved in a chain. The simplest way to deal with this is to mark it as involved and consider it occupied by a pair. This is pretty generous and we could likely reduce the search space by being more deliberate in how we do this. But the en passant is too rare to be worth more optimization here.

Rays through resting pieces

Since the chain is always started by lifting a free piece and starting to move it, this frees a piece. We'll later be able to move across this free space. So when we follow the queen rays to find involved squares, we need to look through one free piece at the piece behind that. If it is a pair, it is noted as involved as well.

This provides an opportunity for second-level optimization. After the first iteration of the reverse amazon algorithm, we'll know resting pieces that are not involved so we know that rays can't move through them. We can rerun a slightly modified algorithm where we don't look through the uninvolved resting pieces which potentially reduces the search space for the graph search, maybe even eliminating it.

More optimization opportunities

The second-level optimization also works in other ways. If we see e.g. that there are no involved knights, we can rerun the reverse amazon search with a restricted move set. This gives us a chance to reduce the set of involved pieces again and maybe run an even more restricted amazon search again. Right now no second-order optimizations are implemented. If you are interested in exploring this yourself, have a look at the code on GitHub!

A reverse amazon search should also be used to determine if castling is legal but I have not upgraded the castling code yet. That part is definitely on my to-do list and I'll keep a ticket open until I get around to it.

Paco Ŝako is a chess variant where you can't kill any pieces. Everything stays on the board and the game gradually evolves into an ever more complex web of possibilities.

Right now, asking the game engine to evaluate all 3500 games that were played takes 5.2 seconds on my machine using a single core. I installed cargo flamegraph to get an idea what it is spending it's time on. After a bit of fiddling with the parameters, I arrived at this:

> cargo flamegraph --test validate_all_played_games

running 2 tests
test build_regression_file ... ignored
test regression_run ... ok

test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured;
    0 filtered out; finished in 5.29s

[ perf record: Woken up 172 times to write data ]
[ perf record: Captured and wrote 42,916 MB perf.data (5274 samples) ]
writing flamegraph to "flamegraph.svg"

That's colorful, but what does it actually tell us? Let's zoom in!

Flamegraphs are upside down. So the outer most method we are interested in is regression_run at the bottom which is the integration test that got executed by our integration test. It spends a large chunk of its time in actions to find out which moves are legal. Most of that time goes to pieces_that_can_be_lifted where we figure out which pieces are allowed to start a move.

Now this is surprising, because shouldn't that be a trivial operation? You can lift all the pieces of your color, right? That used to be right, but at some point I reduced the ways to get stuck with a board and needing to rollback. Seems like this was initially triggered by the AI project, but it also makes playing online a lot nicer.

This is also the change that broke games 218 and 219 for the regression test. We had to exclude those from the data set in the last article.

/// To help the AIs a bit, we are not allowing it to lift any pieces
/// that get stuck instantly. They need to have at least one
/// position where they can be placed down again.
fn pieces_that_can_be_lifted(&self) -> Result<Vec<PacoAction>, PacoError> {
    let mut result = vec![];

    for p in self.active_positions() {
        let is_pair = self.opponent_present(p);
        let piece = *self.active_pieces().get(p.0 as usize).unwrap();
        if let Some(piece) = piece {
            let targets = self.place_targets(p, piece, is_pair)?;
            if !targets.is_empty() {
                result.push(PacoAction::Lift(p));
            }
        }
    }

    Ok(result)
}

Now that could probably use some improvements in method names and helper function availability... The gist is, that we iterate over all our pieces and find out some information about them (where are they, are they dancing, is it a pawn, rook, bishop, ..?). Then we ask the place_targets method if there is any way to put this piece down again.

This already smells quite inefficient, because we are constructing a vector of all possible positions where we can put the piece down when we really only want to know if a single target exists. Looking a step deeper, it is the place_targets_king which eats most of the time in determine_all_threats. What is that?

As in regular chess, you can only castle with king and rook if none of the squares the king would pass over are threatened by the opponent. (Including start and end square.) While in regular chess this is relatively simple, in Paco Ŝako it is not immediately obvious which pieces can threaten a square. This is because when a free piece is placed on a dancing couple your first piece enters the dance and the second one leaves, giving you essentially an extra move.

Here we have a short video of a chain where the king gets captured. There are several pieces involved and the knight even loop around to give the pawn an extra move.

Essentially this means we need to search a directed graph of game states (possibly containing infinite loops) to see if castling is even possible.

Luckily we don't need to optimize the graph algorithm yet. Whenever the king is able to castle, it is also able to move just a single square. Since we only care if any movement is possible at all, we can introduce a special case for the king:

fn pieces_that_can_be_lifted(&self) -> Result<Vec<PacoAction>, PacoError> {
    let mut result = vec![];

    for p in self.active_positions() {
        let is_pair = self.opponent_present(p);
        let piece = *self.active_pieces().get(p.0 as usize).unwrap();
        if let Some(piece) = piece {
            // For the King we need a special case, otherwise we would be
            // checking castling options which is expensive.
            // Since a castling option implies a move option, there is no
            // need to check for castling options.
            if piece == PieceType::King {
                let targets = self.place_targets_king_without_castling(p);
                if !targets.is_empty() {
                    result.push(PacoAction::Lift(p));
                }
            } else {
                let targets = self.place_targets(p, piece, is_pair)?;
                if !targets.is_empty() {
                    result.push(PacoAction::Lift(p));
                }
            }
        }
    }

    Ok(result)
}

The king always has the same type King and never is in a pair which cuts down on variables we need to pass into place_targets_king_without_castling.

Let's make another flamegraph to see if we improved!

Well, that tells me precisely nothing :-(

Flamegraphs are a great tool at telling you where you spend time and where you should look for optimizations. But they can't really tell you how you improved because they are all relative.

But we do have the absolute time it took to run the regression test suite:

> cargo flamegraph --test validate_all_played_games

running 2 tests
test build_regression_file ... ignored
test regression_run ... ok

test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured;
    0 filtered out; finished in 1.53s

[ perf record: Woken up 51 times to write data ]
[ perf record: Captured and wrote 12,744 MB perf.data (1526 samples) ]
writing flamegraph to "flamegraph.svg"

With only 1.53 seconds instead of 5.29 seconds as before, we are more than 3 times faster!

So where do we go next?

Zooming in again, we see that determine_all_threats still consumes most of the time. But now it mostly gets called from place_targets where we figure out where to place the current piece. That is where we expect it to be called, thought it still surprises me that it takes such a big percentage.

Most of this this time seems to be spend inside HashMap and indeed we have a set where we store all states we have ever seen. We need that information because otherwise we could get stuck in infinite loops.

For chess board there is a technique called Zobrist hashing which stores the hash value of a board and incrementally updates it whenever a move is made. That should save almost all the computation required for this. I'll look this as a way to optimize this further in a follow up article.

Also, I just glanced over the "directed graph of game states containing infinite loops" that we need to search if we want to see if castling is even possible. That should warrant a small article on its own.

It's not a great implementation, mind you. I have no idea how to properly build a chess engine. But it seems to be mostly bug free and has been powering the PacoPlay website for a while now.

I'd like to clean the game implementation up a bit and add some variants like PacoŜako960 to it. There are already a few dozens tests to cover all the rules - but I would like to get some extra confidence that I'm not spoiling the fun for our little community or breaking Felix Monday evening stream. Luckily, we have already played a few thousand games, maybe I can leverage that?

There is a way, and it's called Snapshot Testing, or Output comparison testing as Wikipedia suggests instead. I'll also be calling it Regression Testing here which is what I use in the code as well.

Getting the Game Data

I started by getting a copy of the production database and asking it for all games with at least one move. That whittled it down from 4108 to 3552 games already. I then asked Sqlite3 to please glue all of that together into one big json list.

select json_group_array(json_object('id', id, 'history', action_history))
from game where action_history != '[]';

{"id":1,"history":"[
{\"Lift\":12,\"timestamp\":\"2020-12-06T16:27:52.277541068Z\"},
{\"Place\":28,\"timestamp\":\"2020-12-06T16:27:52.934204496Z\"}, ...

We can see that the first ever game on PacoPlay was played in early December 2020, but we don't really care about that for validating that the engine still works. A bit of search and replace later and I got it in a better shape:

[{"id":1,"history":[{"Lift":12},{"Place":28},{"Lift":52},{"Place":36},{"Lift":11},{"Place":27},{"Lift":57},{"Place":42},{"Lift":5},{"Place":33},{"Lift":62},{"Place":45},{"Lift":6},{"Place":21},{"Lift":51},{"Place":43},{"Lift":10},{"Place":18},{"Lift":58},{"Place":37},{"Lift":4},{"Place":6},{"Lift":59},{"Place":51},{"Lift":27},{"Place":35},{"Lift":48},{"Place":40},{"Lift":9},{"Place":25},{"Lift":45},{"Place":28},{"Lift":33},{"Place":42},{"Lift":42},{"Place":27},{"Lift":18},{"Place":27},{"Place":34},{"Lift":43},{"Place":34},{"Lift":21},{"Place":38},{"Lift":36},{"Place":27},{"Place":12},{"Lift":6},{"Place":7},{"Lift":37},{"Place":28},{"Place":22},{"Lift":3},{"Place":27},{"Place":34},{"Place":52},{"Lift":22},{"Place":7}]}]

I'm storing moves as a "Lift" together with a separate "Place", because Paco Ŝako can have long move chains where the first placing frees a second piece which frees a third piece which frees ... But really, the content doesn't matter. I now have something that the engine accepts as input. I'll be able to read that in the test and then generate all the legal moves to verify that they didn't change.

Turning it into a Test

I'd like this to be separate from the other library tests, as this code gets a bit longer. Rust allows you to define "integration tests" which live outside your main code in the tests directory. Both resulting files are in the commit on GitHub.

In here, I have two tests:

• One "test" to generate the regression suite from the manually prepared input file. That one is not really a test though, so I'll need to put it on ignore. It's the "build suite" arrow from the diagram.
• A second real test to verify that logic still does what it is supposed to do.

Building the Regression Suite

#[derive(Deserialize, Clone)]
struct RegressionInput {
    id: usize,
    history: Vec<PacoAction>,
}

With serde_json I can quickly get the file content into a Vec<RegressionInput>. I then step through all the moves that were done on the board and record the legal moves in each step. That will give me a Vec<RegressionValidation>:

#[derive(Deserialize, Serialize, PartialEq, Eq, Debug)]
struct RegressionValidation {
    id: usize,
    history: Vec<PacoAction>,
    legal_moves: Vec<Vec<PacoAction>>,
}

I got some errors on the first try, turns out that games 218 and 219 were actually in an illegal state because I changed the implementation at some point after that. But I can just filter them out and the rest of the games processed cleanly.

#[ignore = "This is not a real test, but rather the utility
    used to build the regression database"]
#[test]
fn build_regression_file() {
    let input: Vec<RegressionInput> =
        load_game_database("tests/all_non_empty_games.json");

    // Remove games where the engine now does something else.
    let input: Vec<RegressionInput> = input
        .iter()
        .filter(|data| !FILTERED_OUT.contains(&data.id))
        .cloned()
        .collect();

    // Map each input to an output given the current logic
    let output: Vec<RegressionValidation> = input
        .into_iter()
        .map(map_input_to_validation)
        .collect::<Vec<_>>();

    // Write the output to a file
    write_regression_database(output, "tests/regression_database.json");
}

Here map_input_to_validation is the method that steps through the moves that were done.

Running the Regression Suite

Now we have a suite, we need to actually run it. Turns out all the pieces for this are already readily available:

#[test]
fn regression_run() {
    let games: Vec<RegressionValidation> = load_regression_database();

    for game in games {
        let input = RegressionInput {
            id: game.id,
            history: game.history.clone(),
        };
        let recomputed_game = map_input_to_validation(input);
        assert_eq!(game, recomputed_game);
    }
}

When we now ask cargo to run the test, we are left waiting for quite a while. At least we get an ok after 1.5 minutes:

Running tests/validate_all_played_games.rs

running 1 test
test regression_run ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured;
    1 filtered out; finished in 90.22s

Let's try it again with --release:

Running tests/validate_all_played_games.rs

running 1 test
test regression_run ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured;
    1 filtered out; finished in 5.29s

Much better! I'll be able to change the implementation of the core mechanics now and still sleep sound :-)

What's Next?

I ran some more analysis and found that ten (<3%) of the games take more than 50% of the time. I excluded them for now, because the unit tests don't run with the --release flag in CI. I'll certainly need to dig into those some more.

I'd love to show you some graphics of the runtime distribution, but my skills at that have somewhat degraded after leaving university.

The regression test also gives me a good performance benchmark. I'll use that to figure out which places of the move generator are terribly slow. I'm hoping to write the next blog posts about some optimizations.