Freecell Solver - Evolution of a C Program
1 Introduction
-
I wrote the first version of
Freecell Solver during a Spring break, because I was interested
to find out if a design I thought about would work.
-
Since then, it saw 13 major versions, each one adding more features
or improving the overall speed or memory consumption.
-
In this lecture, I would like to tell about some of the changes I
embodied in the program, many of which would be relevant to programming
different applications.
1.1 Rules of the Game
-
There are 8 stacks of cards, 4 freecells, and 4 foundations.
The game is played with one deck.
-
At the beginning of play, the cards are dealt to the stacks
(4*7+4*6). The faces of all the cards are visible at the beginning
of play.
-
A freecell can hold one single card.
-
The foundations are built from ace to king and the object of the
game is to move all the cards to the foundations.
-
A card can be placed on top of a card that is of the opposite
colour, and is higher in rank by 1, assuming the latter is at
the top of a stack.
-
An empty stack may be filled with any card.
-
An atomic move consists of:
-
Moving a card from a stack to a vacant freecell.
-
Moving a card from a freecell to a parent card on a stack.
-
Moving a card from the top of a stack, on top of a parent
card on a different stack.
-
Moving a card from the top of a stack or from a freecell to the
foundations.
1.1.1 Common Strategies
-
A sequence of more than one card can be moved by moving the top
cards to the freecells or to unoccupied stacks. This is commonly called
a sequence move or a supermove if it involves temporarily putting some
cards in a vacant stack.
-
Sometimes, it is useful to move cards to the freecells or temporarily
to an empty column, so the card below them can serve as a basis for
a sequence.
1.2 Copyrights and Disclaimer
Freecell Solver is distributed under the public domain. This lecture, on
the other hand, is a copyrighted, non free-content work. All rights are
reserved.
I hereby disclaim any damage that reading this work may cause you, either
implicit or explicit, direct or indirect.
2 Freecell Solver 0.2's Architecture
2.1 The Scan that was used
Pseudo-Code
Solve-State(state, prev_states, ret)
if (state == empty_state)
Push(ret, state)
return SOLVED
for each move possible on state
new_state <- apply(state, move)
if (new_state in prev_states)
; Do nothing
else
add new_state to prev_states
if (Solve-State(new_state, prev_states, ret) == SOLVED)
Push(ret, state)
return SOLVED
return UNSOLVED
Freecell-Solver(init_state)
if (Solve-State(init_state, Null, ret) == SOLVED)
print "State was solved"
while (state <- Pop(ret))
print state
else
print "I could not solve this board";
It's actually a Depth-First Search (DFS) scan.
3 Evolution of the States' Collection
In order to search efficiently we need to keep a collection of states
which we encountered so far.
That is so they (and their derived states) won't be traversed to again
and again.
But how do we manage this collection?
3.1 Initial Perl Version - Flat List
-
An unordered list of states.
-
Insertion is O(1), but lookup is O(n).
-
Dirt Slow.
-
As a general rule O(n) lookup is a very bad idea.
3.2 First C Version - Sorted Array
I kept a sorted array of states.
New states were added to the end of the array, which was kept momentarily
unsorted.
After a few of them were collected, they were added to the main array using
qsort.
O(log(n)) lookup and O(n*log(n)) - O(n2) addition.
Reasonable performance for most boards.
3.3 Binary-Search-Based Merge to Add the Sort Margin
-
Instead of qsorting the sort margin in, keep it sorted and
then use a binary-search based merge to merge it with the
main array.
-
The reason I used a binary search based merge, instead of a
linear merge, is because I reasoned that the main array
would become much larger than the margin (which has a constant
size), and so this would result in a lot of comparisons.
-
O(log(n)) lookup, O(n) insertion, and O(n2)
accumulative complexity.
-
Noticeably faster than qsorting.
3.4 A Balanced Binary Tree
-
A balanced binary tree is a a tree that is explicitly kept balanced.
-
It has a lookup and insertion complexity of O(log(n)), and an accumulative
complexity of O(n*log(n)). All of them worst case!
-
I only had a general idea how to implement them, but I was able to find
predefined open-source APIs on the web that could do the job for me:
libavl,
libredblack,
GLib's Balanced Binary Tree.
-
It turned out to be faster than the sorted array, by about a factor of 2.
3.5 A Hash Table
Can we do better than O(n*log(n))?
-
If we want to keep the states sorted - no.
-
If, however, we just want to determine if a given state is present
or not, then the answer is: "usually".
What is a hash?
As far as we are concerned a hash is an array of buckets. The index
of the bucket into which a given state goes is determined according to a
deterministic hash function.
A hash function is highly magical and should make sure similar records
have very distinct hash values, and all hash values are generated
in roughly equal frequency.
3.5.1 The Choice of a Hash Function
-
The first hash function I tried to use with GLib's hash was a function
that did a 4-byte wide XOR checksum on the data.
-
The performance of the hash was horrible, much more than anything else I tried.
-
I then switched to MD5 and so it performed much better.
-
After some time, I realized MD5 was a relatively slow, cryptologically secure,
hash function, and I could find faster functions.
-
Perl's hash function performed at least equally as well.
-
Your hash is only as good as your hash function!
3.5.2 Hash Optimizations
-
Storing the (non-moduloed) hash values along with the keys. That way, the
hash values can be compared first before comparing the entire keys.
-
Moving elements that were succesfully hit to the start of their chains.
-
When rehashing (= extending the hash to a greater number of buckets),
use the same malloc'ed elements, only re-link them to their new
followers.
3.6 Benchmarks
FILL IN
4 Moves Management
4.1 Meta-Moves (instead of Atomic ones)
-
Starting from the early versions, I decided to use meta-moves instead
of atomic moves.
-
Namely, I do several moves at once while trying to achieve a certain
"desirable" end in mind.
-
Examples:
-
Put top stack cards in the foundations.
-
Put Freecell cards in the foundations.
-
Put non-top stack cards in the foundations. (by moving cards above them
to freecells or vacant stacks).
-
Move stack cards to different stacks.
-
Move sequences of cards onto free stacks.
4.2 Stack to Stack Moves
-
To test the initial versions I generated a 1000 test boards.
-
I noticed some of them were not solvable.
-
I played some of them by hand, and noticed they were solvable using
a move that I missed.
-
The latter was a meta-move that placed a card on a parent card on the
same stack
-
By implementing it, all of my boards turned out to be solvable.
4.3 More Moves Generalization
-
Someone reported that some of the Microsoft boards were reported as
unsolvable by FCS.
-
I realized some of my meta-moves were not generic enough so I generalized
them.
-
Now, Freecell Solver can solve all of the Microsoft deals (except 11982
which was reported to be unsolvable by any human or computerized solver).
-
Somewhat later,
Tom Holroyd
reported a few
"Seahaven Towers" boards which he generated to be unsolvable.
-
Once again I was able to improve the solver to accommodate for solving them.
4.4 Non-Solvable Deals
-
Adrian Ettlinger sent me a few layouts he tested, with fewer Freecells
than usual, that Freecell Solver could not solve.
-
I realized that the reason Freecell Solver could not solve them was that
a meta-move moved two cards in the opposite manner that would allow
for a solution.
-
The only way I could think of to solve this without messing my code completely
would be to code move functions that would simply perform atomic moves.
-
It was not done, yet.
5 Scanning
-
Freecell Solver first implemented only a regular Depth-First Search
scan using procedural recursion.
-
The scan performed all the types of meta-moves in one static order.
-
As time progressed, I found it desirable to make the scans more flexible.
5.1 Specifying the Order of Tests
-
A test is a function that attempts to generate new boards using meta-moves.
-
Each test used to be implemented in its own separate block of code,
but they were only run one by one.
-
Later on, they were put in separate functions, each was assigned an ID,
and the user could specify in what order to run them (or a subset of them).
-
Choosing a good subset turned out to solve some difficult to solve boards,
very quickly.
5.2 Best-First Search
-
Best-First Search is a type of scan that:
-
Gives priority to each state based on some weight function.
-
Determines which state to go to next, according to their highest
priority.
-
I implemented Best-First Search whose weight function was a linear
combination of several weighting functions I thought about.
-
It turned out that for most boards, I could find some relative
weights that could solve it very quickly, but no configuration
was good for any board.
5.3 Soft DFS
-
PySol board No. 980662 recursed into a depth of over 3000. On Win32, this
caused a stack overflow which resulted in an ugly segfault.
-
I decided to implement a Soft-DFS scan which does not utilize procedural
recursion but rather its own dedicated stack.
-
This turned out to have an O(1) suspend and resume time instead of O(d)
for hard-DFS.
-
(I later on discovered that a Win32 program can be compiled with more stack
space, but I still think Soft-DFS is a better idea.
5.4 The BFS Optimization Scan
-
Stephan Kulow
(of KDE fame) complained that Freecell Solver generated some extremely
long solutions, which just moved sequences from one stack to the the other.
-
I suggested to use a Breadth-First Search scan restricted to the states
that were found in the solution path to try to eliminate redundant moves.
-
This turned out to be quite beneficial in most cases.
-
I later on implemented a scheme in which each state stored a pointer
to its "parent" state (the state from which it was initially discovered).
and used back-tracking to trace the solution down there.
-
This turned out to optimize solutions as well, but the BrFS optimization
improved it a bit too sometimes.
6 The State Representation
6.1 Reducing the Data Type Bit-Width
-
When Freecell Solver started it represented each card and stack length
specifier as a 32-bit quantity.
-
This caused it to run out of memory or stack in some cases.
-
To resolve it, I converted it to use 8-bit bytes.
-
This did not only consume less memory but also made it much faster.
-
I
queried the Linux-IL mailing-list about it and received some answers.
-
The most probable reasons: less memory -> less cache misses, and handling
bytes is just as fast as handling ints.
6.2 Pointers to Stacks
-
I wanted to add support for such games as Der Katzenschwanz and Die Schlange
in which stacks could be initialized to several dozens of cards.
-
That made stacks way too long and caused every board to consume a lot of
memory.
-
Solution: keep one copy of each stack once in a dynamically allocated memory.
-
Each state contains an array of pointers to each of its stacks.
-
That made it possible to scale up to a million states and more.
6.3 Remembering the Original Stack and Freecell Locations
-
In Freecell, it is possible that there several similar states would be
reached, that are only different in the order of the stacks or of the
freecells.
-
To solve this, I sorted the stacks and the freecells.
-
This, however, made the run-time display of the states display
them with a confused order.
6.3.1 Solution
-
Keep the indices of the stacks and freecells outside the main state
data-structure and sort the two arrays together. (i.e: in a struct that
contains the stacks-and-freecells struct as its first member)
-
The collection considers only the first sizeof(internal) bytes when
comparing two states.
-
I later used this external information place to store other information like
depth in the search tree, the parent state, etc.
7 Board Auto-Generators
-
A short time after the release of Freecell Solver 0.2.0, Eric Warmenhoven
sent me a program he prepared to generate the initial boards of GNOME
Freecell so they can later be inputted into Freecell Solver.
-
I thanked him for his effort, and decided to continue the theme.
-
So I wrote similar programs to generate the board layouts of GNOME AisleRiot,
PySol, and the Microsoft Freecell/Freecell Pro deals (the latter are considered the standard among hard-core Freecell enthusiasts).
-
Some programs have integrated the Freecell Solver library to allow for
automatically solving a board starting at a position that the player
has reached.
8 Why not C++?
Markus Oberhumer
(of PySol
fame asked if I thought about converting Freecell Solver to C++. (I suppose
he meant with STL, templates and all). Here is a full answer why I'm
not going to do it:
-
The solver is already working in ANSI C.
-
The code is not overly object-oriented. Whatever OOP concepts exist there.
fit well enough within what ANSI C gives me.
-
C++/STL may make it slower, perhaps even considerably. I'd rather not
spend time risking something like that, only to roll it back later.
-
ANSI C compiles much faster. (at least with gcc)
-
ANSI C is cleaner, more portable and causes less unexpected problems
than C++.
I'm more willing to integrate C++-derived features there into the
ANSI C code. Things like namespaces, (non-inherited) classes, or inline
functions. However, for the time being, I'd like to maintain the code as
pure ANSI C.
For that matter, some of the gcc extensions can prove very useful too, but
I cannot use them either.
9 The fc-solve-discuss flame-war
-
Recent versions of Freecell Solver pass the moves from the initial position
to the final solution to the layer above them.
-
The stack -> stack moves are being outputted with the number of cards
that are moved.
-
Freecell Pro, on the other hand, used to ignore the number of cards and
expected such moves to comply with what the original Microsoft Freecell
would do in that case.
-
This caused problems in playback of a large percentage of the games in
its integration with Freecell Solver.
-
A post I made to the mailing list about it sparked a very big flamewar that diverted
to cover related topics.
-
I was actually happy that there was some action there.
-
Usually the only things that happen there is that I announce new releases or
ask the members questions and no-one or few people reply.
-
Eventually, Adrian Ettlinger (an FC-Pro hacker) have extended Freecell Pro
to make use of an optional number of cards to move argument. This made
playback of Freecell Solver solutions perfectly smooth.
10 The story of the user API
-
Starting of Freecell Solver 1.0.0, FCS had a set of functions
entitled freecell_solver_user_ (after their prefix) which were meant
for integration into a larger software needing solving capabilities.
-
When Stephan Kulow integrated them into kpat (a solitaire suite for KDE),
he did not use it, because it did not give him everything he needed. Instead,
he used the more basic internal functions.
-
I told him that "I would sleep better at night" knowing that fcs_user_ will
be used, and asked him to implement the missing parts himself, and send
me the patch. He did.
-
Markus Oberhumer (of PySol fame), created a Python interface for the
library, and again sent me some functions he wrote.
-
Eventually, I converted the command line executable itself to use
fcs_user (while adding a lot of functions in the process) to make sure I
give the embedding application all of the functionality that I use.
-
I also ended up creating an API to analyze a command line and configure a
solver instance accordingly.
-
Sometimes later, I found it encouraging that <FILL IN>, an engineer
who worked on Freecell 3D, was able to integrate fcs_user_ without my help,
and just informed me of its release.
-
The importance of the "Eating your own Dog-Food" concept cannot be stressed enough.
11 Auto-confisication and Friends
-
Once upon a time, Freecell Solver was distributed only with a makefile
for GNU make, and everyone were happy.
-
When Stephan Kulow embedded its code into kpat, he adapted the makefile
to use the standard KDE Autoconf-based build process.
-
Somewhat afterwards, I decided that I want the build process to be more
portable and modular, and to give users the ability to build a shared
library.
-
The solution - converting to Autoconf, Automake and Libtool.
-
How to do that exactly is out of the scope of this lecture. I can just say
that it is not very straightforward, and that it required me a lot of tweaking
and trial and error.
-
I then decided that having an RPM of Freecell Solver would be nice. So I created
an RPM spec for it.
-
Again, it require quite a lot of tweaking and experimenting.
(Tzafrir Cohen's Lecture
and various web-resources were a great help)
-
I eventually integrated generating an up-to-date RPM SPEC into the Autoconf
process, and thus, was able to build an RPM by issuing a static sequence of
commands.
-
A corresponding Debian package has been initially created by Yotam Rubin and
is now maintained by Risko Gergely, who also uploads it to the Debian pool
of packages.
12 The Freshmeat Effect (and how to avoid it)
-
When I created the first version of FCS, and gave it the final touches,
I decided to call it "Freecell Solver" in order for it to have a
descriptive title
-
I posted announcements for the first release, and subsequent (usually
stable) releases on Freshmeat, and
so made many people aware of it.
-
Starting at the very first days, a
Google search for "freecell solver" yielded its homepage as the
first link.
-
Today, the situation is much worse: now most of the Google hits of this query
have something to do with it.
-
The query "freecell solver" is generic enough that someone may wish to
find any of the available Freecell solvers.
-
Solution: do your best to give an original name for your program, so it
won't clog up searches.