Vaibhav Sagar's blog

SATisfying Solutions to Difficult Problems!

2025-10-22

Posted on 22 October 2025

Tags: programming

This post covers the same material as my !!Con 2024 talk, for which the slides are here.

What are SAT solvers, and how are they useful? Let’s start by briefly touching on NP-complete problems!

NP-complete problems

NP-complete problems are decision problems, i.e. the solution to them is “yes” or “no”. When these solutions exist, they can be verified in polynomial time, but we don’t know how to find solutions in polynomial time, or even if this is possible at all (this is the P versus NP problem). An important characteristic of NP-complete problems is that any NP-complete problem can be reduced to any other NP-complete problem. Examples of NP-complete problems include:

We’re specifically interested in the boolean satisfiability problem.

Boolean satisfiability problem

One definition of the boolean satisfiability problem is

Given a propositional logic formula, can we assign truth values to each variable such that the formula is satisfied?

When working with these formulas, we commonly express them in Conjunctive normal form as a conjunction (ANDed together) of clauses that consist of a series of disjunctions (ORed together) of literals, e.g.

\[(x \vee y \vee z) \wedge (x \vee \neg y) \wedge (\neg y \vee \neg x) \wedge (\neg z)\]

With this context, I can finally tell you what SAT solvers are!

SAT solvers

SAT solvers are programs that solve boolean satisfiability problems by providing satisfying assignments (when they exist)! In other words, they are programs for solving NP-complete problems expressed as instances of the boolean satisfiability problem(!!)

Sudoku

To demonstrate how this works in practice, let’s look at how to reduce Sudoku to a Boolean satisfiability problem.

The rules of Sudoku are as follows:

Each cell contains exactly one digit
Each digit occurs once per row
Each digit occurs once per column
Each digit occurs once per sub-grid
The solution must use the filled-in cells

A useful insight is that we can use \(n\) boolean variables to represent each digit when we add the constraint that at most one of those variables can be true.

Now we can express those rules in propositional logic:

Each cell has at least one value

\[\displaylines{ \definecolor{comment}{RGB}{161,161,180} {\color{comment}\textit{\small{row 1, column 1 is one of 1,2,...9}}} \\ (x_{1,1,1} \vee x_{1,1,2} \vee \dots \vee x_{1,1,9}) \wedge \\ {\color{comment}\textit{\small{row 1, column 2 is one of 1,2,...9}}} \\ (x_{1,2,1} \vee x_{1,2,2} \vee \dots \vee x_{1,2,9}) \wedge \\ \dots \\ {\color{comment}\textit{\small{row 9, column 9 is one of 1,2,...9}}} \\ (x_{9,9,1} \vee x_{9,9,2} \vee \dots \vee x_{9,9,9}) }\]

Each cell has at most one value

\[\displaylines{ \definecolor{comment}{RGB}{161,161,180} {\color{comment}\textit{\small{row 1, column 1 is not both 1 and 2}}} \\ (\neg x_{1,1,1} \vee \neg x_{1,1,2}) \wedge \\ {\color{comment}\textit{\small{row 1, column 1 is not both 1 and 3}}} \\ (\neg x_{1,1,1} \vee \neg x_{1,1,3}) \wedge \\ \dots \\ {\color{comment}\textit{\small{row 9, column 9 is not both 8 and 9}}} \\ (\neg x_{9,9,8} \vee \neg x_{9,9,9})}\]

Each row has all values

\[\displaylines{ \definecolor{comment}{RGB}{161,161,180} {\color{comment}\textit{\small{row 1 has a 1}}} \\ (x_{1,1,1} \vee x_{1,2,1} \vee \dots \vee x_{1,9,1}) \wedge \\ {\color{comment}\textit{\small{row 1 has a 2}}} \\ (x_{1,1,2} \vee x_{1,2,2} \vee \dots \vee x_{1,9,2}) \wedge \\ \dots \\ {\color{comment}\textit{\small{row 9 has a 9}}} \\ (x_{9,1,9} \vee x_{9,2,9} \vee \dots \vee x_{9,9,9})}\]

Each column has all values

\[\displaylines{ \definecolor{comment}{RGB}{161,161,180} {\color{comment}\textit{\small{column 1 has a 1}}} \\ (x_{1,1,1} \vee x_{2,1,1} \vee \dots \vee x_{9,1,1}) \wedge \\ {\color{comment}\textit{\small{column 1 has a 2}}} \\ (x_{1,1,2} \vee x_{2,1,2} \vee \dots \vee x_{9,1,2}) \wedge \\ \dots \\ {\color{comment}\textit{\small{column 9 has a 9}}} \\ (x_{1,9,9} \vee x_{2,9,9} \vee \dots \vee x_{9,9,9})}\]

Each sub-grid has all values

\[\displaylines{ \definecolor{comment}{RGB}{161,161,180} {\color{comment}\textit{\small{sub-grid 1 has a 1}}} \\ (x_{1,1,1} \vee x_{1,2,1} \vee \dots \vee x_{3,3,1}) \wedge \\ {\color{comment}\textit{\small{sub-grid 1 has a 2}}} \\ (x_{1,1,2} \vee x_{1,2,2} \vee \dots \vee x_{3,3,2}) \wedge \\ \dots \\ {\color{comment}\textit{\small{sub-grid 9 has a 9}}} \\ (x_{7,7,9} \vee x_{7,8,9} \vee \dots \vee x_{9,9,9})}\]

The solution must use the filled-in cells

For a puzzle such as

Tim Stellmach, CC0, via Wikimedia Commons

This looks like

\[\displaylines{ x_{1,1,5} \wedge x_{1,2,3} \wedge x_{1,5,7} \wedge \\ x_{2,1,6} \wedge x_{2,4,1} \wedge x_{2,5,9} \wedge x_{2,6,5} \wedge \\ \dots \\ x_{9,5,8} \wedge x_{9,8,7} \wedge x_{9,9,9}}\]

Solving

To solve this Sudoku (or indeed any NP-complete problem that we have expressed as a Boolean satisfiability problem), all we need to do is provide the resulting propositional logic formula as input to a SAT solver!

How do these marvellous programs work?

DPLL

One algorithm is known as DPLL. To explain how it works, let’s look at an example. Not so coincidentally, this is the same propositional logic formula from earlier! At the beginning, we don’t know the values of \(x\), \(y\), or \(z\).

\[(x \vee y \vee z) \wedge (x \vee \neg y) \wedge (\neg y \vee \neg x) \wedge (\neg z)\]

\(x\): 🤷
\(y\): 🤷
\(z\): 🤷

We begin by picking a variable and assigning it a truth value, preferring unit clauses (clauses with a single literal). In this case, let’s set \(z\) to \(False\). Now we can perform unit propagation.

Unit propagation

When performing unit propagation, we assign the appropriate truth value to a literal, which is obvious when it occurs in a unit clause. Then we remove all clauses that are satisfied, since we don’t need to consider them going forward. Next we remove the literal where it is \(False\), since it cannot contribute to that clause being satisfied.

\[(x \vee y \vee \cancel{{\color{red} z}}) \wedge (x \vee \neg y) \wedge (\neg y \vee \neg x) \wedge \cancel{{\color{green} (\neg z)}}\]

\(x\): 🤷
\(y\): 🤷
\(z\): False

Next we pick another variable and continue. Let’s set \(y\) to \(True\).

\[\cancel{{\color{green}(x \vee y \vee z)}} \wedge (x \vee \cancel{{\color{red} \neg y}}) \wedge (\cancel{{\color{red} \neg y}} \vee \neg x) \wedge \cancel{{\color{green}(\neg z)}}\]

\(x\): 🤷
\(y\): True
\(z\): False

Unfortunately, we now have a conflict, since two of the remaining clauses are \(x\) and \(\neg x\).

\[\cancel{{\color{green}(x \vee y \vee z)}} \wedge ({\color{blue} x} \vee \cancel{{\color{red} \neg y}}) \wedge (\cancel{{\color{red} \neg y}} \vee {\color{blue}\neg x}) \wedge \cancel{{\color{green}(\neg z)}}\]

\(x\): 🤷
\(y\): True
\(z\): False

The only appropriate thing to do here is backtrack, so we undo our previous assignment and try the other truth value, setting \(y\) to \(False\).

\[(x \vee \cancel{{\color{red} y}} \vee \cancel{{\color{red} z}}) \wedge \cancel{{\color{green}(x \vee \neg y)}} \wedge \cancel{{\color{green} (\neg y \vee \neg x)}} \wedge \cancel{{\color{green} (\neg z)}}\]

\(x\): 🤷
\(y\): False
\(z\): False

Now we can perform pure literal elimination.

Pure literal elimination

When literals involving a variable in a propositional logic formula are either always \(True\) (\(x\)) or always \(False\) (\(\neg x\)), then these are called pure literals, and it’s obvious what truth value to assign to them. In this case we set \(x\) to \(True\).

\[\cancel{{\color{green}(x \vee y \vee z)}} \wedge \cancel{{\color{green}(x \vee \neg y)}} \wedge \cancel{{\color{green} (\neg y \vee \neg x)}} \wedge \cancel{{\color{green} (\neg z)}}\]

\(x\): True
\(y\): False
\(z\): False

And that’s DPLL (Davis-Putnam-Logemann-Loveland)!

Davis-Putnam-Logemann-Loveland

Davis-Putnam-Logemann-Loveland is exhaustive backtracking search with unit propagation and pure literal elimination. Although it works reasonably well for small numbers of clauses, it has a tendency to repeatedly run into the same conflicts, and when it does it only backtracks one level at a time. It would be great to somehow remember and learn from these conflicts when we encounter them, so we can scale up to more complex problems. Does such an algorithm exist? It does.

CDCL

CDCL starts out very similarly to DPLL in that it still features unit propagation and pure literal elimination. However it also involves additional bookkeeping, distinguishing between decisions (when we choose a truth value for a variable) and implications (truth values determined through unit propagation and pure literal elimination). It also keeps track of the implication graph created by decisions and implications.

Let’s look at a more complex example.

\[\begin{align} & (a \vee d) \wedge \\ & (a \vee \neg c \vee \neg f) \wedge \\ & (a \vee f \vee j) \wedge \\ & (b \vee i) \wedge \\ & (\neg e \vee \neg c \vee g) \wedge \\ & (\neg e \vee f \vee \neg g) \wedge \\ & (e \vee f \vee \neg h) \wedge \\ & (e \vee h \vee \neg j) \end{align}\]

We start by setting \(a\) to \(False\), which implies \(d\) through the clause \(a \vee d\).

\[\begin{align} & {\color{red} a} \vee {\color{green} d} \\ & {\color{red} a} \vee \neg c \vee \neg f \\ & {\color{red} a} \vee f \vee j \\ & b \vee i \\ & \neg e \vee \neg c \vee g \\ & \neg e \vee f \vee \neg g \\ & e \vee f \vee \neg h \\ & e \vee h \vee \neg j \end{align}\]

Next we set \(c\) to \(True\), which implies \(\neg f\) through the clause \(a \vee \neg c \vee \neg f\) and \(j\) through the clause \(a \vee f \vee j\).

\[\begin{align} & {\color{red} a} \vee {\color{green} d} \\ & {\color{red} a} \vee {\color{red}\neg c} \vee {\color{green} \neg f} \\ & {\color{red} a} \vee {\color{red} f} \vee {\color{green} j} \\ & b \vee i \\ & \neg e \vee {\color{red}\neg c} \vee g \\ & \neg e \vee {\color{red} f} \vee \neg g \\ & e \vee {\color{red} f} \vee \neg h \\ & e \vee h \vee {\color{red}\neg j} \end{align}\]

We continue by setting \(b\) to \(False\), which implies \(i\) through the clause \(b \vee i\).

\[\begin{align} & {\color{red} a} \vee {\color{green} d} \\ & {\color{red} a} \vee {\color{red}\neg c} \vee {\color{green} \neg f} \\ & {\color{red} a} \vee {\color{red} f} \vee {\color{green} j} \\ & {\color{red} b} \vee {\color{green} i} \\ & \neg e \vee {\color{red}\neg c} \vee g \\ & \neg e \vee {\color{red} f} \vee \neg g \\ & e \vee {\color{red} f} \vee \neg h \\ & e \vee h \vee {\color{red}\neg j} \end{align}\]

Then we set \(e\) to \(True\), which causes a conflict because \(g\) is implied to be both \(True\) (through the clause \(\neg e \vee \neg c \vee g\)) and \(False\) (through the clause \(\neg e \vee f \vee \neg g\)).

\[\begin{align} & {\color{red} a} \vee {\color{green} d} \\ & {\color{red} a} \vee {\color{red}\neg c} \vee {\color{green} \neg f} \\ & {\color{red} a} \vee {\color{red} f} \vee {\color{green} j} \\ & {\color{red} b} \vee {\color{green} i} \\ & {\color{red}\neg e} \vee {\color{red}\neg c} \vee {\color{blue} g} \\ & {\color{red}\neg e} \vee {\color{red} f} \vee {\color{blue}\neg g} \\ & {\color{green} e} \vee {\color{red} f} \vee \neg h \\ & {\color{green} e} \vee h \vee {\color{red}\neg j} \end{align}\]

Clause learning

Fortunately we can analyse the implication graph to determine a Unique Implication Point that all edges from the latest decision node to the conflict node pass through, and the corresponding UIP cut corresponding to a clause. In this case the UIP is the decision node \(e\) and the clause is \(\neg f \vee c \vee e\). We want to remove the possibility of reaching this state again, so we negate this clause (by De Morgan’s theorem).

\[\displaylines{\neg (\neg f \wedge c \wedge e) \\ \iff \\ (f \vee \neg c \vee \neg e)}\]

And this gives us our learned clause!

\[(f \vee \neg c \vee \neg e)\]

We can then add it to our formula.

\[\begin{align} \definecolor{comment}{RGB}{161,161,180} \definecolor{emphasis}{RGB}{88,110,117} & {\color{comment}(a \vee d) \wedge} \\ & {\color{comment}(a \vee \neg c \vee \neg f) \wedge} \\ & {\color{comment}(a \vee f \vee j) \wedge} \\ & {\color{comment}(b \vee i) \wedge} \\ & {\color{comment}(\neg e \vee \neg c \vee g) \wedge} \\ & {\color{comment}(\neg e \vee f \vee \neg g) \wedge} \\ & {\color{comment}(e \vee f \vee \neg h) \wedge} \\ & {\color{comment}(e \vee h \vee \neg j) \wedge} \\\ & (f \vee \neg c \vee \neg e) \end{align}\]

Non-chronological backjumping

Next we backjump non-chronologically to the second-highest decision level of the literals in our clause, which in this case is \(2\), and repeat.

That’s CDCL (Conflict-driven clause learning)!

Conflict-driven Clause Learning

Conflict-driven clause learning is an extension of DPLL with learned clauses and non-chronological backtracking, effectively addressing most of DPLL’s downsides. It forms the basis of most modern SAT solvers.

SLS

In contrast to the rigorous and structured approaches we’ve seen already, what if we tried something more ad-hoc? We could generate a random assignment of \(True\) and \(False\) values for each of our variables, pick a clause at random, and flip either the “best” variable (whose negation causes the fewest conflicts) or some other variable. We could loop this selection and flipping a number of times, and restart the whole assignment upon getting stuck, finishing after either finding a solution or after a predetermined number of tries.

Stochastic Local Search

This is known as stochastic local search and it’s surprisingly effective! The specific algorithm I described above is called WalkSAT and it’s possible to do it in parallel and use a form of clause learning. Unfortunately, this approach cannot conclusively determine unsatisfiability because an inconclusive result might be due to the solver running out of attempts. A more general technique that this family of algorithms reminds me of is called simulated annealing.

SMT

SAT solvers are great when it’s straightforward to express your problem as a boolean satisfiability problem, but what if we want to solve more complex problems where we might not have the ability or desire to do so?

\[ \begin{align} SEND &\\ + MORE &\\ \hline MONEY \end{align} \]

To solve this puzzle, we need to assign digit values to each of the letters such that the sum of the 4-digit number represented by \(SEND\) and the 4-digit number represented by \(MORE\) equals the 5-digit number \(MONEY\).

We’d have to essentially teach our SAT solver how to do enough arithmetic for it to solve the equation

\[ \begin{align} (1000 \cdot (S+M)) + (100 \cdot (E+O)) + (10 \cdot (N+R)) + (D+E) &\\ = (10000 \cdot M) + (1000 \cdot O) + (100 \cdot N) + (10 \cdot E) + Y \end{align} \]

Satisfiability Modulo Theories

This is what an SMT (Satisfiability Modulo Theories) solver is! A SAT solver can be extended to reason over bitvectors, fixed-length arrays, Presburger arithmetic, algebraic datatypes, and more. One way of doing this would be to pre-process a problem into a series of CNF clauses that can be fed to a normal SAT solver; this is known as bit-blasting, but deeper integrations are often more practical. Examples of SMT solvers include Z3, CVC5, Yices, and Bitwuzla.

That’s all!

I hope you have a basic understanding of what a SAT/SMT solver is, how they (broadly) work, and when it might be a good idea to use them!

Resources

Thanks to Alex Chen for multiple rounds of excellent feedback on my presentation.

GHCi in the Browser

2024-07-03

Posted on 3 July 2024

Tags: haskell, programming

I’m happy to announce that you can now run GHCi entirely in your browser (if your browser supports WebAssembly and you’re willing to download approximately 220MB of compressed WASM).

Where?

Here or here.

How?

I used container2wasm to convert an OCI image containing GHC to a WASM blob that I could serve using a lightly modified container2wasm-demo. If you’re curious, the website repo is here and the chunks of WASM are here.

As of this writing, only images with an uncompressed size below 2GB can be used with container2wasm (tracked here) and my initial attempts using an OCI image generated by Nix were unsuccessful because of duplicate filenames (tracked here).

I also separately used WebVM which has the same file size limitation as container2wasm, only works on x86 binaries (which is why I used the i386 build of GHC), and is closed-source 😢, but potentially offers better performance depending on how well the JIT compiler performs on this workload.

Why?

I’ve wanted to do something like this for a long time. In my capacity as a maintainer of IHaskell, installation issues are the most common category of support request I receive. Wouldn’t it be great if a user could simply navigate to a webpage and have a correctly configured Jupyter notebook waiting for them? The Jupyter folks also seem to be thinking the same thing, based on the existence of JupyterLite. Unfortunately we’re a long way off from Haskell support¹, but I hope my proof-of-concept shows that this is possible.

Even outside Jupyter-land, a fully-functional GHCi REPL in the browser would be generally useful. For example, currently Haskell.org has a “Try it!” section where you can enter expressions, which are currently passed to a backend server to execute. A client-side GHCi could provide a better experience and allow us to get rid of the backend entirely. Another wild idea: the Hackage documentation for a package could provide a REPL with that package pre-installed for users to try out immediately. Wouldn’t that be amazing?

Why not compile GHCi directly to JavaScript/WASM using the new backends?

I don’t think that would work/result in a usable Haskell interpreter with access to base or other GHC boot packages. As of this writing it is on the roadmap for GHC 9.12+ so hopefully that will eventually be possible (tracked here). If you get this working I’d love to know about it!

It’s not something I’m working on and I don’t know how to go from this Goldbergian blob of WASM to a kernel that would work with JupyterLite. If you have ideas, please get in touch!↩︎

The Real Hash Was the Friends We Made along the Way

2024-02-14

Posted on 14 February 2024

Tags: programming

When I lived in Singapore, I attended a fascinating talk at FOSSASIA 2018 about Indeed’s fast and compact immutable key-value stores that went almost completely over my head. In fact, if you listen carefully during the Q&A session at the end, you can hear me ask some not-very-good questions in an ill-advised and ultimately futile attempt to relate to the speaker.

This was my first encounter with the concept of minimal perfect hashing. Unfortunately for me, I found most of the existing literature so impenetrable that I gave up on learning more. Hash, displace, and compress? Hypergraph peeling? RecSplit? Eventually I found a suitable entry point: Fast and scalable minimal perfect hashing for massive key sets.

Minimal perfect hashing

Let’s start with what minimal perfect hashing is:

Hashing

One definition of hashing is a process that converts some key to a value of some fixed size (e.g. an integer). We can think of this in terms of a hash function that takes some input and produces an integer as output.

Perfect

In practice, sometimes these hash functions produce the same output for different inputs, known as a hash collision. This is pretty annoying and causes lots of problems, and it would be nice if we could guarantee that distinct inputs always hash to different values, i.e. that the function is injective. Hash functions with this useful property are known as perfect hash functions. This requires all possible inputs to be known in advance.

Minimal perfect hashing

Bringing it all together, a minimal perfect hash function is one that has no gaps in its outputs, i.e. it bijectively maps \(n\) different inputs to \(n\) consecutive integers, e.g. \([0..n)\) or \([1..n]\). It’s important to note that minimal does not imply anything about the space or time complexity of these functions, e.g. it would be totally valid to have an internal hashtable that maps each input to a distinct integer without gaps and then use that to implement our hash function. In practice, however, we want these functions to be as efficient as possible to construct, store, and use, and this is an active area of research.

You’d probably want to use a minimal perfect hash when

all possible keys are known in advance
the set of keys doesn’t change
space is at a premium

One attractive property of a minimal perfect hash function is that you can use it to create a minimal perfect hash table by associating it with an array where each value’s index is the hash of the corresponding key.

How it works

The approach used in the paper is based on cascading collisionless bitarrays, as illustrated below. I have a more detailed example later so if you aren’t able to follow this one that’s totally okay! It exists to give you a quick taste of the algorithm.

In the example, keys \(k_1\) to \(k_6\) are hashed and positions where there are no collisions are recorded. The keys that collide at each level are removed and retried at the next level until all the keys are used. For the first bitarray \(A_0\), \(k_3\) and \(k_6\) do not collide when using the hash function \(h_0\). For the next bitarray \(A_1\), \(k_1\) and \(k_5\) do not collide when using \(h_1\). Finally for \(A_2\), \(k_2\) and \(k_4\) do not collide using \(h_2\) and we have no more keys left. To compute the hash for a key, in this example \(k_2\), we find the position where \(A_n[h_n(k_2)] \equiv 1\) and count the number of 1s at or preceding this position, also known as the rank, which will always give us a number \([1..n]\). For \(k2\), the hash is \(5\).

Prerequisites

To implement this, we’ll need

a family of hash functions
bitvectors supporting \(rank\) (and \(select\)) operations

For the hash functions, I used hashWithSalt from the hashable package, and for the bitvectors I used the bv-little package because past Vaibhav asked for rank and select support.

Construction

At a high level, this is what the construction algorithm looks like:

Repeat the following steps until the maximum level is reached or we have no more keys:
1. Hash each key to a number \(i \in [0..n)\)
2. If \(bitvector[i]\) has not been set this iteration, set it to \(1\), otherwise unset it
3. Remove all keys that have been set successfully
If there are any leftover keys, store them separately

Hashing

As I mentioned previously, I used hashWithSalt:

value = hashWithSalt currentLevel key `mod` (gamma * currentLength)

The role of gamma is to control the amount of “slack” in the bitvector, since sometimes making it larger than strictly necessary can reduce the probability of collisions. More on this later.

Populating the bitvector

The approach described in the paper involves using an auxiliary bitvector \(C\) to keep track of collisions:

Initialise two bitvectors \(B\) and \(C\) with \(0\)s
When setting an index \(i\):
1. If \(B[i] \equiv 0\) and \(C[i] \equiv 0\) then set \(B[i] = 1\)
2. If \(B[i] \equiv 1\) then set \(B[i] = 0\) and \(C[i] = 1\)
3. If \(B[i] \equiv 0\) and \(C[i] \equiv 1\) then do nothing

Lookup

To actually use our hash function, we can do the following:

For each level:
1. Hash the key and check if the corresponding index is set
2. If so, find the rank
3. If not, increment the level count and repeat
Otherwise check the leftovers

Example

Let’s look at a small example. The Bondi to Coogee walk here in Sydney passes through the following beaches:

Bondi
Tamarama
Bronte
Clovelly
Gordons Bay
Coogee

and we can use these as keys for a minimal perfect hash function.

Construction

The results of the first iteration are

Level 0

Low-effort Dependency Pinning with Nix Flakes

2024-02-09

Posted on 9 February 2024

Tags: programming, nix

Back in 2018 I wrote a blog post about pinning nixpkgs which describes an approach I’ve used happily and successfully since then to manage dependencies (and not just nixpkgs) for small projects. In short, it involves

versions.json, a JSON file storing dependency information
updater, an updater script
pkgs.nix, a Nix expression that makes each dependency available

Here’s what each of those files might look like:

versions.json

Binary Trees To Hash Array Mapped Tries, Step by Step

2023-10-07

Posted on 7 October 2023

Tags: programming, haskell

Hash Array Mapped Tries (HAMTs) are a persistent data structure used to implement hashmaps. They’re heavily used in Clojure and used to be the backbone of Haskell’s aeson library until relatively recently. I’ve written about HAMTs before but wanted to try a different approach: starting with a binary tree (or something close to it) and then making a series of straightforward modifications until we end up with the implementation detailed there.

Let’s start with some language extensions and imports:

{-# LANGUAGE GeneralizedNewtypeDeriving #-}
{-# LANGUAGE TypeSynonymInstances #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE RankNTypes #-}


import Data.Bits             (Bits (bit, complement, popCount, shiftR, (.&.), (.|.), testBit),
                              FiniteBits (finiteBitSize))
import Data.ByteArray.Hash   (FnvHash32 (..), fnv1Hash)
import Data.ByteString.Char8 (pack)
import Data.Char             (intToDigit)
import Data.Semigroup        ((<>))
import Data.Vector           (Vector, drop, singleton, take, replicate, (!), (//))
import Data.Word             (Word16, Word32)
import Numeric               (showIntAtBase)
import Prelude               hiding (drop, lookup, take, replicate)
import qualified             Prelude
import System.TimeIt         (timeIt)
import Text.Show.Pretty      (pPrint)

I think it’s useful to be able to visualise these structures, for which we need some more imports:

import IHaskell.Display.Graphviz
import Control.Monad.Trans.State.Strict
import Control.Monad.Trans.Writer.CPS
import Control.Monad.Trans.Class
import qualified Data.Vector as Vector
import Data.List (intercalate, intersperse, foldl')

I’m going to define some instances for pretty-printing hashes:

newtype Binary a = Binary a
    deriving (Enum, Ord, Real, Integral, Eq, Num, Bits, FiniteBits)

instance (FiniteBits a, Show a, Integral a) => Show (Binary a) where
    show (Binary n) = let
        str = showIntAtBase 2 intToDigit n ""
        size = finiteBitSize n
        in Prelude.replicate (size - length str) '0' <> str

type Hash = Binary Word32

class Hashable a where
    hash :: a -> Hash

One can think of hashing as mapping values of some type to fixed-size values of another type, and in this case I’ve decided to hash Ints to themselves for demonstration purposes. I would strongly recommend against doing this in production, but when explaining how these trees are constructed it’s handy to be able to immediately tell what the hash of some Int will be.

instance Hashable String where
    hash s = let
        FnvHash32 h = fnv1Hash (pack s)
        in Binary h

instance Hashable Int where
    hash int = Binary (fromIntegral int)

I’m also defining some helpers so that we can generate DOT representations and use ihaskell-graphviz to display each of the structures defined here:

Graphviz helper functions

Using `ghc-syntax-highlighter` with Hakyll

2023-01-29

Posted on 29 January 2023

Tags: haskell, programming

In 2018, Mark Karpov announced ghc-syntax-highlighter, a project which uses GHC’s own lexer to tokenise Haskell source code for the best possible syntax highlighting. I thought this was extremely cool, and really wanted to use it for this blog. Unfortunately, this is what the post had to say about pandoc, which Hakyll uses to process Markdown:

skylighting is what Pandoc uses btw. And from what I can tell it’s hardcoded to use only that library for highlighting, so some creativity may be necessary to get it work.

I briefly looked into this and reached the same conclusion (and as of this writing it is still the case) so, as a deeply uncreative individual, I sighed deeply and resigned myself to never knowing this particular joy.

Until, just a few days ago, I read this lovely blog post by Tony Zorman about customising Hakyll’s syntax highlighting which included this gem of a sentence in the very first paragraph:

Using pygmentize as an example, I will show you how you can swap out pandoc’s native syntax highlighting with pretty much any third party tool that can output HTML.

And in fact this is an accurate description of what follows. This sounds like exactly what I want to do, and between this and Mark’s mmark-ext (which implements ghc-syntax-highlighter support as an extension for mmark) I was able to successfully follow the instructions to get ghc-syntax-highlighter working with my blog. Let me walk you through what I did.

Here are the language extensions I will be using:

{-# LANGUAGE LambdaCase        #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE ViewPatterns      #-}

and these additional imports:

import           GHC.SyntaxHighlighter (Token(..), tokenizeHaskell)
import           Text.Blaze.Html.Renderer.Text (renderHtml)
import           Text.Pandoc.Definition (Block (CodeBlock, RawBlock), Pandoc)
import           Text.Pandoc.Walk (walk)
import qualified Text.Blaze.Html5 as H
import qualified Text.Blaze.Html5.Attributes as A

I chose to use blaze-html since it is already a transitive dependency of pandoc and using it has no impact on our dependency tree.

Tony uses walkM since an external program (pygmentize) is involved, but since we are working with pure Haskell code we can get away with just walk:

ghcSyntaxHighlight :: Pandoc -> Pandoc
ghcSyntaxHighlight = walk $ \case
    CodeBlock (_, (isHaskell -> True):_, _) (tokenizeHaskell -> Just tokens) ->
        RawBlock "html" . L.toStrict . renderHtml $ formatHaskellTokens tokens
    block -> block
    where isHaskell = (== "haskell")

This only matches Haskell code blocks which tokenizeHaskell is able to successfully tokenise and otherwise falls back on existing pandoc behaviour.

formatHaskellTokens generates markup very similarly to what pandoc already does:

formatHaskellTokens :: [(Token, T.Text)] -> H.Html
formatHaskellTokens tokens =
    H.div H.! A.class_ "sourceCode" $
        H.pre H.! A.class_ "sourceCode haskell" $
            H.code H.! A.class_ "sourceCode haskell" $
                mapM_ tokenToHtml tokens

tokenizeHaskell produces a list of pairs of the token type (KeywordToken, VariableToken, etc.) and the matched text, and the tokenToHtml (adapted from mmark-ext) function creates a span element with the appropriate class name for our CSS to style:

tokenToHtml :: (Token, T.Text) -> H.Html
tokenToHtml (tokenClass -> className, text) =
    H.span H.!? (not $ T.null className, A.class_ (H.toValue className)) $
        H.toHtml text

tokenClass (also adapted from mmark-ext) outputs the appropriate class name for each token, and I made only minor changes for styling purposes:

tokenClass :: Token -> T.Text
tokenClass = \case
    KeywordTok -> "kw"
    PragmaTok -> "pp" -- Preprocessor
    SymbolTok -> "ot" -- Other
    VariableTok -> "va"
    ConstructorTok -> "dt" -- DataType
    OperatorTok -> "op"
    CharTok -> "ch"
    StringTok -> "st"
    IntegerTok -> "dv" -- DecVal
    RationalTok -> "dv" -- DecVal
    CommentTok -> "co"
    SpaceTok -> ""
    OtherTok -> "ot"

Finally we have to actually use ghcSyntaxHighlight, for which we define a replacement for pandocCompiler called (imaginatively) customPandocCompiler and use it everywhere:

customPandocCompiler :: Compiler (Item String)
customPandocCompiler =
    pandocCompilerWithTransform
        defaultHakyllReaderOptions
        defaultHakyllWriterOptions
        ghcSyntaxHighlight

Again, since we are using pure functions, we can get away with pandocCompilerWithTransform instead of pandocCompilerWithTransformM.

And we’re done! I also had to tweak my CSS slightly since pandoc was generating a span for each line of source code instead of each token like ghc-syntax-highlighter does. For the complete listing, see here.

Updating IHaskell to a Newer GHC

2021-05-02

Posted on 2 May 2021

Tags: haskell, programming, nix

As the current maintainer of IHaskell, I see myself as having one primary responsibility: keeping it up-to-date with newer GHC releases. The chain of events that led to me becoming a maintainer started with the then-latest version of IHaskell not having support for GHC 8.0, and I still remember how frustrated I felt when dealing with this limitation.

Since then I’ve had the opportunity to add GHC 8.2, 8.4, 8.6, 8.8, 8.10, and now 9.0 support, but because I only have to do this every 6 months or so (at the earliest) I promptly forget the details of this work afterwards and have to spelunk through old, often heavily amended, commits to rediscover what past me (who is notoriously bad at documentation) did.

At the time of writing, GHC 9.2 is expected to be released soon and I don’t want to forget everything I’ve just (re)learned when that happens. Additionally, it is conceivable that at some point someone other than me would like to take a crack at updating IHaskell to the newest version of GHC. This blog post details the steps I took to make these tasks easier in the future.

Building IHaskell’s dependencies

I should start by saying that my current approach relies heavily on Nix and the infrastructure available in nixpkgs. If you don’t want to use Nix for whatever reason the general ideas might still translate to whatever method you use instead but the details will almost certainly vary widely.

The objective of this first step is to get us to the point where all of IHaskell’s dependencies are building, so that we can then focus on ghc-parser, ipython-kernel, and ihaskell exclusively.

I start with a version of Nixpkgs that has the necessary GHC version and package overrides to minimise work. As of this writing, the Nixpkgs maintainers base the Haskell package set they use on Stackage Nightlies with overrides added from head.hackage. Updates seem to go into the haskell-updates branch first and are then periodically merged into master. I started with this commit but it had issues with building alex that I sent pull requests for to Nixpkgs and to the project. In the meantime it’s very easy to make any required changes to a fork or a local copy of Nixpkgs. I start by copying release.nix from the IHaskell project root and changing the reference to Nixpkgs:

Changing Nixpkgs

Writing GitHub Secrets to a Repository You Don't Own

2020-05-04

Posted on 4 May 2020

Tags: ci, programming

I’ve been having a lot of fun migrating the CI systems of my repositories to use GitHub Actions, but it’s been more difficult to do the same with projects that are owned by someone else because I don’t have access to the repository settings that would allow me to create secrets. This means that I can build and test those projects but not e.g. upload a Docker container as part of a successful build or upload artifacts somewhere else.

I’ve tried to work around this limitation by creating a separate repository that I own and using the cron functionality to do this on a schedule, but this is a poor substitute. I’ve been frustrated by this situation for a while, and while reading the documentation I noticed this interesting snippet:

If you are using the REST API to create secrets, anyone with write access to the repository can create secrets. For more information, see “GitHub Actions secrets API” in the GitHub Developer documentation.

Amazing! This makes it sound like it’s purely a UI issue. So emboldened, I was able to create and use my secrets only a couple of hours later by poking at the GitHub API.

I’m going to go ahead and write down the steps I took in order to make this happen, because this seems like the kind of thing I might have to do more than once and it’s just fiddly enough that I will quickly forget if I don’t.

The first thing I need is a GitHub Personal Access Token with the repo scope, which I can create from this page.

The next thing to do is to retrieve the public key for the relevant repository:

$ curl -H "Authorization: token $TOKEN" https://api.github.com/repos/gibiansky/IHaskell/actions/secrets/public-key
{
  "key_id": "$KEY_ID",
  "key": "$PUBLIC_KEY"
}

Then I can see what secrets are available:

$ curl -H "Authorization: token $TOKEN" https://api.github.com/repos/gibiansky/IHaskell/actions/secrets
{
  "total_count": 0,
  "secrets": [
  ]
}

The secrets need to be encrypted, and there is sample code for doing this in Python:

from base64 import b64encode
from nacl import encoding, public

def encrypt(public_key: str, secret_value: str) -> str:
    """Encrypt a Unicode string using the public key."""
    public_key = public.PublicKey(public_key.encode("utf-8"), encoding.Base64Encoder())
    sealed_box = public.SealedBox(public_key)
    encrypted = sealed_box.encrypt(secret_value.encode("utf-8"))
    return b64encode(encrypted).decode("utf-8")

I added a Nix shebang line and decided to generate all the encrypted secrets I needed:

secret.py

#! /usr/bin/env nix-shell
#! nix-shell -i python
#! nix-shell -p "python3.withPackages (p: [ p.pynacl ])"

from base64 import b64encode
from nacl import encoding, public

def encrypt(public_key: str, secret_value: str) -> str:
    """Encrypt a Unicode string using the public key."""
    public_key = public.PublicKey(public_key.encode("utf-8"), encoding.Base64Encoder())
    sealed_box = public.SealedBox(public_key)
    encrypted = sealed_box.encrypt(secret_value.encode("utf-8"))
    return b64encode(encrypted).decode("utf-8")

public_key = "$PUBLIC_KEY"

print("CACHIX_SIGNING_KEY=", encrypt(public_key, "$CACHIX_SIGNING_KEY"))
print("DOCKER_USERNAME=", encrypt(public_key, '$DOCKER_USERNAME'))
print("DOCKER_PASSWORD=", encrypt(public_key, '$DOCKER_PASSWORD'))

This was easy to run:

$ chmod +x secret.py
$ ./secret.py
CACHIX_SIGNING_KEY= $ENCRYPTED_CACHIX_SIGNING_KEY
DOCKER_USERNAME= $ENCRYPTED_DOCKER_USERNAME
DOCKER_PASSWORD= $ENCRYPTED_DOCKER_PASSWORD

And I chose to update the secrets manually with curl even though I could have automated it with requests or something similar (which I might if I have to do this again soon), for example:

$ curl -X PUT -H "Authorization: token $TOKEN" -H "Content-Type: application/json" -i https://api.github.com/repos/gibiansky/IHaskell/actions/secrets/CACHIX_SIGNING_KEY -d '{"key_id": "$KEY_ID", "encrypted_value": "$ENCRYPTED_CACHIX_SIGNING_KEY"}'

Finally I can check that the secrets were created correctly:

$ curl -H "Authorization: token $TOKEN" https://api.github.com/repos/gibiansky/IHaskell/actions/secrets
{
  "total_count": 3,
  "secrets": [
    {
      "name": "CACHIX_SIGNING_KEY",
      "created_at": "2020-05-03T04:45:07Z",
      "updated_at": "2020-05-03T04:45:07Z"
    },
    {
      "name": "DOCKER_PASSWORD",
      "created_at": "2020-05-03T04:49:59Z",
      "updated_at": "2020-05-03T04:49:59Z"
    },
    {
      "name": "DOCKER_USERNAME",
      "created_at": "2020-05-03T04:48:52Z",
      "updated_at": "2020-05-03T04:48:52Z"
    }
  ]
}

I hope these instructions are useful, future me!

Getting Along with JavaScript

2019-10-29

Posted on 29 October 2019

Tags: programming, haskell, nix

For the last couple of weeks, I’ve been obsessed with the idea of running Haskell in the browser. I know this is possible, because this is what I do at work every day, but the applications I work on professionally are complex beasts with Haskell backends and dedicated servers making them available to users. I’m looking for something lighter that I can serve statically using GitHub Pages or Glitch, so I can plop some code on a webpage and never worry about hosting ever again.

My first instinct was to reach for a tool like Obelisk, which bills itself as “an easy way to develop and deploy your Reflex project”. Although it does work as advertised(!), it is geared towards the needs of the large apps I mentioned above. It prerenders webpages where possible to make projects as snappy as possible, works best within the confines of the Obelisk libraries, and assumes at least one NixOS target that will host your website, all of which mean it doesn’t yet scale down to my comparatively modest needs. It is possible to use Obelisk anyway, but I found myself using too few of its features to justify the effort, and I decided to move down a level and use Reflex Platform directly, which is a set of changes and overrides to a revision of Nixpkgs to best support building full-stack and mobile Haskell applications.

If you’d like to follow along, I have the code available at this gist with each revision representing a step in the progression.

Setting up reflex-platform

I like to use the updater script described in a previous blog post, so I’ll start by copying that over and creating a versions.json with the following contents:

versions.json

You Won’t Believe This One Weird CPU Instruction!

2019-09-08

Posted on 8 September 2019

Tags: programming

Translated to Russian by Babur Muradov and Uzbek by Leonid Popov.

This is a pseudo-transcript of a presentation I did at !!Con 2019.

Most CPU architectures in use today have an instruction called popcount, short for “population count”. Here’s what it does: it counts the number of set bits in a machine word. For example (assuming 8-bit words for simplicity), popcount(00100110) is 3 and popcount(01100000) is 2.

You might be wondering, like I was, if there’s more to this instruction, but that’s all it does! This doesn’t seem very useful, right?

I thought this might be a recent addition for some hyperspecialised use case, but it has in fact been present in CPU architectures since at least 1961:

1961: IBM Stretch
1964: CDC 6000
1975: Cray-1
2005: SPARC
2005: ARM NEON
2007: AMD K10
2008: Intel Nehalem

So what’s going on?

The NSA Instruction

popcount is also known as “The NSA Instruction”, and a very entertaining thread on comp.arch discusses its uses inside and outside cryptography. It is rumoured that it was originally added to CPU instructions at the behest of the NSA. As this archived email thread puts it:

It was almost a tradition that one of the first of any new faster CDC machine was delivered to a “good customer” - picked up at the factory by an anonymous truck, and never heard from again.

This makes for a great story, but what were they using it for?

One measure of information content is the Hamming weight, which is the number of symbols in a string that are different from the zero-symbol of the alphabet. For a binary string, this is exactly popcount!

As explained here, the NSA wanted to do cryptanalysis on intercepted messages, and since the CDC 6000 had 60-bit words, one word was enough to store most alphabets they were interested in. They were able to:

Split a message into lines
Set a bit for each unique character they encountered per line
Use popcount to count the distinct characters
Use the count as a hash for further cryptanalysis

Curiously, popcount seems to have disappeared from instruction sets between the mid-1970s and the mid-2000s, so there has to be more to it than cryptographic applications to explain its return. What else can it be used for?

Error Correction

Related to the concept of Hamming weight is Hamming distance, which is the number of differing positions between two strings of identical length. For two binary strings x and y, this is just the popcount of them XORed together. For example:

00100110
01100000 ^
--------
01000110

popcount(01000110) = 3

For telecommunications applications, this helps us calculate the signal distance, where a known word is sent over the wire and the number of flipped bits are counted to provide an estimate of the error introduced by transmission.

We can then design an error-correcting code accordingly, e.g. if we want to be robust against up to 2 flipped bits, our code words need to differ in Hamming distance by at least 5.

Binary Convolutional Neural Networks

And now for something completely different: binary convolutional neural networks! But first, what are they?

Binary means that we’re using matrices consisting of only the values +1 (coded as 1) and -1 (coded as 0), as opposed to 32-bit floating-point values.
Convolutional means matrix multiplication is involved?
Neural networks are systems inspired by animal brains (I’m a bit hazy on this part).

In summary, we have to do binary matrix multiplication. But what’s special about binary matrices?

Ordinary matrix multiplication on 32-bit values is a good fit on desktop computers with powerful CPUs and GPUs, but increasingly we also want to do useful work on smaller and simpler devices, such as smartphones, routers, smartwatches, etc. We can decompose these more complex matrices into layers of binary matrices, and these resulting matrices are so much easier to store and operate on that we are better off even though there are more layers.

Where does popcount come into play? It’s used to calculate the dot product of two binary matrices:

a = xnor(x, y)
b = popcount(a)
c = len(a)
dot(x, y) = 2 × b − c

More details are available here and here.

Chess Programming

Many chess programs store data using a bitboard representation, which conveniently fits into a 64-bit word. Population Count has been used to perform meaningful operations with this representation, such as calculating the mobility of a piece.

Molecular Fingerprinting

This is related to the notion of Hamming distance above: molecules are hashed in some way and compared (with popcount) to determine how similar they are. More details on that here.

Hash Array Mapped Tries

This is where I first learned about popcount! The HAMT is a data structure (pioneered by Phil Bagwell) that can store a very large number of values (usually 32 or 64) in an array at each node of the trie. However, allocating memory for a 32 or 64-element array every time can be incredibly wasteful, especially if the array only actually contains a handful of elements. The solution is to add a bitmask in which the number of bits that are set corresponds to the number of elements in the array, which allows the array to grow and shrink as required. Calculating the index for a given element efficiently can then be done using popcount. You can learn more about how they work from this blog post, where I implement them myself.

Succinct Data Structures

This is an exciting new area of research that focuses on how to store data in as little space as possible, without having to decompress it in order to do useful work. One technique is to think in terms of arrays of bits (bitvectors), which can be queried using two operations:

rank(i) counts the number of bits set upto the ith index in the bitvector
select(i) finds the index where the ith ranked bit is set

Making these operations efficient on large bitvectors requires constructing an index and using it effectively, both involving popcount. There’s a good overview of the RRR index here, and as far as I can tell the current state-of-the-art approach is described in Space-Efficient, High-Performance Rank & Select Structures on Uncompressed Bit Sequences.

Compiler Optimisations

popcount has become so pervasive that both GCC and Clang will detect an implementation of popcount and replace it with the built-in instruction. Imagine Clippy going “I see you are trying to implement popcount, let me go ahead and fix that for you”! The relevant LLVM code is here. Daniel Lemire points to this as an example of the surprising cleverness of modern compilers.

Conclusion

From beginnings shrouded in mystery, popcount has emerged as a generally useful, if slightly unusual, CPU instruction. I love how it ties together such different fields of computing, and I wonder how many other similarly weird instructions are out there. If you have a favourite, I’d love to hear about it!