In principle my idea is nothing fancy:

1. download all the required dependencies from hackage to the disk and extract them in a single directory,
2. determine the order in which they need to be installed,
3. build each library with GHC HEAD, resolving the build errors if necessary
4. register each library with GHC HEAD (see Appendix below)

Doing these things for the first time was very tedious and took me about 2-3 hours. Determining package dependencies was probably the most time consuming. Resolving build errors wasn’t that bad, though there were a couple of difficulties. It turned out that many packages put an upper bound on the version of the base package and removing these dependency is the only change required to build that package.

The key to my solution is that once you figure out in what order packages should be installed and remove the build errors, you can write a shell script that builds and installs packages automatically. This means that after installing GHC HEAD in a sandbox (see Appendix below) you can run the script to build and install all the packages. This will give you a fully working GHC installation in which you can write Criterion benchmarks for new features that you implemented in the compiler. Here’s what the script looks like (full version available here):

#!/bin/bash
 
PKGS="\
primitive-0.5.0.1 \
vector-0.10.0.1 \
dlist-0.5 \
vector-algorithms-0.5.4.2 \
..." # more packages in this list
 
if [[ $# -gt 1 ]]; then
    echo "Too many parameters"
    exit
elif [[ $# -eq 1 ]]; then
    if [[ $1 == "clean" ]]; then
        echo -n "Cleaning"
        for i in $PKGS
        do
            echo -n "."
            cd $i
            rm -rf dist
            rm -f Setup Setup.o Setup.hi
            cd ..
        done
        echo "done"
    else
        echo "Invalid parameter: $1"
        exit
    fi
else
    for i in $PKGS
    do
        echo "Installing package $i"
        cd $i
        ((if [[ -f Setup.lhs ]]; then ghc Setup.lhs; else ghc Setup.hs; fi) && \
            ./Setup configure --user --enable-shared \
            && ./Setup build && ./Setup install) \
            || exit
        cd ..
    done
fi

The script is nothing elaborate. Running without any parameters will build and install all packages on the list. If you run it with “clean” parameter it will remove build artefacts from package directories. If for some reason the script fails –  e.g. one of the libraries fails to build – you can comment out already installed libraries so that the script resumes from the point it previously stopped.

Summary

Using the approach described above I can finally write criterion benchmarks for GHC HEAD. There are a couple of considerations though:

• things are likely to break as HEAD gets updated. Be prepared to add new libraries as dependencies, change compilation parameters or fix new build errors,
• since some time you need to pass --enable-shared flag to cabal configure when building a shared library. This causes every library to be compiled twice. I don’t know if there’s anything one can do about that,
• you need to manually download new versions of libraries,
• fixing build errors manually may not be easy,
• rerunning the script when something fails may be tedious,
• changes in HEAD might cause performance problems in libraries you are using. If this goes unnoticed the benchmarking results might be invalid (I think this problem is hypothetical).

You can download my script and the source code for all the modified packages here. I’m not giving you any guarantee that it will work for you, since HEAD changes all the time. It’s also quite possible that you don’t need some of the libraries I’m using, for example Repa.

Appendix: Sandboxing GHC

For the above method to work effectively you need to have a sandboxed installation of GHC. There are tools designed for sandboxing GHC (e.g. hsenv) but I use a method described here. It’s perfectly suited for my needs. I like to have full manual control when needed but I also have this shell script to automate switching of sandboxes:

#!/bin/bash
 
SANDBOX_DIR="/path/to/ghc-sandbox/"
ACTIVE_SYMLINK="${SANDBOX_DIR}active"
STARTCOLOR="\e[32m";
ENDCOLOR="\e[0m";
 
active_link_name=`readlink ${ACTIVE_SYMLINK}`
active_name=`basename ${active_link_name}`
 
if [[ $# -lt 1 ]]; then
  for i in `ls ${SANDBOX_DIR}`; do
    if [[ $i != "active" ]]; then
      if [[ $i == $active_name ]]; then
        echo -e "* $STARTCOLOR$i$ENDCOLOR"
      else
        echo "  $i"
      fi
    fi
  done
  exit
fi
 
for i in `ls ${SANDBOX_DIR}`; do
  if [[ $i == $1 ]]; then
    cd $SANDBOX_DIR
    rm ${ACTIVE_SYMLINK}
    ln -s $1 ${ACTIVE_SYMLINK}
    exit
  fi
done
 
echo "Sandbox $1 not found"

It displays list of sandboxes when run without any parameter (the active sandbox is displayed in green and marked with an asterisk) and switches the active sandbox when given a command-line parameter. Together with bash auto completion feature switching between different GHC versions is a matter of seconds.

Here is my simple piece of code:

sumSqrL :: [Int] -> Int
sumSqrL = sum . map (^2) . filter odd

It takes a list of Ints, removes all even numbers from it, squares the remaining odd numbers and computes the sum. This is idiomatic Haskell code: it uses built-in list processing functions from the standard Prelude and relies on function composition to get code that is both readable and modular. So how can we make that faster? The simplest thing to do is to switch to a more efficient data structure, namely an unboxed Vector:

import Data.Vector.Unboxed as U
 
sumSqrV :: U.Vector Int -> Int
sumSqrV = U.sum . U.map (^2) . U.filter odd

The code practically does not change, except for the type signature and namespace prefix to avoid clashing with the names from Prelude. As you will see in a moment this code is approximately three times faster than the one working on lists.

Can we do better than that? Yes, we can. The code below is three times faster than the one using Vector, but there is a price to pay. We need to sacrifice modularity and elegance of the code:

sumSqrPOp :: U.Vector Int -> Int
sumSqrPOp vec = runST $ do
  let add a x = do
        let !(I# v#) = x
            odd#     = v# `andI#` 1#
        return $ a + I# (odd# *# v# *# v#)
  foldM` add 0 vec -- replace ` with ' here

This code works on an unboxed vector. The add function, used to fold the vector, takes an accumulator a (initiated to 0 in the call to foldM') and an element of the vector. To check parity of the element the function unboxes it and zeros all its bits except the least significant one. If the vector element is even then odd# will contain 0, if the element is odd then odd# will contain 1. By multiplying square of the vector element by odd# we avoid a conditional branch instruction at the expense of possibly performing unnecessary multiplication and addition for even elements.

Let’s see how these functions compile into Core intermediate language. The sumSqrV looks like this:

$wa =
  \vec >
    case vec of _ { Vector vecAddressBase vecLength vecData ->
    letrec {
      workerLoop =
        \index acc ->
          case >=# index vecLength of _ {
            False ->
              case indexIntArray# vecData (+# vecAddressBase index)
              of element { __DEFAULT ->
              case remInt# element 2 of _ {
                __DEFAULT ->
                  workerLoop (+# index 1) (+# acc (*# element element));
                0 -> workerLoop (+# index 1) acc
              }
              };
            True -> acc
          }; } in
    workerLoop 0 0
    }

while sumSqrPOp compiles to:

$wsumSqrPrimOp =
  \ vec ->
    runSTRep
      ( (\ @ s_X1rU ->
          case vec of _ { Vector vecAddressBase vecLength vecData ->
          (\ w1_s37C ->
             letrec {
               workerLoop =
                 \ state index acc ->
                   case >=# index vecLength of _ {
                     False ->
                       case indexIntArray# vecData (+# vecAddressBase index)
                       of element { __DEFAULT ->
                       workerLoop
                         state
                         (+# index 1)
                         (+# acc (*# (*# (andI# element 1) element) element))
                       };
                     True -> (# state, I# acc #)
                   }; } in
             workerLoop w1_s37C 0 0)
          })
       )

I cleaned up the code a bit to make it easier to read. In the second version there is some noise from the ST monad, but aside from that both pieces of code are very similar. They differ in how the worker loop is called inside the most nested case expression. First version does a conditional call of one of the two possible calls to workerLoop, whereas the second version does an unconditional call. This may seem not much, but it turns out that this makes the difference between the code that is comparable in performance with C and code that is three times slower.

Let’s take a look at the assembly generated by the LLVM backend. The main loop of sumSqrV compiles to:

LBB1_4:
    imulq    %rdx, %rdx
    addq     %rdx, %rbx
.LBB1_1:
    leaq    (%r8,%rsi), %rdx
    leaq    (%rcx,%rdx,8), %rdi
    .align  16, 0x90
.LBB1_2:
    cmpq    %rax, %rsi
    jge     .LBB1_5
    incq    %rsi
    movq    (%rdi), %rdx
    addq    $8, %rdi
    testb   $1, %dl
    je      .LBB1_2
    jmp     .LBB1_4

While the main loop of sumSqrPOp compiles to:

.LBB0_4:
    movq    (%rsi), %rbx
    movq    %rbx, %rax
    imulq   %rax, %rax
    andq    $1, %rbx
    negq    %rbx
    andq    %rax, %rbx
    addq    %rbx, %rcx
    addq    $8, %rsi
    decq    %rdi
    jne     .LBB0_4

No need to be an assembly expert to see that the second version is much more dense.

I promised you comparison with C. Here’s the code:

long int c_sumSqrC( long int* xs, long int xn ) {
  long int index   = 0;
  long int result  = 0;
  long int element = 0;
 Loop:
  if (index == xn) goto Return;
  element = xs[index];
  index++;
  if ((0x1L & element) == 0) goto Loop;
  result += element * element;
  goto Loop;
 Return:
  return result;
}

You’re probably wondering why the hell did I use gotos. The reason is that the whole idea of this sum-square-of-odds function was taken from the paper “Automatic transformation of series expressions into loops” by Richard Waters and I intended to closely mimic the solution produced by his fusion framework.

I used criterion to compare the performance of four presented implementations: based on list, base on vector, based on vector using foldM+primops and C. I used FFI to call C implementation from Haskell so that I can benchmark it with criterion as well. Here are the results for a list/vector containing one million elements:

C version is still faster than the one based on primops by about 8%. I think this is a very good achievement given that the version based on Vector library is three times slower.

A few words of summary

The vector library uses stream fusion under the hood to optimize the code working on vectors. In the blog posts I mentioned in the beginning Don Stewart talks a bit about stream fusion, but if you want to learn more you’ll probably be interested in two papers: Stream Fusion. From Lists to Streams to Nothing at All and  Haskell Beats C Using Generalized Stream Fusion. My sumSqrPOp function, although as fast as C, is in fact pretty ugly and I wouldn’t recommend anyone to write Haskell code in such a way. You might have realized that while efficiency of sumSqrPOp comes from avoiding the conditional instruction within the loop, the C version does in fact use the conditional instruction within the loop to determine the parity of the vector element. The interesting thing is that this conditional is eliminated by gcc during the compilation.

As you can see it might be possible to write Haskell code that is as fast as C. The bad thing is that to get efficient code you might be forced to sacrifice the elegance and abstraction of functional programming. I hope that one day Haskell will have a fusion framework capable of doing more optimisations than the frameworks existing today and that we will be able to have both the elegance of code and high performance. After all, if gcc is able to get rid of unnecessary conditional instructions then it should be possible to make GHC do the same.

A short appendix

To dump Core produced by GHC use -ddump-simpl flag during compilation. I also recommend using -dsuppress-all flag, which suppresses all information about types – this makes the Core much easier to read.

To dump the assembly produced by GHC use -ddump-asm flag. When compiling with LLVM backend you need to use -keep-s-files flag instead.

To disassemble compiled object files (e.g. compiled C files) use the objdump -d command.

Update – discussion on Reddit

There was some discussion about this post on reddit and I’d like to address some of the objections that were raised there and in the comments below.

Mikhail Glushenkov pointed out that the following Haskell code produces the same result as my sumSqrPOp function:

sumSqrB :: U.Vector Int -> Int
sumSqrB = U.sum . U.map (\x -> (x .&. 1) * x * x)

I admit I didn’t notice this simple solution and could have come with a better example were such a solution would not be possible.

There was a request to compare performance with idiomatic C code, because the C code I have shown clearly is not idiomatic. So here’s the most idiomatic C code I can come up with (not necessarily the fastest one):

long int c_sumSqrC( long int* xs, long int xn ) {
  long int result = 0;
  long int i = 0;
  long int e;
  for ( ; i < xn; i++ ) {
    e = xs[ i ];
    if ( e % 2 != 0 ) {
      result += e * e;
    }
  }
  return result;
}

The performance turns out to be the same as before (“Bits” represents Mikhail Glushenkov’s solution, “C” now represents the new C code):

There was a suggestion to use the following C code:

for(int i = 0; i < xn; i++) {
    result += xs[i] * xs[i] * (xs[i] & 1);
}

Author claims that this code is faster than the version I proposed, but I cannot confirm that on my machine – I get results that are noticeably slower (2.7ms vs 1.7ms for vectors of 1 million elements). Perhaps this comes from me using GCC 4.5, while the latest available version is 4.8.

Finally, there were questions about overhead added by calling C code via FFI. I was concerned with this also when I first wanted to benchmark my C code via FFI. After making some experiments it turned out that this overhead is so small that it can be ignored. For more information see this post.

My first attempts of installing GHC and the Haskell Platform a year ago relied on using packages from my distribution’s repository. This quickly turned out to be problematic so I decided for a direct installation of the Haskell Platform. This worked perfectly fine except for the fact that Haskell packages were installed in different subdirectories of /usr/local, which lead to a bit of a mess and problems with controlling what is installed where (this is useful if you want to remove a package). So the second time I was installing Haskell Platform I was smarter and refined the whole process. This time I confined the installation to a single directory so that both GHC and all the packages are located in a single, easy to find place.

Yesterday I figured out it would be great to get a new version of GHC. GHC 7.6.1 was released on 6th September 2012 and the updated 7.6.2 version is only two weeks old. While GHC 7.6.1 has been out for over 5 months it is still not part of the Haskell Platform and it won’t be for the next three months. That’s too long a wait for me so I decided to send the Platform to /dev/null and just install GHC and its environment from scratch.

My plan to install GHC from precompiled binaries went up the spout:

This build requires libgmp.so.3.

Watwatwat? Now what is that supposed to mean? Previously released binaries didn’t depend on one particular version of libgmp library. Of course my system has libgmp.so.10 and any attempt to install an older version results in breaking package dependencies. I downloaded binaries anyway and tried to run them:

[killy@xerxes : ~/ghc-7.6.2/ghc/stage2/build/tmp] ./ghc-stage2 --interactive
 ./ghc-stage2: error while loading shared libraries: libgmp.so.3: cannot open shared object file: No such file or directory

OK, so that requirement is true – you need the exact version of libgmp. So what now? I know! Compilation from sources! I’ve been hacking on GHC recently so I already have sources on my drive. Unfortunately it turned out that after switching GHC repo and all its subrepos to ghc-7.6 branch I get some compilation errors. I wasn’t in the mood for debugging this so I switched everything back to master and downloaded the source snapshot. From now on things are easy, assuming that you already have an older version of GHC on your system. After extracting the sources I copied $(TOP)/mk/build.mk.sample to $(TOP)/mk/build.mk ($(TOP) refers to directory containing GHC sources) and uncommented the line BuildFlavour = perf-llvm. This gives me fully optimized build using LLVM. Now the compilation:

perl boot
./configure --prefix=/usr/local/ghc-7.6.2
make

This will build GHC and prepare it for installation in /usr/local/ghc-7.6.2. Fully optimized build takes much over an hour on all 4 cores. After the build is done all one needs to do is run make install as root. At this stage old GHC can be removed. You of course need to add /usr/local/ghc-7.6.2/bin to PATH environmental variable. As I already have mentioned I have the habit of installing all the packages system-wide in a single directory. For that I need to edit /root/.cabal/config file by adding the following entry:

install-dirs global
    prefix:/usr/local/ghc-7.6.2

All that is left now is installing cabal-install. Grab the sources from hackage, extract them and run (as root) sh bootstrap.sh --global in the source directory. This installs cabal-install with its dependencies. Now you can start installing other packages that you need (a.k.a. compile the World).

This completes Yet Another Installation of GHC.

Taking compilers course is a good thing because it takes magic out of compilers.

It may have been put into words differently but this was the meaning. Having taken compilers course and learning about GHC’s internals I think this is so true. A compiler is no longer a black box. It’s just a very complicated program built according to some general rules. In this post I want to reveal a little bit of magic that GHC does behind the scenes when compiling your Haskell program.

Imagine you are writing an image processing algorithm and you want to check whether pixel coordinates x and y are within the image. That’s simple: if any of these coordinates is less than 0 or if x equals or exceeds image width or if y equals or exceeds image height than coordinates are not within the image. Here’s how we can express this condition in Haskell:

case (x < 0) || (x >= width) || (y < 0) || (y >= height) of
    False -> e1 -- do this when (x,y) is within the image
    True  -> e2 -- do that when (x,y) is outside of image

When compiling a Haskell program GHC uses language called Core as an intermediate representation. It’s a very simplified¹ form of Haskell that has only let and case expressions and type annotations. You don’t need any knowledge of Core to understand this post but if you want to learn more I suggest to start with a blog post by Gabriel Gonzalez and then take a look at Edward Z. Yang’s post, that also shows GHC’s other intermediate languages: STG and Cmm. You can see Core representation to which your program was transformed by passing -ddump-simpl flag to GHC (I also use -dsuppress-all to get rid of all type informations that obscure the output). If you compile above case expression with optimisations (pass the -O2 option to GHC) you end up with following Core representation:

case <# x 0# of _ {
  False ->
    case >=# x width of _ {
      False ->
        case <# y 0# of _ {
          False ->
            case >=# y height of _ {
              False -> e1;
              True  -> e2;
            };
          True -> e2;
        };
      True -> e2;
    };
  True -> e2;
}

GHC turned our infix comparison operators into prefix notation. It also unboxed integer variables. This can be noticed by # appended to integer literals and comparison operators. There’s also some syntactic change in case expressions: there are braces surrounding the branches, semicolon is used to delimit branches from each other and there is a mysterious underscore after the keyword of. We can rewrite this in a more familiar Haskell syntax (I will also reverse the order of True and False branches – it will be more readable):

case x < 0 of
  True  -> e2
  False ->
    case x >= width of 
      True  -> e2
      False ->
        case y < 0 of
          True  -> e2
          False ->
            case y >= height of
              True  -> e2
              False -> e1

The most noticeable thing however is that our original case expression has suddenly turned into four nested cases, which resulted in duplicating expression e2 four times. In this post I will show you how GHC arrived at this representation.

A bit of theory

There are some things you need to know in order to understand how GHC transformed the code. First is the definition of (||) operator (logical or):

(||) :: Bool -> Bool -> Bool
(||) x y = case x of
    True  -> True
    False -> y

When optimizations are turned on GHC performs inlining of short functions. This means that function calls are replaced by function definitions and this will be the case with (||) function.

Second thing is case-to-case code transformation. Imagine a code fragment like this:

case ( 
  case C of 
      B1 -> F1
      B2 -> F2
 ) of
    A1 -> E1
    A2 -> E2

We have a case expression nested within a scrutinee² of another case expression. You may be thinking that you would never write such a code and you are right. GHC however compiles programs by performing subsequent Core-to-Core transformations and such nesting of case expressions is often generated during that process (as we will see in a moment). If nested case expressions appear in the Core representation of a program they are turned inside out by case-of-case transformation: the nested case scrutinising C becomes the outer case expression and the outer case expression is pushed into branches B1 and B2:

case C of    
    B1 -> case F1 of
              A1 -> E1
              A2 -> E2
    B2 -> case F2 of
              A1 -> E1
              A2 -> E2

You see that code for E1 and E2 has been duplicated. This is worst case scenario. In real life programs one of the branches can often be simplified using case-of-known-constructor transformation. See what happens when expression returned by a branch of nested case is a constructor that is matched by outer case (A1 in this example):

case ( 
  case C of 
      B1 -> A1
      B2 -> F2
 ) of
    A1 -> E1
    A2 -> E2

After performing case-of-case transformation we end up with:

case C of    
    B1 -> case A1 of
              A1 -> E1
              A2 -> E2
    B2 -> case F2 of
              A1 -> E1
              A2 -> E2

In the first branch of outer case expression we are now matching A1 against A1. So we know that first branch will be taken and thus can get rid of this case expression reducing it to E1:

case C of    
    B1 -> E1
    B2 -> case F2 of
              A1 -> E1
              A2 -> E2

Thus only E1 was duplicated. We will see that happen a lot in a moment.

The fun begins

Knowing all this we can begin optimizing our code:

case (x < 0) || (x >= width) || (y < 0) || (y >= height) of
    False -> e1
    True  -> e2

First thing that happens is inlining of (||) operators. The call to (x < 0) || (x >= width) is replaced by definition of (||):

case ( 
    case (x < 0) of  -- this case comes from definition of ||
        True  -> True
        False -> (x >= width)
    ) || (y < 0) || (y >= height) of
    False -> e1
    True  -> e2

Let’s inline next use of (||):

case ( 
    case (  -- this case is introduced by inlining of second ||
        case (x < 0) of
            True  -> True
            False -> (x >= width)
        ) of 
        True  -> True
        False -> (y < 0)
    ) || (y >= height) of
    False -> e1
    True  -> e2

One more inlining and where done with (||):

case ( 
    case (  -- this case is introduced by inlining of last ||
        case (
            case (x < 0) of
                True  -> True
                False -> (x >= width)
            ) of 
            True  -> True
            False -> (y < 0)
        ) of 
        True  -> True
        False -> (y >= height)
    ) of
    False -> e1
    True  -> e2

We ended up with three cases nested as scrutinees of other cases – I told you this will happen. Now GHC will start applying case-of-case transformation to get rid of all this nesting. Let’s focus on two most internal cases for simplicity:

case (
    case (x < 0) of
        True  -> True
        False -> (x >= width)
    ) of 
    True  -> True
    False -> (y < 0)

Performing case-of-case transformation on them gives:

case (x > 0) of  -- nested case is floated out
    True -> 
        case True of  -- outer case is pushed into this branch...
            True  -> True
            False -> (y < 0)
    False -> 
        case (x >= width) of -- ...and into this branch
            True  -> True
            False -> (y < 0)

Looking at first nested case we see that case-of-known-constructor transformation can be applied:

case (x < 0) of
    True  -> True  -- case-of-known-constructor eliminated 
                   -- case expression in this branch
    False -> 
        case (x >= width) of
            True  -> True
            False -> (y < 0)

Now let’s put these cases back into our expression:

case ( 
    case ( 
        case (x < 0) of
            True  -> True
            False -> 
                case (x >= width) of
                    True  -> True
                    False -> (y < 0)
        ) of
        True  -> True
        False -> (y >= height)
    ) of
    False -> e1
    True  -> e2

Now we only have two cases nested as scrutinees of other case. Applying case-of-case one more time will get rid of the first nesting:

case ( 
    case (x < 0) of
        True ->  -- we can use case-of-known-constructor here
            case True of  
                True  -> True
                False -> (y >= height)
        False -> 
            case ( -- these nested cases weren't here before!
                case (x >= width) of
                    True -> True
                    False -> (y < 0)
                ) of
                True  -> True
                False -> (y >= height)
    ) of
    False -> e1
    True  -> e2

Hey, that’s something new here! We eliminated nested cases in one place only to introduce them in another. But we know what to do with nested cases – use case-of-case of course. Let’s apply it to the second branch and case-of-known-constructor to the first one:

case ( 
    case (x < 0) of
        True  -> True  -- case-of-known-constructor used here
        False ->       
            case (x >= width) of    -- case-of-case used here
                True -> 
                    case True of    -- what about this case?
                        True  -> True
                        False -> (y >= height)
                False -> 
                    case (y < 0) of
                        True  -> True
                        False -> (y >= height)
    ) of
    False -> e1
    True  -> e2

We just got another chance to perform case-of-known-constructor:

case ( 
    case (x < 0) of
        True  -> True
        False -> 
            case (x >= width) of
                True  -> True  -- case-of-known-constructor
                False -> 
                   case (y < 0) of
                       True -> True
                       False -> (y >= height)
    ) of
    False -> e1
    True  -> e2

We have on more nested case to eliminate. Let’s hit it:

case (x < 0) of
    True -> 
        case True of
            False -> e1
            True  -> e2
    False -> 
        case (
            case (x >= width) of
                True  -> True
                False -> 
                    case (y < 0) of
                        True -> True
                        False -> (y >= height)
                ) of
                False -> e1
                True  -> e2

Do you see how expressions e1 and e2 got duplicated? Let’s apply case-of-known-constructor in the first branch and case-of-case + case-of-known-constructor in the second one:

case (x < 0) of
    True  -> e2
    False -> 
        case (x >= width) of
            True  -> e2
            False -> 
                case (
                    case (y < 0) of
                        True  -> True
                        False -> (y >= height)
                    ) of
                    False -> e1
                    True  -> e2

One more case-of-case:

case (x < 0) of
    True  -> e2
    False -> 
        case (x >= width) of
            True  -> e2
            False -> 
                case (y < 0) of
                    True -> 
                        case True of
                            False -> e1
                            True  -> e2
                    False -> 
                        case (y >= height) of
                            False -> e1
                            True  -> e2

And one more case-of-known-constructor:

case (x < 0) of
    True  -> e2
    False -> 
         case (x >= width) of
             True  -> e2
             False -> 
                 case (y < 0) of
                     True  -> e2
                     False -> 
                         case (y >= height) of
                             False -> e1
                             True  -> e2

And we’re done! We arrived at the same expression that GHC compiled. Wasn’t that simple?

Summary

This should give you an idea of how GHC’s core-to-core transformations work. I’ve only shown you two of them – case-of-case and case-of-known-constructor – but there are many more. If you’re interested in learning others take a look at paper by Simon Peyton Jones and Andre Satnos “A transformation-based optimiser for Haskell”. If you want to learn more details than the paper provides see Andre Santos’ PhD thesis “Compilation by Transformation in Non-Strict Functional Languages”. You can also take a look at a discussion at GHC ticket 6135.

1. We usually call it “desugared”, because simplifying Haskell to Core simply removes syntactic sugar.
2. A scrutinee is an expression which value is checked by the the case expression. Scrutinee is placed between words ‘case’ and ‘of’.

Real World Haskell – commonly referred to as RWH – was written by Bryan O’Sullivan, John Goerzen and Don Stewart and published in 2008 by O’Reilly. With 28 chapters on 670 pages it is the book about Haskell. There is no more comprehensive book on Haskell at this moment. The best thing is it is available on-line for free. I don’t like reading large amounts of text from a computer screen, so when I decided to learn Haskell 10 months ago I mailed my university’s library and asked if they could get the book for me. Three weeks later brand new copy of RWH was on my desk. Up till now I read chapters 1-11, 17 and 25 which is about half of the book. Perhaps a review should be based on reading the whole, but after over 300 pages I already have my opinion and I don’t think it will change much after reading remaining chapters.

The book assumes no prior knowledge of Haskell or functional programming. It starts off with simple, introductory topics and explains concepts of functional approach to programming. While the first examples of code are rather simple the book quickly moves to real-world applications (as the title rightly suggests). Aside from standard Haskell topics like lazy evaluation, typeclasses or monads, the book covers also parsing, databases, GUI programming, testing, profiling or interfacing with C. There are also many case studies. The good thing about the book is that chapters about particular technologies (that would be from 16 onwards) are self-contained and can be read in any order. On the other hand initial chapters seem to be a bit messy – they contain lots of valuable information scattered around in completely unexpected places.

While the book is a great source of information there are some reasons not to be happy with it. First of all I think that it is not suitable for most beginners. While RWH assumes no prior knowledge, the examples quickly get complicated and it might be hard to figure out what is really going on in the code. Authors often use top-down approach, that is they present a lot of code out of nowhere and then go into explaining what it does (this is the analytical approach). For that reason I had a lot of hard time with parsing described in chapter 10 and after reading the chapter twice I still don’t understand all of it. I think that in some cases bottom-up (a.k.a synthetic) approach to algorithm construction would be more suitable. This is of course very subjective – some people might be able to understand everything without problems. Anyway, the book is demanding and the reader should be prepared for it. Also, be prepared that chapters about more advanced technologies give mostly an overview and are not comprehensive treatment of the subject. After reading chapter about FFI I wasn’t able to write my own C bindings and only after reading the official FFI specification things became clear.

I was disappointed with the lack of discussion of GHC internals. I think it would be useful to explain things like thunks, WHNF as well as lazy evaluation model and possible performance issues related to it (memory leaks for example). There is one chapter about performance and profiling in general but it doesn’t go into details of how the compiler and runtime system works.

Speaking of the performance chapter, RWH is also affected by changes made to GHC since the book was published. Luckily this is not a major issue as almost all examples work as they should. Chapter about performance is seriously affected though. I wasn’t able to reproduce any of the presented results with GHC 7.4.2, but of course authors can’t be blamed for that. Also, one code snippet does not work because of changes made in GHC but again this is not a big problem.

I had a feeling that some basic concepts are not explained clearly enough. For example the book shows usage of if instruction in place where a more experienced functional programmer would probably use pattern matching (page 30). This might be fine since at this point the reader was not yet introduced to pattern matching. The problem is that this is only clarified on page 207, way too late in my opinion, especially that patterns are introduced on page 50. I also had a feeling that currying was not explained clearly and stressed enough. Moreover the book does not mention the term “referential transparency” (might be a good thing to teach wannabe Haskell programmers some jargon they are likely to encounter) and from the comments on the web I noticed that people not familiar with the concept of “side effects” are confused by the book’s explanation of that term. I also found that omitting type signatures in some function definitions makes the code harder to understand. Finally, one note about editorial side of the book. The book uses different kinds of crosses, stars, paragraphs and other signs to denote footnotes. It looks as though editors have decided not to use the same mark twice which is a bit annoying for me, but this is of course very subjective and definitely not related to the quality of the book itself.

Although I complained that some concepts could have been explained in a different way I still think that RWH is a must-read for any serious Haskell programmer. Mostly because it gathers a lot of practical information in one place. I hope to find more time to finish reading it as I’m very curious about chapters on monads and monad transformers.

I work on a project in which the same algorithm is implemented using different data structures – one implementation is done in C, another using Vector library and yet another using Repa. Everything is benchmarked with Criterion and C implementation is the fastest one (look at first value after mean – this is mean time of running a function):

benchmarking DWT/C1
mean: 87.26403 us, lb 86.50825 us, ub 90.05830 us, ci 0.950
std dev: 6.501161 us, lb 1.597160 us, ub 14.81257 us, ci 0.950
 
benchmarking DWT/Vector1
mean: 209.4814 us, lb 208.8169 us, ub 210.5628 us, ci 0.950
std dev: 4.270757 us, lb 2.978532 us, ub 6.790762 us, ci 0.950

This algorithm uses a simpler lattice function that is repeated a couple of times. I wrote benchmarks that measure time needed by a single invocation of lattice:

benchmarking C1/Lattice Seq
mean: 58.36111 us, lb 58.14981 us, ub 58.65387 us, ci 0.950
std dev: 1.260742 us, lb 978.6512 ns, ub 1.617153 us, ci 0.950
 
benchmarking Vector1/Lattice Seq
mean: 34.97816 us, lb 34.87454 us, ub 35.14377 us, ci 0.950
std dev: 661.5554 ns, lb 455.7412 ns, ub 1.013466 us, ci 0.950

Hey, what’s this!? Vector implementation is suddenly faster than C? Not possible given that DWT in C is faster than DWT using Vector. After some investigation it turned out that the first C benchmark runs correctly while subsequent benchmarks of C functions take performance hit. I managed to create a simple code that demonstrates the problem in as few lines as possible. I implemented a copy function in C that takes an array and copies it to another array. Here’s copy.c:

#include
#include "copy.h"
 
double* c_copy( double* inArr, int arrLen ) {
  double* outArr = malloc( arrLen * sizeof( double ) );
 
  for ( int i = 0; i < arrLen; i++ ) {
    outArr[ i ] = inArr[ i ];
  }
 
  return outArr;
}

and copy.h:

#ifndef _COPY_H_
#define _COPY_H_
 
double* c_copy( double*, int );
 
#endif

I wrote a simple binding for that function and benchmarked it multiple times in a row:

module Main where
 
import Criterion.Main
import Data.Vector.Storable hiding (copy)
import Control.Monad (liftM)
import Foreign hiding (unsafePerformIO)
import Foreign.C
import System.IO.Unsafe (unsafePerformIO)
 
foreign import ccall unsafe "copy.h"
  c_copy :: Ptr CDouble -> CInt -> IO (Ptr CDouble)
 
signal :: Vector Double
signal = fromList [1.0 .. 16384.0]
 
copy :: Vector Double -> Vector Double
copy sig = unsafePerformIO $ do
    let (fpSig, _, lenSig) = unsafeToForeignPtr sig
    pLattice <- liftM castPtr $ withForeignPtr fpSig $ \ptrSig ->
                c_copy (castPtr ptrSig) (fromIntegral lenSig)
    fpLattice <- newForeignPtr finalizerFree pLattice
    return $ unsafeFromForeignPtr0 fpLattice lenSig
 
 
main :: IO ()
main = defaultMain [
         bgroup "FFI" [
           bench "C binding" $ whnf copy signal
         , bench "C binding" $ whnf copy signal
         , bench "C binding" $ whnf copy signal
         , bench "C binding" $ whnf copy signal
         , bench "C binding" $ whnf copy signal
         , bench "C binding" $ whnf copy signal
         , bench "C binding" $ whnf copy signal
         , bench "C binding" $ whnf copy signal
         , bench "C binding" $ whnf copy signal
         ]
       ]

Compiling and running this benchmark with:

$ ghc -O2 -Wall -optc -std=c99 ffi_crit.hs copy.c
$ ./ffi_crit -g

gave me this results:

benchmarking FFI/C binding
mean: 17.44777 us, lb 16.82549 us, ub 19.84387 us, ci 0.950
std dev: 5.627304 us, lb 968.1911 ns, ub 13.18222 us, ci 0.950
 
benchmarking FFI/C binding
mean: 45.46269 us, lb 45.17545 us, ub 46.01435 us, ci 0.950
std dev: 1.950915 us, lb 1.169448 us, ub 3.201935 us, ci 0.950
 
benchmarking FFI/C binding
mean: 45.79727 us, lb 45.55681 us, ub 46.26911 us, ci 0.950
std dev: 1.669191 us, lb 1.029116 us, ub 3.098384 us, ci 0.950

The first run takes about 17μs, later runs take about 45μs. I found this result repeatable across different runs, although in about 10-20% of runs all benchmarks – including the first one – took about 45μs. I obtained this results on GHC 7.4.1, openSUSE 64-bit linux with 2.6.37 kernel, Intel Core i7 M 620 CPU. I posted this on Haskell-cafe and #haskell. Surprisingly nobody could replicate the result! I was confused so I gave it a try on my second machine: Debian Squeeze, 64-bit, GHC 7.4.2, 2.6.32 kernel, Intel Core 2 Due T8300 CPU. At first the problem did not appear:

benchmarking FFI/C binding
mean: 107.3837 us, lb 107.2013 us, ub 107.5862 us, ci 0.950
std dev: 983.6046 ns, lb 822.6750 ns, ub 1.292724 us, ci 0.950
 
benchmarking FFI/C binding
mean: 108.1152 us, lb 107.9457 us, ub 108.3052 us, ci 0.950
std dev: 916.2469 ns, lb 793.1004 ns, ub 1.122127 us, ci 0.950

All benchmarks took about 107μs. Now watch what happens when I increase size of the copied vector from 16K elements to 32K:

benchmarking FFI/C binding
mean: 38.50100 us, lb 36.71525 us, ub 46.87665 us, ci 0.950
std dev: 16.93131 us, lb 1.033678 us, ub 40.23900 us, ci 0.950
 
benchmarking FFI/C binding
mean: 209.9733 us, lb 209.5316 us, ub 210.4680 us, ci 0.950
std dev: 2.401398 us, lb 2.052981 us, ub 2.889688 us, ci 0.950

This first run is 2.5 time faster (!), while all other runs are two times slower. While the latter could be expected, the former certainly is not.

So what exactly is going on? I tried analysing eventlog of the program but I wasn’t able to figure out the cause of the problem. I noticed that if I comment out the loop in C function so that it only allocates memory and returns an empty vector then the problem disappears. Someone on Haskell-cafe suggested that these are cache effects, but I am sceptical about this explanation. If this is caused by cache then why did the first benchmark sped up when size of the vector was increased? And why does this effect occur for 16K length vectors on a machine with 4MB cache, while machine with 3MB cache needs twice longer vector for the problem to occur? So if anyone has a clue what causes this strange behaviour please let me know. I would be happy to resolve that since now result of my benchmarks are distorted (perhaps yours are too only you didn’t notice).

module Main where
 
import Control.Parallel
 
main :: IO ()
main = print . parSumFibEuler 38 $ 5300
 
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)
 
mkList :: Int -> [Int]
mkList n = [1..n-1]
 
relprime :: Int -> Int -> Bool
relprime x y = gcd x y == 1
 
euler :: Int -> Int
euler n = length (filter (relprime n) (mkList n))
 
sumEuler :: Int -> Int
sumEuler = sum . (map euler) . mkList
 
sumFibEuler :: Int -> Int -> Int
sumFibEuler a b = fib a + sumEuler b
 
parSumFibEuler :: Int -> Int -> Int
parSumFibEuler a b = f `par` (e `pseq` (e + f))
    where f = fib a
          e = sumEuler b

In the paper authors show that this code performs computation of fib ans sumEuler in parallel and that good speed-up is achieved:

To make the parallelism more robust, we need to be explicit about the evaluation order we intend. The way to do this is to use pseq in combination with par, the idea being to ensure that the main thread works on sumEuler while the sparked thread works on fib. (…) This version does not make any assumptions about the evaluation order of +, but relies only on the evaluation order of pseq, which is guaranteed to be stable.

These results were obtained on older GHC version¹. However, compiling program with:

$ ghc -O2 -Wall -threaded -rtsopts -fforce-recomp -eventlog parallel.hs

and then running on GHC 7.4.1 using:

$ ./parallel +RTS -qa -g1 -s -ls -N2

yields a completely different result. These are statistics for a parallel run on two cores:

SPARKS: 1 (1 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
 
INIT    time    0.00s  (  0.00s elapsed)
MUT     time    2.52s  (  2.51s elapsed)
GC      time    0.03s  (  0.05s elapsed)
EXIT    time    0.00s  (  0.00s elapsed)
Total   time    2.55s  (  2.56s elapsed)

Running the same code on one core results in:

SPARKS: 1 (0 converted, 0 overflowed, 0 dud, 1 GC'd, 0 fizzled)
 
INIT    time    0.00s  (  0.00s elapsed)
MUT     time    2.51s  (  2.53s elapsed)
GC      time    0.03s  (  0.05s elapsed)
EXIT    time    0.00s  (  0.00s elapsed)
Total   time    2.55s  (  2.58s elapsed)

Looking and MUT (mutator time) it looks that there is no speed-up at all. Investigating eventlog using ThreadScope sheds some light on execution of a program:

Both threads start computation, but HEC 1 soon blocks and only resumes when HEC 0 finishes computation. Zooming in it looks that HEC 1 stops because it requests garbage collection, but HEC 0 does not respond to that request so GC begins only when HEC 0 is done with its computation:

Why does this happen? I am no expert on GHC’s garbage collection, my only knowledge of that comes from section 6 of “Runtime Support for Multicore Haskell“. If I understood correctly this should not happen – it certainly didn’t happen when the paper was published. Do we have a regression or am I misunderstanding something?

1. Paper does not mention which version exactly. I believe it was 6.10, since “Runtime Support for Multicore Haskell” by the same authors released at the same time uses GHC 6.10

I’ve written some very ugly Haskell code that creates a vector using destructive updates. It is in fact an imperative algorithm, not a functional one. When the initialization is over the vector is frozen using unsafeFreeze. I wrote my code using read and write functions, tested it using QuickCheck and when the tests passed I switched to unsafeRead and unsafeWrite to make my program faster. Some time later I started getting random segfaults when running my tests. This never happened before in any of my Haskell programs so I almost panicked. At first I didn’t had a slightest idea how to even approach this problem. I suspected that this might even be a bug in GHC. Then I started disabling groups of tests trying to track down the bug and finally managed to locate a single test that was causing the problem. Guess what – it was the test of my vector initialization with unsafe functions. What happened is that after switching to unsafeRead/unsafeWrite I refactored the function and its test. I made a mistake in the testing code and passed incorrect data that resulted with an attempt to write an element at address -1. Finding this bug took me a little over an hour. A factor that made debugging harder was that disabling tests that seemed to cause the segfault resulted in problems appearing in a completely different part of the program – or so I thought by looking at the output of my program. Looks like I completely forgot about lazy evaluation and unspecified evaluation order!

Second bug I encountered was even trickier. I wrote functions that perform cyclic shifts of a signal by any value. For example shifting [1,2,3,4] left by 1 yields [2,3,4,1]. Note that shifting by 5, 9, 13 and so on gives exactly the same result – the shift function is periodic. You might recall that I used shift functions to demonstrate code testing. This time however I written shifts using Repa library. Then I created QuickCheck property stating that shifting any signal left and then right by same value yields the original signal. This is pretty obvious property and the tests passed without any problems. Later, when writing some other tests I ended up with one of the tests failing. Using ghci I checked that this error should not be happening at all, but after careful debugging it turned out that during actual tests some of the values in the results become zeros. After two hours of debugging I realized that the actual bug is in the shifting functions – they worked only for the basic period, that is shift values from 0 to signal length. Why QuickCheck didn’t manage to falsify my property of left/right shift compositions? Repa is a smart (and tricky) library that attempts to fuse many operations on an array into one. And it fused application of left shift followed by right shift into identity transform! Well, this is great. After all this is the kind of optimization we would like to have in our programs. But it turns out that it can also impact tests! After realizing what is going on it was actually a matter of 5 minutes to fix the bug, but finding it was not a trivial task.

Luckily for me there is a better solution: Foreign Function Interface (FFI). This is an extension of Haskell 98 standard – and part of Haskell 2010 – that allows to call functions written in C¹. This means that I could write my function in C, wrap it in a pure Haskell function and benchmark that wrapper with Criterion. The results would be comparable with Haskell implementation, but I was afraid that overheads related to data copying would affect the performance measurements. As it turned out I was wrong.

I started with chapter 17 of Real World Haskell. It presents a real world example – I guess that title of the book is very adequate – of creating bindings for an already existing library. Sadly, after reading it I felt very confused. I had a general idea of what should be done but I didn’t understand many of the details. I had serious doubts about proper usage of Ptr and ForeignPtr data types and these are in fact very important when working with FFI. Someone on #haskell advised me to read the official specification of FFI and this was a spot-on. This is actually one of the few official specifications that are a real pleasure to read (if you read R5RS then you know what I mean). It is concise (30 pages) and provides a comprehensive overview of all data types and functions used for making foreign calls.

After reading the specification it was rather straightforward to write my own bindings to C. Here’s a prototype of called C function, located in dwt.h header file:

double* c_dwt(double* ls, int ln, double* xs, int xn);

The corresponding dwt.c source file contains:

double* c_dwt( double* ls, int ln, double* xs, int xn ) {
  double* ds = malloc( xn * sizeof( double ) );
 
  // fill ds array with result
 
  return ds;
}

The important thing is that C function mallocates new memory which we will later manage using Haskell’s garbage collector. Haskell binding for such a function looks like this:

foreign import ccall unsafe "dwt.h"
  c_dwt :: Ptr CDouble -> CInt -> Ptr CDouble -> CInt -> IO (Ptr CDouble)

Here’s what it does: ccall denotes C calling convention, unsafe improves performance of the call at the cost of safety² and "dwt.h" points to a header file. Finally, I define the name of the function and it’s type. This name is the same as the name of original C function, but if it were different I would have to specify name of C function in the string that specifies name of the header file. You probably already noticed that type int from C is represented by CInt in Haskell and double by CDouble. You can convert between Int and CInt with fromIntegral and between Double and CDouble with realToFrac. Pointers from C became Ptr, so double* from C is represented as Ptr Double in Haskell binding. What might be surprising about this type signature is that the result is in the IO monad, that is our function from C is denoted as impure. The reason for this is that every time we run c_dwt function a different memory address will be allocated by malloc, so indeed the function will return different results given the same input. In my function however the array addressed by that pointer will always contain exactly the same values (for the same input data), so in fact my function is pure. The problem is that Haskell doesn’t know that and we will have to fix that problem using the infamous unsafePerformIO. For that we have to create a wrapper function that has pure interface:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

import Control.Monad (liftM)
import Data.Vector.Storable
import Foreign hiding (unsafePerformIO)
import Foreign.C
import System.IO.Unsafe
 
dwt :: Vector Double -> Vector Double -> Vector Double
dwt ls sig = unsafePerformIO $ do
    let (fpLs , _, lenLs ) = unsafeToForeignPtr ls
        (fpSig, _, lenSig) = unsafeToForeignPtr sig
    pDwt <- liftM castPtr $ withForeignPtr fpLs $ \ptrLs ->
            withForeignPtr fpSig $ \ptrSig ->
                c_dwt (castPtr ptrLs ) (fromIntegral lenLs )
                      (castPtr ptrSig) (fromIntegral lenSig)
    fpDwt <- newForeignPtr finalizerFree pDwt
    return $ unsafeFromForeignPtr0 fpDwt lenSig

Our wrapper function takes two Vectors as input and returns a new Vector. To interface with C we need to use storable vectors, which store data that can be written to raw memory (that’s what the C function is doing). I wasn’t able to figure out what is the difference between storable and unboxed vectors. It seems that both store primitive values in continuous memory block and therefore both offer similar performance (assumed, not verified). First thing to do is to get ForeignPtrs out of input vectors. ForeignPtr is a Ptr with a finalizer attached. Finalizer is a function called when the object is no longer in use and needs to be garbage collected. In this case we need a function that will free memory allocated with malloc. This is a common task, so FFI implementation already provides a finalizerFree function for that. The actual call to foreign function is made on lines 11-14. We can operate on Ptr values stored in ForeignPtr using withForeignPtr function. However, since we have vectors of Doubles as input, we also have Ptr Double, not Ptr CDouble that c_dwt function expects. There are two possible solutions to that problem. One would be to copy memory, converting every value in a vector using realToFrac. I did not try that assuming this would kill performance. Instead I used castPtr which casts pointer of one type to a pointer of another type. This is potentially dangerous and relies on the fact that Double and CDouble have the same internal structure. This is in fact expected, but by no means it is guaranteed by any specification! I wouldn’t be surprised it that didn’t work on some sort of exotic hardware architecture. Anyway, I written tests to make sure that this cast does work the way I want it to. This little trick allows to avoid copying the input data. The output pointer has to be cast from Ptr CDouble to Ptr Double and since the result is in the IO monad the castPtr has to be lifted with liftM. After getting the result as Ptr Double we wrap it in a ForeignPtr with a memory-freeing finalizer (line 15) and use that foreign pointer to construct the resulting vector of Doubles.

Summary

I had two concerns when writing this binding. First was the possible performance overhead. Thanks to using pointer casts it was possible to avoid any sort of data copying and that makes this binding real quick. Measuring execution time with criterion shows that calling C function that does only memory allocation (as shown in this post) takes about 250µs. After adding the rest of C code that actually does computation the execution time jumps to about 55ms, so the FFI calling overhead does not skew the performance tests. Big thanks go to Mikhail Glushenkov who convinced me with his answer on StackOverflow to use FFI. My second concern was the necessity to use many functions with the word “unsafe”, especially the unsafePerformIO. I googled a bit and it seems that this is a normal thing when working with FFI and I guess there is no reason to worry, provided that the binding is thoroughly tested. So in the end I am very happy with the result. It is fast, Haskell manages garbage collection of memory allocated with C and most importantly I can benchmark C code using Criterion.

1. Specification mentions also the calling conventions for other languages and platforms (Java VM, .Net and C++) but I think that currently there is no implementation of these.
2. Calls need to be safe only when called C code calls Haskell code, which I think is rare

The code that Bryan refers to looks like this:

main :: IO ()
main = newStdGen >>= defaultMainWith benchConfig (return ()) . benchmarks
 
benchmarks :: RandomGen g => g -> [Benchmark]
benchmarks = ...

Each time a benchmark suite is run a different random numbers generator is created with newStdGen. This generator is then used by the benchmarks function to create values used for benchmarking. When I designed this I made an assumption that values of the data don’t influence the flow of computations. I believe that this holds for the shift functions I benchmarked in my tutorial. It doesn’t really matter what values are in the shifted list. As long as lists have the same length on different runs of the benchmark the results are comparable, but if you want to have the same random values generated on each run you can create a StdGen based on a seed that you supply. The modified main function would look like this:

main = return (mkStdGen 123456) >>= defaultMainWith benchConfig (return ()) . benchmarks

What happens however when data values do influence the flow of computation? In that case you definitely don’t want newStdGen, as it would make results of benchmark incomparable between different runs: you wouldn’t know if the speed-up is caused by changes in the code or data. It is also very likely that you don’t want to use mkStdGen. Why? Well, you would certainly get results comparable between different runs. The problem is that you wouldn’t know the characteristics of the data used for this particular benchmark. For example let’s assume that your algorithm executes faster when the data it processes contains many zeros. You benchmark the algorithm with random values created with a fixed StdGen and get a very good result. But how many zeros were in the data used for benchmarking? Perhaps 50% of input were zeros? You don’t know that. In this case you definitely want to prepare your own input data sets (e.g. one with many zeros and one without any) to measure the performance of your code based on input it receives. I guess Bryan is right here – careless use of random data generation for benchmarking can be a shot in the foot.