How to Make a Thing in Haskell, Part 3: Buildin' with Tests, Cookin' with Gas

Of course, we can’t Build Real Software without a good testing rig. In this post, I’ll talk about using Hspec, which Haskeleton uses by default, and I like it a lot so far.

Hspec

Hspec defines a testing DSL inspired by RSpec. Like all good Pythonists, I’m totally jealous of RSpec, with its pretty describes and its:

describe Calculator do
  describe '#add' do
    it 'returns the sum of its arguments' do
      expect(Calculator.new.add(1, 2)).to eq(3)
    end
  end
end

But, also like all good Pythonists, I’m afraid of the terrible magic that must be happening to make this work. Even Aaron Patterson, “the only engineer in the world who is on the core Ruby and core Rails teams” is not entirely sure how RSpec does its thing:

Can I use inheritance? … where does super go? These are not the types of questions I want to be pondering when I have 3000 test failures and really long spec files to read through. My mind spins off in to thoughts like “I wonder how RSpec works?”, and “I wonder what my cat is doing?”

(This is not a knock on tenderlove or on RSpec. RSpec is beautiful, but it’s complicated enough that even its most brilliant users aren’t sure what happens when they define a test.)

Hspec seems to provide a similarly pretty way to define tests:

import Test.Hspec
import Test.QuickCheck
import Control.Exception (evaluate)

main :: IO ()
main = hspec $ do
    describe "Prelude.head" $ do
        it "returns the first element of a list" $ do
            head [23 ..] `shouldBe` (23 :: Int)

Pretty, at least, if you’ve accepted that your sequential code will be littered with $ and do, which you probably have if you’re writing Haskell.

Hspec works differently than Rspec or pytest in interesting way – it doesn’t use metaprogramming. There’s a test-file-discovery tool, and that tool finds files ending in Spec.hs that export a spec function. However, there’s no test-discovery algorithm operating over lists of functions at runtime; those spec functions are just sequences of actions that execute tests. That means it works in, but is not bound to, do-notation. It also means that if you’re used to reasoning about sequential Haskell code, you can probably reason about the environment in which the tests run without thinking too hard, like you might have to with a dynamic test runner.

Running Tests

But, enough introductions! How do you actually use Hspec? In my previous post, as part of setting up Haskeleton, I put {-# OPTIONS_GHC -F -pgmF hspec-discover #-} in Spec.hs, as described in the Hspec docs. This seems to basically be a preprocessor directive for Haskell environments to run the hspec-discover command as though it were this file’s main function. It’s just a bit of an incantation to me, and frankly, I’m surprised Haskell has a preprocessor! That seems like asking for all kinds of trouble. But, in any event, hspec-discover finds the files ending in Spec.hs where tests are defined.

cabal test can take some extra options to expose Hspec’s RSpec-style output. Given the tests defined in the base Haskeleton setup:

spec =
    describe "skellect" $ do
        it "returns unit" $
            skellect `shouldBe` ()
        it "is equal to unit" $ property $
            \x -> skellect `shouldBe` x

the command cabal test --show-details=always --test-option=--color, in addition to cabal build infromation and final test results, shows this:

Skellect
  skellect
    returns unit
    is equal to unit

Finished in 0.0010 seconds
2 examples, 0 failures

However, this has some drawbacks that I avoid by using this as my test script:

#!/usr/bin/env bash
set -e # log each command before execution
set -x # fail fast

cabal build >/dev/null # build, showing errors but not regular output
cabal exec -- runhaskell test-suite/HLint.hs # lint
cabal exec -- runhaskell -ilibrary -itest-suite test-suite/Spec.hs # test!

Essentially, this is a hand-rolled cabal test. cabal exec executes the command after -- in the current sandbox Using the -i options to the Spec call indicates where runhaskell should look for modules. This command fails fast if there are linter errors, prints Hspec’s colored and formatted output, and avoids printing a bunch of unnecessary information from the build and from cabal’s testing harness.

Basic Testing

So, the first thing the program needs to do is take in a \n-separated list of strings. That will involve some basic I/O and some simple list and string processing. Seems like a good piece of behavior to start with. This isn’t going to be tied to the I/O that happens in main (Haskell will make sure of it!). It’ll live in a generic library/Skellect/Utils.hs file, and its tests will live in test-suite/UtilsSpec.hs.

Annoyingly, I have to explicitly add Skellect.Utils to the Skellect library’s exposed-modules in skellect.cabal. I’m used to Python’s module system, where importing a top-level module gives access to its submodules as attributes. I suppose this difference makes sense – where Python modules are objects with accessible attributes, Haskell modules are purely namespaces, and the dot notation is just a way to group related namespaces.

Here’s the suite of unit tests for nonEmptyLines:

module UtilsSpec (spec) where

import Skellect.Utils (nonEmptyLines)
import Test.Hspec (describe, it, shouldBe, Spec)

spec :: Spec
spec = do
  describe "nonEmptyLines" $
    it "splits great-newline-pancakes into [\"great\", \"pancakes\"]" $
      nonEmptyLines "great\npancakes" `shouldBe` ["great", "pancakes"]
    it "ignores newlines at the beginning of the string" $ do
      nonEmptyLines "\noh\nno!" `shouldBe` ["oh", "no!"]
    it "ignores newlines at the end of the string" $ do
      nonEmptyLines "just\nfine\n" `shouldBe` ["just", "fine"]
    it "ignores the empty string between two newlines" $ do
      nonEmptyLines "what\n\nnow?" `shouldBe` ["what", "now?"]

Naturally, that doesn’t compile, because nonEmptyLines doesn’t exist yet. So, I’ll make the simplest possible implementation over in Skellect/Utils.hs:

module Skellect.Utils (nonEmptyLines) where

nonEmptyLines :: String -> [String]
nonEmptyLines = lines

lines is a built-in function that, like it says, breaks a string into lines. So, for instance, in ghci:

> lines "oh\nheck\nyeah"
["oh","heck","yeah"]

This is great, but has some corner cases we want to avoid. In particular, it’ll split empty strings off if newlines appear at the ends of the string or next to one another.

Util
  nonEmptyLines
    splits great-newline-pancakes into ["great","pancakes"]
    ignores newlines at the beginning of the string FAILED [1]
    ignores newlines at the end of the string
    ignores the empty string between two newlines FAILED [2]

Failures:

  1) Util.nonEmptyLines ignores newlines at the beginning of the string
       expected: ["oh","no!"]
        but got: ["","oh","no!"]

     # test-suite/UtilSpec.hs:11 (best-effort)

  2) Util.nonEmptyLines ignores the empty string between two newlines
       expected: ["what","now?"]
        but got: ["what","","now?"]

     # test-suite/UtilSpec.hs:15 (best-effort)

Ah! Looks like I was wrong about the behavior of newlines at the end of strings. Not a problem – I’ve still got my red tests.

So, all of the failures are caused by the presence of empty strings. That’s simple enough to get rid of them with a simple filter, so:

nonEmptyLines :: String -> [String]
nonEmptyLines = filter (not . null) . lines

And we’re in the green!

Util
  nonEmptyLines
    splits great-newline-pancakes into ["great","pancakes"]
    ignores newlines at the beginning of the string
    ignores newlines at the end of the string
    ignores the empty string between two newlines

Finished in 0.0002 seconds
4 examples, 0 failures

QuickCheck

Just for fun, I added some QuickCheck fuzz tests. To use it, I imported QuickCheck’s property function in my tests:

import Test.QuickCheck (property)

and added QuickCheck the test dependencies in skellect.cabal.

To use property, I need to define boolean functions that indicate if a function works correctly on a given input. QuickCheck will use its type signature to generate that input. For example, this simple property takes a String, splits it with nonEmptyLines, and returns False to indicate failure if there’s an empty string in the output:

noEmptyElementsProperty :: String -> Bool
noEmptyElementsProperty s = "" `notElem` (nonEmptyLines s)

I know^H^H^H^Hhope^H^H^H^Hthink that it will be true for any and all strings s. Hspec uses property to fuzz this function and see if the behavior holds:

it "doesn't let empty strings in its output" $ property $
    \s -> "" `notElem` nonEmptyLines s

QuickCheck is apparently very smart about type inference, so it can say “ok, this property is of type String -> Bool. Let’s throw a lot of Strings at it!”

And the tests say I’m still in the green! Just to convince myself that it actually works, I used this command to run my tests:

cabal build && cabal exec -- runhaskell -ilibrary -itest-suite test-suite/Spec.hs -a 10000

What that does is:

If QuickCheck has to run a long time, it shows a progress counter. 10000 tests takes long enough that the counter is displayed long enough to see it. And I saw it! Success! My first parameterized test.

And, of course, QuickCheck can test more complex properties of the software. For instance, this recursive function takes a string and a list of strings and determines if the latter is a valid split for the former:

validLineSplits :: String -> [String] -> Bool
validLineSplits "" [] = True
validLineSplits ('\n':xs) ys = validLineSplits xs ys
validLineSplits (x:xs) (y:ys)
    | x == head y = validLineSplits xs (remainingSplit (tail y: ys))
    | otherwise   = False
        where
            remainingSplit ("":as) = as
            remainingSplit as = as
validLineSplits _  [] = False
validLineSplits "" _  = False

In other words,

The reduction here looks like:

"oh\nhi" <=> ["oh","hi"], remove 'o'
 "h\nhi" <=> ["h","hi"],  remove 'h', remove empty string from list
  "\nhi" <=> ["hi"],      ignore '\n'
    "hi" <=> ["hi"]       remove 'h'
     "i" <=> ["i"]        remove 'i'
      "" <=> []           True!

Each step of the way results in another valid mapping from string to split lines. Pretty neat! But does the implementation conform to that? QuickCheck can fuzz the assertion with a new test case:

it "always provides a valid split" $ property $
    \s -> validLineSplits s (nonEmptyLines s)

So, let’s check. Let’s check 10000 TIMES:

$ cabal build && cabal exec -- runhaskell -ilibrary -itest-suite test-suite/Spec.hs -a 10000

SkellectUtils
  SkellectUtilsSpec
    nonEmptyLines
      splits great-newline-pancakes into ["great", "pancakes"]
      ignores newlines at the beginning of the input
      doesn't add empty strings on adjacent newlines
      ignores newlines at the end of the input
      doesn't let empty strings in its output
      splits lines in a valid way

Finished in 1.4608 seconds
6 examples, 0 failures

BAM. Works, at least on the 10000 inputs QuickCheck just generated.

Side note: This is probably overkill, especially for such a simple function. I actually spent more time debugging validLineSplits than I did writing nonEmptyLines. But, it was fun, and I fuzzed the heck out of it!

Just for good measure, let’s hook this up to stdin and see what happens! We’ll have our main function (back in executable/Main.hs) take input from a pipe, split it with nonEmptyLines, and print it to the screen:

module Main where

import System.IO (getContents, putStrLn)
import SkellectUtils

main :: IO ()
main = do
    choicesInput <- getContents
    putStrLn $ unlines $ nonEmptyLines choicesInput

So this behavior is now available at the terminal:

$ ls | cabal run
Preprocessing library skellect-0.0.0...
In-place registering skellect-0.0.0...
Preprocessing executable 'skellect' for skellect-0.0.0...
Running skellect...
benchmark
cabal.sandbox.config
dist
executable
library
skellect.cabal
t.sh
test-suite

cabal subcommands take a verbosity option -v with values 0-4, which means we can get rid of that cabal output stuff by running it with verbosity set to 0:

$ echo "\nhello\n\nworld" | cabal run -v0
hello
world

So, I feel familiar enough with testing in Haskell to move along and write some well-tested functions I can trust. You can find the state of the code, at the time of writing, here.