Issue #35: Special about testing, part 2

A while ago I did a special issue of Elm Bits about testing, it was issue #10. In 35 weeks as I've been writing this newsletter, my style has changed quite a bit. Few weeks ago I looked at that post again and it felt so empty, so small. Not only have I learned a bit more on testing in Elm, but I did end up going into the rabbit hole of testing in general. And it was quite exciting. So here we go again, round 2, it's testing week.

When you read about testing in Elm, often times you can hear about fuzz testing. You might have heard the term "fuzzing" in relation to information security domain. This means that you provide the app (or a function) random, invalid, or unexpected data as input and monitor the output, whether it behaves correctly or crashes. How does it look like?

When writing a normal unit test, we usually provide some predefined input, for example to test the sum function you would write test with 1 and 3 values to test positive numbers, 0 and -100 to test negative, and 5 with -3 to test both. Three simple test cases. To test more cases we can just write 100 more tests. Or grab a list of value pairs and check them in a loop. Or even grab a set of random numbers and feed them to the test. And the latter is essentially how fuzz testing works. In this case the random number generator acts as a fuzzer, and Elm library contains a fair bit of fuzzers for every taste.

Of course every time your run your fuzz tests input values would change, however, when the test crashes, Elm is being helpful and provides you the seed value which you can use to re-run the test with the exact same sequence of inputs: To reproduce these results, run: elm-test --fuzz 100 --seed <value>. By default each fuzz test runs 100 times, but this can be configured along with other parameters. The amazing Elm Programming book by Pawan Poudel gives a very good description on how to write fuzz tests in Elm. To see a real project using fuzz tests, have a look at elm-geometry.

Fuzzing is an exciting topic which is not limited to random inputs only. There is a very interesting piece of software called american fuzzy lop (yeah, like the rabbit) which uses genetic algorithms, among others, to shrink the number of test cases and find the most clean and noteworthy ones.

Test shrinking is another important concept in testing employed by various frameworks. Let's assume that a test found an input which causes the code to crash, and that input is a list of integers. Depending on the length of the list, the space of possible variations of that list may be enormous, and so the test framework would try to "shrink", or rather reduce, that input to its simplest form. For the task it can use different approaches, for example:

  • discard some of the elements in the list at random
  • divide all the numbers by a specific number
  • discard a single value
  • divide a single value by a specific number
  • substract 1 from a single value

and so on. There is usually no guarantee that any approach would return the simplest possible input that triggers the error, but if it did find something, that input would for sure be easier to understand.

While unrelated to Elm, theft library documentation is a great source of knowledge about shrinking in particular, and property testing in general. Hypothesis, which we'll talk about here as well, has posts discussing integrated vs type-based shrinking and compositional shrinking. And you can also check the chapter on shrinking from the book about PropEr, an property-based testing framework for Erlang and Elixir. Have a look at the source code for Elm's Shrink module which also contains a very detailed explanation of what it does and how. Finally, if you're feeling brave enough, you can read "Test-Case Reduction via Test-Case Generation:Insights From the Hypothesis Reducer" which is a paper co-authored by one of the authors of Hypothesis.

The idea of investigating the topic of testing in one of the special issues started slowly getting into my mind after reading the announcement post for elm-minithesis. Back then I didn't understand a word that was discussed there, but slowly (and especially as I was preparing this email) I got hooked at this exciting topic (and hopefully so are you).

Elm-minithesis is a port of minithesis, which is a property testing framework for Python. Minithesis itself is a mini version of Hypothesis and implements just its core ideas. 'Nuff with that inception stuff, what do all these frameworks have in common, and what are they good for?

Like elm-explorations/test's Fuzz module, it's also a property-based testing library. The difference, however, is that elm-minithesis employs a different shrinking strategy, the integrated approach versus type-based that elm-test uses. This makes it possible to use andThen:

describe "int"
    [ testMinithesis "Can work with negative numbers"
        (F.int -10 10)
        (\n -> n >= -10 && n <= 10)
        Passes
    , testMinithesisRejects "min > max"
        (F.int (Random.minInt + 1) Random.maxInt
            |> F.andThen
                (\from ->
                    F.int Random.minInt (from - 1)
                        |> F.andThen (F.int from)

Like any topic, if you go deep enough, it becomes a rabbit hole. There is so much to read and learn about testing that I cannot fit inside an email that would be comfortable to read. I do hope however that I sparked some interest in you to explore further and read linked articles and posts.

Curiosity, the overwhelming desire to know, is not characteristic of dead matter. Nor does it seem to be characteristic of some forms of living organism, which, for that very reason, we can scarcely bring ourselves to consider alive.

by Isaac Asimov

Show Comments