ABSTRACT: In this article, I present the QuickCheck.wl package for the QuickCheck[] automated property-based testing function with strongly-typed assumptions. The function can be used with symbolic variables, equalities, inequalities, and assumptions, which are all dynamically and replaced arbitrarily using pseudo-random number generators. Throughout the discussion of this article, I present the benefits of property-based testing with some interesting cases.
Caution: QuickCheck.wl is not a silver bullet. Besides being well built, fast, performant, and great at it’s purpose, it’s made for primarily fuzzing functions and randomized property checking, but it cannot mathematically prove that some function is always correct.
What is Property-Based Testing?
What is Property-Based Testing?
Property-based tests are designed to test the aspects of a property that should always be true. Instead of creating multiple unit test cases for a function, verifying one absolutely truthful property can outperform the complexity of writing and thinking about all of the cases that might fail.
Suppose if we want to test the following property for N booleans, if we have two booleans, that would be 4 test cases in total, for a big number such as 4, that would be 16 total cases, thus, making it not so convenient for handwriting all of the tests.
Instead, we can just check the actual property specifying which variables will assume with arbitrary values:
Quick-check a pretty stubborn property with some crazy assumptions.
In[]:=
QuickCheck["a really long hypothetical property that should take forever to write test cases for",x!=!(y!=!z)==w==v∨x,"Assume"->{x:>ΤBoolean,y->!x,z->!y,w->(y!=!z),v->(!z)∨y}]
»
Falsifiable after 11 test(s) on 0.044446 seconds.
Out[]=
{False≠!(False≠!True)FalseFalse||False}
After some tests, the property could be falsified, this means that the property didn’t hold for a certain case. A convenient feature of the QuickCheck[] function is that it outputs a list of all failed cases, so you can know how exactly where and how your function failed.
Quick-check the pretty stubborn property for 128 falsifiable cases and delete the duplicates.
In[]:=
DeleteDuplicates@QuickCheck["a really long hypothetical property that should take forever to write test cases for",x!=!(y!=!z)==w==v∨x,"Assume"->{x:>ΤBoolean,y->!x,z->!y,w->(y!=!z),v->(!z)∨y},"MaxFails"->128]
»
Falsifiable after 979 test(s) on 0.455327 seconds.
Out[]=
{False≠!(True≠!True)TrueTrue||False,False≠!(False≠!False)TrueTrue||False,False≠!(True≠!False)FalseFalse||False,False≠!(False≠!True)FalseFalse||False}
But the interesting thing of QuickCheck is that it can work for multiple types, we will look into this later on this article.
List the number of QuickCheck types.
In[]:=
Length[QuickCheckTypes]
Out[]=
18
Why Testing in the First Place?
Why Testing in the First Place?
Suppose that you have a function that adds a list of numbers..
Define the function “AddsNumbers” that sums a list of numbers using tail recursion.
In[]:=
AddsNumbers[{}]:=0;AddsNumbers[{head_,tail___}]:=head+AddsNumbers[{tail}];
How could you quickly verify that this function works for lists of integers? The answer is by writing tests. But writing tests can be extremely complex and sometimes impossible for complex functions.
Quick-check the property “AddNumbers works the same way as Total”.
In[]:=
QuickCheck["AddNumbers works the same way as Total",AddsNumbers[x]==Total[x],"Assume"->{x:>ΤList[ΤInteger]}];
»
Okay! Property "AddNumbers works the same way as Total passed 1024 test(s) on 0.434405 seconds.
Conjugate Checking
Conjugate Checking
Let’s quick-check if the Conjugate[] function holds the property of “flipping” the imaginary sign.
Quick-check the property “Conjugating a complex flips the imaginary sign”.
In[]:=
QuickCheck["Conjugating a complex flips the imaginary sign",Conjugate[a]==Re[b]+-1Im[b]I,"Assume"->{a->RandomChoice[{ΤComplexInteger,ΤComplexDecimal}],b->a}];
»
Okay! Property "Conjugating a complex flips the imaginary sign passed 1024 test(s) on 0.554513 seconds.
All tests pass! Conjugate[] works as intended for many complex integers.
Digits Splitting Checking
Digits Splitting Checking
Let’s create a function that has a similar behavior of IntegerDigits[] then quick-check if the function holds the property of getting the numbers as IntegerDigits[] would.
Define the function “DigitSplit” as the individual parsed digits from a string of a given number.
In[]:=
DigitSplit[number_Integer]:=FromDigits[#]&/@Characters[ToString[number]];
Quick-check the property “DigitSplit works just as IntegerDigits for Integers”.
In[]:=
Quiet@QuickCheck["DigitSplit works just as IntegerDigits for Integers",IntegerDigits[a]==DigitSplit[a],"Assume"->{a->ΤInteger},"MaxFails"->5]
»
Falsifiable after 14 test(s) on 0.04353 seconds.
Out[]=
{IntegerDigits[-45798]DigitSplit[-45798],IntegerDigits[-58278]DigitSplit[-58278],IntegerDigits[-42929]DigitSplit[-42929],IntegerDigits[-31830]DigitSplit[-31830],IntegerDigits[-38984]DigitSplit[-38984]}
Aha! It seems that my function DigitSplit[] does not work for negative integers. This is happening because the function FromDigits[] cannot understand what a “minus sign” is! So that’s why the property fails on these specific cases.
Let’s fix this simple bug and try quick-checking again..!
Define the function “DigitSplit” as the individual parsed digits from a string of the absolute value of a given number.
In[]:=
DigitSplit[number_Integer]:=FromDigits[#]&/@Characters[ToString[Abs[number]]];
Quick-check the property “IntegerDigits works just as Characters for Strings”.
In[]:=
Quiet@QuickCheck["IntegerDigits works just as Characters for Strings",IntegerDigits[a]==DigitSplit[a],"Assume"->{a->ΤInteger},"MaxFails"->5];
»
Okay! Property "IntegerDigits works just as Characters for Strings passed 1024 test(s) on 0.060424 seconds.
The quick-check confidently worked out that the property holds true on many cases!
Uncompress-Compress Checking
Uncompress-Compress Checking
Let’s quick-check if the Compress[] and Uncompress[] functions holds the property of reversibility.
Quick-check the property “Uncompressing a compressed text always outputs the original text”.
QuickCheck["Uncompressing a compressed text always outputs the original text",Uncompress[Compress[a]]==b,"Assume"->{a->ΤPlainText,b->a}]
»
Okay! Property "Uncompressing a compressed text always outputs the original text passed 1024 test(s) on 0.539495 seconds.
Out[]=
{}
Of course it works! Uncompress[] and Compress[] are industrial strength functions.
ImportString-ExportString Checking
ImportString-ExportString Checking
Let’s quick-check if the ImportString[] and ExportString[] functions holds the property of reversibility.
Quick-check the property “Importing an exported ExpressionML string always outputs the original expression”.
In[]:=
QuickCheck["Importing an exported ExpressionML string always outputs the original expression",ImportString[ExportString[a,"ExpressionML"]]==b,"Assume"->{a->ΤPlainText,b->a},"Runs"->100]
»
Okay! Property "Importing an exported ExpressionML string always outputs the original expression passed 100 test(s) on 2.40649 seconds.
Out[]=
{}
No surprise. ExportString[] and ImportString[] are industrial strength functions as well.
All Types and Options Supported
All Types and Options Supported
Currently, QuickCheck.wl supports 18 types.
Map every item of “AllQuickCheckTypes” to it’s usage and then create a dataset.
In[]:=
Dataset[{#,#::usage}&/@QuickCheckTypes]
Out[]=
There are few options for QuickCheck, the most useful are “ExponentRange” if you’re working with big numbers and “MinMaxStringSize” if you’re working with strings.
Display all options of QuickCheck.
Installation
Installation
You can get QuickCheck.wl on the GitHub repository: https://github.com/cabralski/quickcheck.wl and move the quickcheck.wl file into the following folder:
Open the folder for moving the “quickcheck.wl” file into.
Limitations
Limitations
There are some things that QuickCheck.wl cannot do currently. Here’s a list of what it can and can’t do.
◼
Can only receive equalities, inequalities, and logic operators as properties.
◼
Will not work for equalities and inequalities that do not yield a Boolean result.