Wolfram Language meta-programming: implementing dplyr
Wolfram Language meta-programming: implementing dplyr
Introduction
Introduction
I was always curious how exactly Wolfram Language evaluation works. The best way to learn is to build something non-trivial that depends on non-standard evaluation. There is a lot of “black magic” in the below code, and I would be grateful for the feedback from the community on how to make my code more concise and idiomatic. I would also appreciate if you could point to any gaps or inaccuracies in my understanding.
We will implement a functionality similar to what is provided by R package dplyr. It was created by Hadley Wickham, who wrote best-in-class packages for data visualization and data transformation in R. The following excerpt from dplyr documentation site provides an excellent overview:
dplyr is a grammar of data manipulation, providing a consistent set of verbs that help you solve the most common data manipulation challenges: mutate() adds new variables that are functions of existing variables select() picks variables based on their names. filter() picks cases based on their values. summarise() reduces multiple values down to a single summary. arrange() changes the ordering of the rows.These all combine naturally with group_by() which allows you to perform any operation “by group”.
And here is an example of this approach: a Star Wars data set is grouped by species field, two statistics (count and mean) are calculated, and the result is additionally filtered by the newly created fields:
starwars %>%
group_by(species) %>%
summarise(
n = n(),
mass = mean(mass, na.rm = TRUE)
) %>%
filter(n > 1,
mass > 50)
group_by(species) %>%
summarise(
n = n(),
mass = mean(mass, na.rm = TRUE)
) %>%
filter(n > 1,
mass > 50)
From symbols to strings
From symbols to strings
Our first adventure will be a journey from symbols to strings. We’ll use a toy association as a data model, which will give us a taste of how we’ll be working with datasets later:
In[]:=
data=<|"x"12,"y"20,"z"105|>;
Our first function extract will extract a value in the association given a symbol, as opposed to a string key, so that a user, similar to dplyr, can save a few keystrokes:
In[]:=
extract[d_,s_Symbol]:=d[SymbolName[s]]
In[]:=
ClearAll[x,y];{extract[data,x],extract[data,y]}
Out[]=
{12,20}
The solution is straightforward when symbols x and y are clear of any values, but breaks if e.g. x has a value assigned:
In[]:=
x=1;extract[data,x]
Out[]=
extract[x12,y20,z105,1]
Here x was evaluated before it was seen by extract, so we effectively called extract[data,1] which doesn’t work as extract expects a symbol for the second argument. One way to fix this is to hold extract’s argument from the evaluation:
In[]:=
ClearAll[extract];extract[d_,s_Symbol]:=d[SymbolName[Unevaluated@s]];SetAttributes[extract,HoldRest]
In[]:=
x=1;extract[data,x]
Out[]=
12
Notice an important detail: we should also prevent symbol s from being evaluated in the call to SymbolName, which is done by calling Unevaluated, as Hold will only protect s from evaluation when first seen by extract.
Transforming expressions
Transforming expressions
Let’s raise the stakes: we want to extend extract to accept any expression with symbols mapped to keys of data, such that extract[data, x*y + x/y] evaluates to . One way to do this is to define a function based on the expression x*y + x/y, and apply it to data:
12×20+12/20=1203/5
In[]:=
f[d_]:=d["x"]*d["y"]+d["x"]/d["y"]f[data]
Out[]=
1203
5
Building such function automatically based on the expression input will require messing with Wolfram Language standard evaluation process. We will accomplish it with the following steps:
◼
Extract a list of symbols from the expression
◼
From that, create a list of replacement rules of the form x->d[“x”]
◼
Transform the expression using these rules
◼
Package everything into an anonymous function
For the first step, we can try to extract everything that has Symbol as a Head:
In[]:=
SetAttributes[symbols,HoldFirst];symbols[expr_]:=DeleteDuplicates@Cases[Unevaluated@expr,_Symbol,Infinity];
In[]:=
Clear[x,y];symbols[x*y+x/y]
Out[]=
{x,y}
The above definition of symbols has two problems: firstly, we only need to extract the symbols that are present as keys in the association, and secondly, it only works properly if none of the symbols in the expression has any values assigned, as illustrated by the following examples:
In[]:=
symbols[x+Pi]x=1;symbols[x*y+x/y]
Out[]=
{x,π}
Out[]=
{1,y}
Let’s solve the second problem first. We can use the fact that Cases can transform expression that it finds, and prevent them from evaluation with HoldForm, which works like Hold but doesn’t print the head in the output form.
In[]:=
symbols[expr_]:=DeleteDuplicates@Cases[Unevaluated@expr,s_SymbolHoldForm[s],Infinity];{x,y}={1,2};symbols[x*y+x/y]
Out[]=
{x,y}
And since we already transforming the symbols we found with Cases, we can generate a list of replacement rules x->d[“x”] right here as well:
In[]:=
SetAttributes[rules,HoldFirst];rules[expr_,param_]:=DeleteDuplicates@Cases[Unevaluated@expr,s_Symbol:>(HoldPattern[s]param[SymbolName[Unevaluated@s]]),Infinity];{x,y}={1,2};rules[x*y+x/y,d]//FullForm
Out[]//FullForm=
List[Rule[HoldPattern[x],d["x"]],Rule[HoldPattern[y],d["y"]]]
Notice a subtle but important detail: we used :> (RuleDelayed) instead of Rule in the second argument to case, as we don’t want the pattern to be evaluated at the time of the definition. Here is what happens if we use Rule instead:
In[]:=
SetAttributes[badRules,HoldFirst];badRules[expr_,param_]:=DeleteDuplicates@Cases[Unevaluated@expr,s_Symbol->(HoldPattern[s]param[SymbolName[Unevaluated@s]]),Infinity];badRules[x*y+x/y,d]//FullForm
Out[]//FullForm=
List[Rule[HoldPattern[x],d["s$"]],Rule[HoldPattern[y],d["s$"]]]
As you can see, SymbolName[Unevaluated@s] was evaluated at the time of the definition (before s was assigned a value), resulting in constant “s$” name for each variable, which is not what we want.
And to address the first concern (capturing names like Pi that we don’t want to capture), we can introduce additional parameters keys which should be a list of names we want to capture:
In[]:=
rules[expr_,param_,keys_]:=DeleteDuplicates@Cases[Unevaluated@expr,s_Symbol:>(HoldPattern[s]param[SymbolName[Unevaluated@s]])/;MemberQ[keys,SymbolName[Unevaluated@s]],Infinity];{x,y}={1,2};rules[x*y+x/y+Pi,d,{"x","y"}]//FullForm
Out[]//FullForm=
List[Rule[HoldPattern[x],d["x"]],Rule[HoldPattern[y],d["y"]]]
Next, we’ll define a function transformer that completes third and forth steps of our plan:
In[]:=
SetAttributes[transformer,HoldFirst];transformer[expr_,keys_]:=Module[{param},Function[Evaluate@param,Evaluate[ReleaseHold[Hold[expr]/.rules[expr,param,keys]]]]]
There a couple of points related to the evaluation. First, Function is not evaluating its arguments by default, but in our case we want them to be evaluated. Second, we need to protect expr with Hold, as we need the substitution to happen before the evaluator can see the values of the symbols, but we can release the hold once the substitution is being made. Let’s see what function is generated from our expression:
In[]:=
transformer[x*y+x/y+Pi,{"x","y"}]
Out[]=
Functionparam$85973,π++param$85973[x]param$85973[y]
param$85973[x]
param$85973[y]
This is exactly what we need - and we can now successfully define extract!
In[]:=
extract[d_,expr_]:=transformer[expr,Keys[d]][d];extract[data,x*y+x/y]
Out[]=
1203
5
Implementing filter for datasets
Implementing filter for datasets
We are now equipped to tackle our first challenge: implementing a filter function similar to dplyr.filter. We’ll be using the flights dataset, so that all of our examples are directly comparable to Chapter 5: Data Transformation from the “R for Data Science” book. The idea is that you can do filter(flights, month == 1, day == 1) and receive all the flights departed on Jan 1.
Here is how we would solve this with standard Wolfram tools:
In[]:=
Dataset`$DatasetTargetRowCount=10;(*tolimitthenumberofrowsinthedefaultdisplay*)SetDirectory[NotebookDirectory[]];<<datasets/flights.mx
In[]:=
flights[Select[#month1&&#day1&]]
Out[]=
Let’s do this manually using transformer, and then package everything up in a nice interface. First, we’ll get the names of the columns in flights as a list. Since we’ll need to extract these column names quite often, we’ll package this operation as a function:
We can immediately use transformer like this, which gives exactly the same result as the above.
The semantics for R’s filter is that a few conditions can be given as separate arguments and they all will be joined with And, so we can define filter as following:
We define it as an operator, and you can see that it works as expected and that filter is immune to variables such as month and day having a value:
Implementing select for datasets
Implementing select for datasets
Next one is select. It semantics in R allows for narrowing on variables you are interested in. In addition to its simplest form - just listing the variables - select offers a few additional helpers:
◼
year:day will select all variables from year to day, i.e., year, month and day for the above dataset
◼
-(year:day) will select all variables except those from year to day
◼
There are also a few helper functions, such as starts_with, ends_with, contains and matches, that select variables with corresponding names.
Let’s cover the easiest case first. Here is how we can select given columns using the dataset interface
Let’s write a function that takes one or few symbols as arguments and returns a list of their names. A naive approach would simply use Map and SymbolName:
Unfortunately, that doesn’t work if one of the symbols has a value assigned, as Map will evaluate its arguments:
I wish there would be a simpler solution, but here is the one I came up with that protects all the symbols from the evaluation:
A pair of Hold / ReleaseHold and using s_Symbol:>SymbolName[Unevaluated@s] idiom should be already familiar from the previous section. The only other non-trivial part is the levelspec argument for Replace: we don’t want the replace to happen at level 0 (the whole expression), as it will also transform Hold into a string, or at level 1, as it will transform List.
With namer defined, we can now implement the simplest form of select:
Let’s tackle the next case : select should also accept expressions like year:day. We define a helper function to select ranges
We’ll extend the definition of namer to accept a list of available keys, and we’ll add two more rules to Replace, keeping in mind that full form of a:b is Pattern[a,b].
It works as expected, but notice all the trickery with Level, especially for the last two patterns. I haven’t found easier ways to deal with Pattern destructuring in replacement rules, so any advice would be appreciated.
That is a long post already - I will stop here for now, and if/when there is enough interest, I would publish a second part, which aims to define the remaining dplyr functions.