With its extensive variety of functions and resources, the Wolfram Language can be hard to completely learn and grasp—there’s always an easier way to do the same thing. We thus wanted to build a tool that assists new programmers in the Wolfram Language by helping them materialize their ideas using Wolfram Language’s resources, primarily functions from Wolfram’s documentation. The end result is a computational exploration of the problem—to help new programmers implement their ideas in WL by predicting functions from natural language input in English—using multiple approaches.
Introduction
Introduction
As programmers in the Wolfram Language, we’ve all occasionally spent hours looking through the Wolfram documentation to find functions that’d help achieve our purpose. We’ve all also hoped for an easier way to do this without having to read through all the functions that come up from “search” or aimlessly scrolling through the documentation. Well, we have just the solution for you! While there have been computational explorations of generating code from natural language input using GPT-3 and Python transformers, this has been slightly limited in Wolfram Language (to the best of our knowledge). With that in mind, over the past semester, we created multiple functions where one can input their code’s general requirements in English and output a list of functions that can potentially be used. Even though it doesn’t tell you exactly what to do, it can provide some direction to new programmers looking for inspiration.
Approach 1: Using Sentence N-grams
Approach 1: Using Sentence N-grams
An approach to selecting WL functions based on an input prompt is to split the prompt into tokens or words and find functions that match a certain sequence of tokens or words. N-grams refer to subsequences of words within sentences. By analysing the n-grams in a prompt, we can determine which WL functions are most appropriate for answering it. N-grams have been more formally defined in subsequent sections.
Removing Stop Words from a Sentence
Removing Stop Words from a Sentence
A stop word is a word that contains little information. Excluding these words while processing sentences may make our approaches more efficient and accurate. For instance, the article ‘the’ can be considered a stop word. Consider the input prompt: ‘Blend the colors red and yellow’ and the description of the function Integrate: ‘gives the indefinite integral...’ If we considered ‘the’, it would match with Integrate. However, the prompt has nothing to do with integration. Hence, this match is undesirable. Thus, removing stop words will improve our results and reduce the number of words needed to be processed.
WL has an inbuilt list of stop words.
In[]:=
WordList["Stopwords"]
Out[]=
{0,1,2,3,4,5,6,7,8,9,a,A,about,above,across,after,again,against,all,almost,alone,along,already,also,although,always,am,among,an,and,another,any,anyone,anything,anywhere,are,aren't,around,as,at,b,B,back,be,became,because,become,becomes,been,before,behind,being,below,between,both,but,by,c,C,can,cannot,can't,could,couldn't,d,D,did,didn't,do,does,doesn't,doing,done,don't,down,during,e,E,each,either,enough,even,ever,every,everyone,everything,everywhere,f,F,few,find,first,for,four,from,full,further,g,G,get,give,go,h,H,had,hadn't,has,hasn't,have,haven't,having,he,he'd,he'll,her,here,here's,hers,herself,he's,him,himself,his,how,however,how's,i,I,i'd,if,i'll,i'm,in,interest,into,is,isn't,it,it's,its,itself,i've,j,J,k,K,keep,l,L,last,least,less,let's,m,M,made,many,may,me,might,more,most,mostly,much,must,mustn't,my,myself,n,N,never,next,no,nobody,noone,nor,not,nothing,now,nowhere,o,O,of,off,often,on,once,one,only,or,other,others,ought,our,ours,ourselves,out,over,own,p,P,part,per,perhaps,put,q,Q,r,R,rather,s,S,same,see,seem,seemed,seeming,seems,several,shan't,she,she'd,she'll,she's,should,shouldn't,show,side,since,so,some,someone,something,somewhere,still,such,t,T,take,than,that,that's,the,their,theirs,them,themselves,then,there,therefore,there's,these,they,they'd,they'll,they're,they've,this,those,though,three,through,thus,to,together,too,toward,two,u,U,under,until,up,upon,us,v,V,very,w,W,was,wasn't,we,we'd,we'll,well,we're,were,weren't,we've,what,what's,when,when's,where,where's,whether,which,while,who,whole,whom,who's,whose,why,why's,will,with,within,without,won't,would,wouldn't,x,X,y,Y,yet,you,you'd,you'll,your,you're,yours,yourself,yourselves,you've,z,Z}
This list of stop words, however, includes numeric characters, I (which could be used while representing complex numbers) and words like ‘first’, ‘last’, ‘one’, etc., all of which convey useful information that might help us while searching for functions.
Thus, we can define our own list of stop words:
In[]:=
stopWords={"a","A","about","above","across","again","against","almost","alone","along","already","also","although","always","am","among","an","and","another","anyone","anything","anywhere","are","aren't","around","as","at","b","B","back","be","became","because","become","becomes","been","before","behind","being","below","but","by","c","C","can","cannot","can't","could","couldn't","d","D","did","didn't","do","does","doesn't","doing","done","don't","down","during","enough","ever","everyone","everything","everywhere","f","F","few","find","for","from","full","further","get","give","go","h","H","had","hadn't","has","hasn't","have","haven't","having","he","he'd","he'll","her","here","here's","hers","herself","he's","him","himself","his","how","however","how's","i'd","i'll","i'm","in","interest","into","is","isn't","it","it's","its","itself","i've","j","J","k","K","keep","l","L","let's","m","M","made","many","may","me","might","mostly","much","must","mustn't","my","myself","n","N","never","no","nobody","noone","nor","now","nowhere","o","O","of","off","often","on","once","only","or","other","others","ought","our","ours","ourselves","out","over","own","p","P","part","per","perhaps","put","q","Q","r","R","rather","s","S","same","see","seem","seemed","seeming","seems","several","shan't","she","she'd","she'll","she's","should","shouldn't","show","side","since","so","someone","something","somewhere","still","such","t","T","take","than","that","that's","the","their","theirs","them","themselves","then","there","therefore","there's","these","they","they'd","they'll","they're","they've","this","those","though","through","thus","to","together","too","toward","u","U","under","until","up","upon","us","v","V","very","w","W","was","wasn't","we","we'd","we'll","well","we're","were","weren't","we've","what","what's","when","when's","where","where's","whether","which","while","who","whole","whom","who's","whose","why","why's","will","with","within","without","won't","would","wouldn't","yet","you","you'd","you'll","your","you're","yours","yourself","yourselves","you've"};
Function to remove stop words from a sentence and join the remaining words with a space:
In[]:=
RemoveStopWords[sentence_]:=StringJoin@@Riffle[Select[TextWords[sentence], Not[MemberQ[stopWords,# ]]&], " "]
In[]:=
RemoveStopWords["Blend the colors red and yellow."]
Out[]=
Blend colors red yellow
In the previous example, the stop words ‘the’ and ‘and’ were omitted. In addition to removing stop words, we can convert the sentence to lower case.
In[]:=
PreProcessPrompt[sentence_]:=ToLowerCase[RemoveStopWords[sentence]]
In[]:=
PreProcessPrompt["Blend the colors red and yellow."]
Out[]=
blend colors red yellow
N-grams in a Sentence
N-grams in a Sentence
An n-gram is defined as a string containing n consecutive words of a sentence. A unigram simply refers to 1 consecutive word of a sentence. Thus, every word in a sentence, on its own, is a unigram. Thus, for a sentence with m words, there would be m unigrams.
A bigram is a pair of consecutive words. There are (m-1) bigrams in a sentence of m words. Similarly, there are (m-2) trigrams, (m-3) 4-grams and so on. There is only one m-gram, which is the entire sentence itself. N-grams are the set of all unigrams, bigrams, trigrams ... m-grams in the sentence. Thus, there are m + (m-1) + (m-2) + ... + 1, or simply m(m+1)/2 n-grams.
We can use the subsequences function to find all n-grams of a sentence.
In[]:=
prompt="Blend the colors red and yellow.";
Splitting the sentence into individual words after pre-processing it:
In[]:=
TextWords[PreProcessPrompt[prompt]]
Out[]=
{blend,colors,red,yellow}
All unigrams in the sentence:
In[]:=
Subsequences[TextWords[PreProcessPrompt[prompt]],{1}]
Out[]=
{{blend},{colors},{red},{yellow}}
All bigrams in the sentence:
In[]:=
Subsequences[TextWords[PreProcessPrompt[prompt]],{2}]
Out[]=
{{blend,colors},{colors,red},{red,yellow}}
All n-grams in the sentence:
In[]:=
Subsequences[TextWords[PreProcessPrompt[prompt]]]
Out[]=
{{},{blend},{colors},{red},{yellow},{blend,colors},{colors,red},{red,yellow},{blend,colors,red},{colors,red,yellow},{blend,colors,red,yellow}}
Since Subsequences always returns {} as a valid subsequence, we must exclude it from the set of n-grams. This can be done using Rest:
In[]:=
Ngrams[sentence_]:=Rest[Subsequences[TextWords[PreProcessPrompt[sentence]]]]
In[]:=
Ngrams[prompt]
Out[]=
{{blend},{colors},{red},{yellow},{blend,colors},{colors,red},{red,yellow},{blend,colors,red},{colors,red,yellow},{blend,colors,red,yellow}}
We can verify that there are indeed m(m+1)/2 n-grams, where m is the number of words in the prompt.
In[]:=
m=Length[TextWords[PreProcessPrompt[prompt]]];m(m+1)/2==Length[Ngrams[prompt]]
Out[]=
True
Obtaining Function Names
Obtaining Function Names
Now that we have found all possible n-grams in the input prompt, we have to try and match these n-grams to functions from the Wolfram Documentation. To accomplish this, we must first find a list of all function names.
Using the Name property of WolframLanguageData, we can find the names of each function:
In[]:=
functions=EntityValue["WolframLanguageSymbol", "Name"];
Some of the symbols returned are not useful. These begin with $. All other useful functions begin with an upper-case letter.
In[]:=
functions[[;;10]]
Out[]=
{$Aborted,$ActivationKey,$AllowDataUpdates,$AllowExternalChannelFunctions,$AllowInternet,$AssertFunction,$Assumptions,$AudioDecoders,$AudioEncoders,$AudioInputDevices}
We will be considering all 5710 symbols from the Wolfram Language that do not start with $.
In[]:=
Length[functions]
Out[]=
5934
In[]:=
functions=Select[EntityValue["WolframLanguageSymbol", "Name"], LetterQ[First[Characters[#]]]&];
functions[[;;20]]
Out[]=
{AASTriangle,APIFunction,ARCHProcess,ARIMAProcess,ARMAProcess,ARProcess,ASATriangle,AbelianGroup,Abort,AbortKernels,AbortProtect,Above,Abs,AbsArg,AbsArgPlot,AbsoluteCorrelation,AbsoluteCorrelationFunction,AbsoluteCurrentValue,AbsoluteDashing,AbsoluteFileName}
A problem that we can see is that some function names consist of multiple words. To match n-grams to function names, we would have to split function names into their constituent words. Consider AbortProtect for instance. Every new word begins with an upper-case letter and lower-case letters belong to the previous word. In the case of AASTriangle, it is not that straightforward. Although AAS are all upper-case letters, only the first A marks the beginning of a new word. We can write out a set of rules:
The examples below demonstrate that SplitFunctionName works, however, it adds extra whitespace to separate words.
To convert the output of SplitFunctionName into a list of words, we can make use of TextWords. We can also convert all function names to lower case.
Extracting function words from all functions:
Word Stemming
Word Stemming
Now that we have the function names as well as the function names, we can try to find exact matches between them. However, if our prompt makes use of the phrase “AAS Triangles”, it will not match with the function “AASTriangle” because of the fact that the plural form of triangles has been used in the prompt. Direct matching also allows the possibility of inflected forms (word+ing, word+ed, etc.) to go unrecognized. Thus, we have to find a way to ensure that, for example, color, coloring, colors, and colored are all interpreted as forms of the same base word.
A possible approach would be reverting each word to its base or root form. A technique to do this is known as word stemming. A word stemming algorithm essentially involves a set of replacement rules. Some of these rules are:
◼
word + ing → word
◼
word + ed → word
◼
word + s → word
◼
word + ies → word + i
More complicated stemming rules may also be included, such as:
◼
word + fulness → word + ful
◼
word + ational → word + ate
◼
word + ate → word
All rules may not be applicable for all words. Some rules may only be applied to words with specific orderings of consonants and vowels. Moreover, rules may also be applied multiple times. To perform word stemming, we can use the inbuilt WordStem function of the Wolfram Language. This makes use of a technique known as Porter Stemming.
Consider the following examples:
In the first example, despite the word national containing the suffix ‘ational’, it did not satisfy the criteria required for the word + ational → word + ate replacement to be applied. Instead, another replacement rule word + al → word has been used. For the second example, the word international satisfies the criteria required for the word + ational → word + ate replacement. Thus, international → internate. Since replacement rules may be applied repeatedly and internate satisfies the criteria for word + ate → word. Thus, internate → intern.
The algorithm may also result in some false positives, albeit infrequently. Both interns and international have the same word stem: ‘intern’
Since we intend to compare word stems instead of the words themselves, we can make modifications to the functions that we defined in previous sections.
All occurrences of the word ‘colors’ have been stemmed to ‘color’:
Stemming all words obtained from splitting function names:
Matching N-grams to Functions
Matching N-grams to Functions
Now, to determine which parts of a sentence correspond to which functions, we can look for matches between sequences of stemmed n-grams and n-grams of stemmed function words.
First, let us find n-grams for each function name:
We can define the following function:
We get a lot of results. Each of the results is of the form ngram → function.
We can re-organize the data so that it is in the form ngram → all functions:
Although we have been able to obtain a set of functions based on user input, this list is too large. We need to narrow it down further.
Sorting by Frequency
Sorting by Frequency
The best way to narrow down the list of functions obtained would be to rule out functions that aren’t used frequently by users. Wolfram Language Data contains a property ‘Frequencies’, using which we can determine the frequency of usage of each function.
The Frequencies property gives us the frequency of usage across a variety of platforms. To simplify this data, we will simply consider the value of ‘All’
Obtaining frequencies for each function:
Mapping functions to frequencies:
Sorting matches by frequency of usage:
We can then consider the top 3 function matches for each n-gram:
Consolidating everything into a single function:
Some examples:
◼
Blend the colors red and yellow
◼
Find the number of digits in 2^128
◼
Integrate the function sin(x+3)
Approach 2: Using String Similarity
Approach 2: Using String Similarity
With the N-gram approach, in some cases, we may not find a direct match to the function that we require. For example, if the user needs to construct a list, they may require ‘Table’ or ‘Range’, both of which do not include the word list in their names. However, the descriptions of these functions may contain the word ‘list’. Therefore, using string similarity functions to compare the description of the function to words in the prompt may yield a useful result.
Accessing Function Descriptions
Accessing Function Descriptions
We can access a function’s description from Wolfram Language Data:
Pre-processing the description:
Descriptions for all functions:
Pre-processing a sample prompt:
Now that we have a list of words from both the function description and the prompt, we can try to find matches using string similarity.
Levenshtein Distance (Edit Distance)
Levenshtein Distance (Edit Distance)
The most common method of measuring the similarity (or rather dissimilarity) between two strings is edit distance or Levenshtein distance. This is defined as the number of one-element additions, deletions or substitutions required to transform one word to another.
We can use the EditDistance function to compute this metric
Levenshtein distance is particularly good at detecting typos or minor errors. This is because typos usually have a different character or one character less or more than the target string. Thus, the edit distance for most typos would be 1, which indicates a close match. However, in this case, typos may be rare. Input prompts are short and may not contain many mistakes. Moreover, the WL documentation also, presumably, has very few typos. Levenshtein distance may not be the most appropriate measure of string (dis)similarity. The example below further illustrates this:
If a user’s prompt includes the word ‘random’, it may not match with ‘pseudorandom’ due to the high value of edit distance. In fact, according to edit distance, the word random is closer to ‘kingdom’. This is not desirable as edit distance may result in a lot of false positives.
Smith-Waterman Similarity
Smith-Waterman Similarity
A similarity measure that may be more appropriate to solve the random-pseudorandom match problem is Smith-Waterman Similarity. This approach finds the optimum local alignment between the two words and returns the number of character matches.
Thus, we can use Smith-Waterman Similarity to match words from the prompt with the function description. To bring all similarity scores between 0 and 1, we can divide the similarity score by length of the target string, or in this case, the length of ‘random’.
Thus, our similarity function is:
Matching Functions to Prompt Words
Matching Functions to Prompt Words
We can define a string similarity function. For every function description, it takes the word with the maximum similarity as that function’s similarity score.
Matches for the word ‘random’
Matches for the word ‘pink’
Now, we can extend this approach to the entire sentence:
The top 10 functions for each word:
Sub-sorting by frequency:
Similarly, for another prompt:
This result is not ideal, especially for the word ‘red’. It returns seemingly unrelated functions. This is probably because other, unrelated words may have matched closely with the word ‘red’. For instance, both ‘red’ and ‘hundred’ would have a similarity score of 1 with ‘red’. It is challenging to fix this problem. However, we can try to combine similarity scores across all words and hope for a better result.
We can combine all individual prompt similarities into a composite association by averaging together the similarity scores of each function for each word:
Top 10 composite matches sorted by frequency:
Although we are able to get ‘Blend’, this approach is not as promising as the n-gram matching approach used previously.
Approach 3: Finding Similar Examples
Approach 3: Finding Similar Examples
Another type of information available to us are Documentation Examples for each function. If we can match the input user prompt to a documentation example, it will be easy to determine what functions are required by the user to solve their query.
Accessing Documentation Examples
Accessing Documentation Examples
Using the DocumentationBasicExamples property, we can find a set of examples for each WL function. Each example is structured as a list of 3 items: the prompt, WL code, and the output of the computation.
Isolating the prompt and the WL code (by excluding the output):
However, this property does not allow us to access ALL the examples available in the documentation. There are still some additional examples under ‘Options’ and ‘Scope’. In fact, Wolfram Language Data does not allow users to directly access all examples within a function. We will have to make use of two properties — “DocumentationExampleText” and “DocumentationExampleInputs”.
This is a very large set (444 examples!) of examples. Despite the possibility that processing all of these prompts may significantly improve the performance of our function, we will only consider basic examples because of computational time concerns.
This is a very large set (444 examples!) of examples. Despite the possibility that processing all of these prompts may significantly improve the performance of our function, we will only consider basic examples because of computational time concerns.
Extracting Words from Prompt
Extracting Words from Prompt
The prompt itself is not a String. It is a RawBox.
Viewing the structure of the prompt expression in tree form may help us understand it better:
The example above illustrates that the prompt ‘Indefinite integral:’ has a simple structure. Within each RawBoxes there is a Cell, which contains the actual prompt as well as a property called ExampleText.
We can confirm this by converting the example into a string directly:
Since we know the structure of the prompt, we can attempt to extract information from it:
However, our function does not work on all prompts:
In this case, the data has been contained inside TextData and must be extracted from that.
Rewriting our original function:
Finally, we can take words from the extracted prompt:
Thus, we have been able to extract words from each prompt.
Extracting Data from WL Code
Extracting Data from WL Code
The WL code inputs are also stored as RawBoxes. Thus, we will have to extract functions from RawBoxes as well. To solve this problem, we can try to look at the tree structure for one such expression.
We can make some observations about the tree structure shown above. Every RawBoxes contains a Cell, which contains some additional properties. Cells may contain either BoxData or TextData (as seen in the case of prompts), which usually contains useful information. Within BoxData, we see RowBoxes or Lists of RowBoxes. Within these RowBoxes or Lists of RowBoxes, we see WL functions!
Thus, we have been able to extract WL functions from each code input.
Preparing Example Data for top WL Functions
Preparing Example Data for top WL Functions
We discovered that it takes a LOT of time to access the basic examples for each function. Thus, to make computation easier, we decided to consider the top 500 most frequently used functions and use basic examples from those top functions only.
The top 500 functions are used 97.8% of the time:
This is reasonable since functions that are NOT a part of the top 500 most frequently used are presumably specific functions (for e.g. ‘MittagLefflerE’) which can be better identified by the n-grams or string similarity approaches.
We have 1552 examples in total:
Selecting the examples that have the form prompt-WL code-output: (this needs to be done because some examples may not contain prompts; these examples will not be helpful to us)
This rules out 50 examples, which isn’t a large number.
Some examples may also not make use of any useful functions or have any useful words in them. These have been ruled out in processedPrompts. This was about 160 more examples.
Matching User Prompt to Documentation Example
Matching User Prompt to Documentation Example
Now, we need to match the user’s prompt to an existing documentation example. To accomplish this, we can consider using our n-grams approach: generating a set of n-grams from the prompt and matching them to the documentation examples. The main drawback to this approach is that n-grams do not account for non-consecutive words. For instance, the phrase ‘randomly chosen integers’ won’t contain the n-gram ‘randomly integers’ since these two words are not consecutive. Thus, it may not match with an example from the RandomInteger function. Instead, we can consider Intersection. The order of the words in the matched example is also not important as long as all keywords are present.
Consolidating, Visualising Approaches
Consolidating, Visualising Approaches
To present our results to the user, we would have to synthesize all of our findings using the three approaches. This section aims to consolidate the outputs of all approaches into a single visualisation.
Preparing Visualisations
Preparing Visualisations
Since we are dealing with stemmed forms of words, we need a way to obtain the original word in the sentence from its stemmed form for the final visualisation. We can use an association of stemmed words mapped to their original forms and query the association each time we require the original word.
Constructing the stemmed words association for an input sentence:
It would also be quite convenient for users to be able to click on a function to directly access its documentation. To accomplish this, we can replace each function with its Wolfram Documentation Link in the visualisation.
Obtaining links for each function:
Creating a function to convert a list of functions (each in String form) to URLs:
Click on a link to access the function’s documentation:
The data obtained from Approach 1 can best be visualised through a dataset table:
Since Approach 2 is similar to Approach 1, we can consider the dataset table visualisation for this approach as well:
For Approach 3, we can display each relevant example’s prompt, the WL code and the functions that we extract from the WL code in the form of a dataset:
Lastly, to make the visualisations more visually appealing, we can add WordClouds for a summary of function matches given by Approach 1 and 2:
Putting everything together into a single visualisation:
Summary and Examples
Summary and Examples
Example 1
Example 1
Although the input prompt was relatively straightforward, Approaches 1 and 2 were unable to identify the function ‘Blend’. A similar example (‘Generate an array of blended colors ranging from red to blue’) was identified, which contained the function ‘Blend’. The reason we weren’t able to identify ‘Blend’ is because the strings ‘mix’ and ‘blend’ have low similarity even though they essentially have the same meaning. Thus, to find semantic matches or matches of strings that have the same meaning, we could consider the synonyms of a string in our approaches.
The synonyms of ‘mix’ contains ‘blend’:
Improvements and Future Scope
Improvements and Future Scope
◼
Our approaches do not consider words with similar meaning. For example, if the user enters the prompt ‘mix two colors’, our approaches output functions like ‘MixedUnit’ instead of ‘Blend’. To account for word meaning, we could consider the ‘Synonyms’ function while searching for n-gram matches. We could also use an ML approach to quantify word meaning through Word Vectors. Using Wolfram Net Repository’s ELMo Contextual Word Representations model, we could represent words as vectors and find matches between vectors to determine relevant functions.
◼
We could consider the types of data inputted by the user. For example, if the user inputs the prompt ‘mix the colors red and blue’, we can identify that red and blue are colors, hence we know that the functions involved have something to do with color. Thus, ‘Blend’ becomes a better recommendation than ‘MixedUnit’. Data in the user's prompt could be identified as follows:
The function ‘Blend’ has been categorised into something called ‘ColorSymbols’. Thus, if we identify colors in the user’s prompt, we should check to see if our recommended function is also a ColorSymbol. Unfortunately, these ‘FunctionalityAreas’ are largely undocumented.
◼
We could try to consider a greater number of examples. Currently only Documentation Basic Examples of the top 500 most frequently used functions have been considered. If, instead, all examples from Options, Scope and Applications for all 5000+ functions are considered, the performance of Approach 3 will improve. We could also consider alternate sources for examples, such as EIWL.
◼
Functions from WFR can also be included in our approaches in the future.
◼
To make our function more accessible, we publish it as a Resource Function. Alternatively, we could deploy it to the Cloud as a FormFunction. When we experimented with CloudDeploy, we discovered that the Dataset visualisations did not load properly on the web. Currently, we are working on fixing the web version.
Conclusion
Conclusion
In conclusion, we were able to achieve our aim of recommending Wolfram Language functions and examples based on the user's prompt. WELP 2021 was an interesting and enjoyable learning experience; we explored NLP techniques including stemming, lemmatization and string similarity and used Wolfram Language Data. While this function is already useful, it can be improved significantly using the approaches outlined in the previous section. Currently, we are working on making this function more accessible to new WL developers through a resource function or a web interface.