Abstract
Abstract
Neural Networks, and Machine Learning in general, have been at the forefront of technological innovation in recent years because of the power that these technologies offer. While these technologies are undoubtedly the most advanced methods we currently possess for tasks like image classification, object detection and much more, it has been observed that their ability to solve certain kinds of problems, for example, those in the realm of calculus, is not nearly as good.
This is interesting since the idea of predicting things in the field of calculus was explored even before computers were invented (Note: private conversation with Stephen Wolfram), but even then, our ability to make predictions about the answers in calculus without actually solving the problems explicitly was barely notable.
As part of this project, I suggest a new method of representing symbolic mathematical data quantitatively and highlight some of the successes and failures of this method. I also analyze the performance of some neural networks on data trained using this method.
This is interesting since the idea of predicting things in the field of calculus was explored even before computers were invented (Note: private conversation with Stephen Wolfram), but even then, our ability to make predictions about the answers in calculus without actually solving the problems explicitly was barely notable.
As part of this project, I suggest a new method of representing symbolic mathematical data quantitatively and highlight some of the successes and failures of this method. I also analyze the performance of some neural networks on data trained using this method.
Introduction
Introduction
Integrals find their application in several different fields, ranging from the engineering associated with microscopic devices to space flight and everything in between. Despite the frequent use of this mathematical construct, especially in academic and theoretical settings, we have been largely unsuccessful in finding a robust method to predict the functions that one would obtain on integrating an integrand without having to explicitly calculate it. The ability to make such predictions would enable us to gain some insight into the possible answers of integrals that are considered to be "unsolvable" today. It would appear that this ability could also find uses in a lot of complicated theoretical research taking place today.
The goal of this project is to lay down a new approach to the quantification of symbolic mathematical expressions and use this method in training a Machine Learning model to predict the functions that will be obtained upon integrating a given integral. In addition, this project also provides an analysis of some of the successes and failures of this quantification method and suggests some possible modifications that could be implemented to potentially further improve the model. It also compares the performance on a few different types of Neural Networks when trained on the data obtained using this method.
The goal of this project is to lay down a new approach to the quantification of symbolic mathematical expressions and use this method in training a Machine Learning model to predict the functions that will be obtained upon integrating a given integral. In addition, this project also provides an analysis of some of the successes and failures of this quantification method and suggests some possible modifications that could be implemented to potentially further improve the model. It also compares the performance on a few different types of Neural Networks when trained on the data obtained using this method.
Initial Data Collection
Initial Data Collection
Given the lack of a dataset of integrands and their corresponding integrals, I had to curate a dataset of my own that could be used for the purpose of this project. This data was collected from 2 different sources:
◼
Wolfram|Alpha Problem Generator
◼
A Coded Random Function Generator
API Calls to the Wolfram|Alpha Problem Generator
API Calls to the Wolfram|Alpha Problem Generator
I first created a function called importIntegrals that runs a call to a Wolfram|Alpha API and scrapes the data of integrals from integration questions generated by the Wolfram|Alpha Problem Generator (do not try to run this function since the user ID and API key have been removed from the URL for security purposes).
importIntegrals[category_String]:=Module[ alphaIntegrals,link="https://www.wolframcloud.com/objects/user- (*ID removed*)/WolframProblemGenerator API?_key=(*Key removed to maintain security*)"<>category<>"&HowMany=3&Difficulty=Advanced", alphaIntegrals= Drop[ Drop[ Flatten[ Import[ link]], 1], -3]; Return[alphaIntegrals]]
A sample of the data obtained upon using this function can be seen upon expanding the output of the code below:
alphaInput=importIntegrals["BasicIntegrate"]
Out[]=
{"20207153171":{,"Question":"Compute integral ((3 t^2)\/2-2\/t^3+t+3\/2 sin(3 t)) dt.",,"Answer":"t^3\/2+t^2\/2+1\/t^2-1\/2 cos(3 t)+constant",,"Distractors":[,"-2 t^3-2 t^2+1\/t^2+2 cos(3 t)+constant",,"6\/t^4+3 t+9\/2 cos(3 t)+1+constant",,2 t^3+2 t^2+1\/t^2-1\/2 cos(3 t)+constant,],},,"20207153172":{,"Question":"Evaluate integral (-5\/(2 x^2)+2\/x^3+3\/(2 x)+(3 cos(x))\/2) dx.",,"Answer":"(3 ln(x))\/2-1\/x^2+5\/(2 x)+(3 sin(x))\/2+constant",,"Distractors":[,"(2 ln(x))\/3-1\/x^2+2\/(5 x)+(2 sin(x))\/3+constant",,"(3 ln(x))\/2-1\/x^2+5\/(2 x)+(3 sin(x))\/(2 x)+constant",,(3 ln(x))\/2-1\/x^2+10\/x+(3 sin(x))\/2+constant,],},,"20207153173":{,"Question":"Evaluate integral (3 t^4+2\/t^4-2 t+5\/2 sin(2 t)) dt.",,"Answer":"(3 t^5)\/5-t^2-2\/(3 t^3)-(5 cos^2(t))\/2+constant",,"Distractors":[,"12 t^3-8\/t^5+5 cos(2 t)-2+constant",,"15 t^5-t^2-6\/t^3-(5 cos^2(t))\/2+constant",,-(3 t^5)\/5-t^2+2\/(3 t^3)+(5 cos^2(t))\/2+constant}
The data obtained from the Problem generator had questions that were structured in a few different ways. The questions imported above (imported from the Advanced questions of the BasicIntegrate category) were available in 2 different formats. While the formats were different for different categories, the definition of the function extractQuesText given below enables the extraction of the questions from both the formats of the BasicIntegrate category's Advanced questions.
extractQuesText[input_List]:=Module[ {questions,fixedQuestions}, questions=Join[ (StringSplit[#,":"][[-1]])&/@{input[[2]],input[[11]],input[[20]]} ]; fixedQuestions=Join[ (StringReplace[ToString[#],"\\/""/"])&/@questions ]; If[StringContainsQ[#,"Evaluate"], Replace[#,# StringDrop[StringDrop[ToString[#],21],-7]], Replace[#,#->StringDrop[StringDrop[ToString[#],-7],20]] ]&/@fixedQuestions]
quesList=extractQuesText[alphaInput]
Out[]=
{(3 t^2)/2-2/t^3+t+3/2 sin(3 t),-5/(2 x^2)+2/x^3+3/(2 x)+(3 cos(x))/2,3 t^4+2/t^4-2 t+5/2 sin(2 t)}
Next, the answers corresponding to the extracted questions had to be extracted as well. This was done using a similar function for the extraction of answers. The answers were always listed in a single format so this function was applicable for all categories and difficulty levels.
extractAnsText[input_List]:=Module[ {answers,fixedAnswers}, answers=Join[ (StringSplit[#,":"][[-1]])&/@{input[[3]],input[[12]],input[[21]]} ]; fixedAnswers=Join[ (StringReplace[ToString[#],"\\/""/"])&/@answers ]; Replace[#,# StringDrop[StringDrop[ToString[#],-2],1]]&/@fixedAnswers]
ansList=extractAnsText[alphaInput]
Out[]=
{t^3/2+t^2/2+1/t^2-1/2 cos(3 t)+constant,(3 ln(x))/2-1/x^2+5/(2 x)+(3 sin(x))/2+constant,(3 t^5)/5-t^2-2/(3 t^3)-(5 cos^2(t))/2+constant}
I also observed that all the questions did not contain just one variable. Some used 'x' as the variable and others used 't'. Thus, I also wrote a function to analyze the extracted questions and identify the variable with respect to which each of those had to be integrated.
integWRT[input_String]:=DeleteDuplicates[ Cases[ WolframAlpha[input,"WolframParse"], a_Symbol/;!NumericQ[a],Infinity]]
wrt=integWRT/@quesList
Out[]=
{{t},{x},{t}}
These four aforementioned functions were called several times and across all the categories and with the varying difficulty levels and over 4000 integrals were obtained using this method
Through a Coded Random Function Generator
Through a Coded Random Function Generator
The second method used for the collection of data was through the use of a coded Random Function generator. This generator was designed by my mentor Peter Barendse in the past and was used to generate functions of varying depths. Since this function returned only functions, a separate function was used to integrate these functions and store the results.
The code below shows the working of the Random Function Generator that was used:
Options[MathTestFunc]="Wrapper"Automatic,"Vars"Automatic,"DepVars"Automatic,"SingleDepVar"Automatic, "Threaded"Automatic,"WithArgsInDepVars"Automatic, "ReverseAcrossEquals"Automatic; Clear@wrapandstring wrapandstring[a_,_]:=a MathTestFunc[dimin_,dimout_,nestlevel_,opt:OptionsPattern[]]:=Module[ {vars,depvars,singledepvarQ=False,temp}, (*choose vars*)vars=If[ OptionValue["Vars"]===Automatic, RandomSample[variableslib,dimin], OptionValue["Vars"]]; (*choose depvars*)depvars=Switch[ OptionValue["DepVars"],False,{},Automatic, If[ TrueQ@OptionValue["SingleDepVar"]|| (OptionValue["SingleDepVar"]===Automatic&&RandomInteger[]0), singledepvarQ=True;#[[1]],#]&@RandomSample[Complement[variableslib,vars], dimout],_,OptionValue["DepVars"]]; If[TrueQ@OptionValue["WithArgsInDepVars"]|| (OptionValue["WithArgsInDepVars"]===Automatic&&RandomInteger[]0), depvars=If[ singledepvarQ, depvars@@vars, (#@@vars&)/@depvars]]; (*recursively construct random functions*)temp=Table[ MathTestFunc[vars,nestlevel], {dimout}]; (*make dependent part*)If[TrueQ[depvars=!={}], temp=If[ TrueQ@OptionValue["ReverseAcrossEquals"]|| OptionValue["ReverseAcrossEquals"]=== Automatic&&RandomInteger[4]0, tempdepvars, depvarstemp]]; (*wrap in parens/curlybrackets and make into a string*)If[ !singledepvarQ&&TrueQ@OptionValue["Threaded"]|| (OptionValue["Threaded"]===Automatic&&RandomInteger[]0), Return[wrapandstring[Thread[temp],OptionValue["Wrapper"]]], Return[wrapandstring[temp,OptionValue["Wrapper"]]]]]libUnary={Tan,Sin,Cos,Log,Sinh,Cosh,Tanh,E^#&};libBinary={Plus,Times,Power};numberslib:={RandomReal[],RandomInteger[1000],-2,-1000,Pi,E,1.2,-2.0};variableslib=Symbol/@{"x","y","z","u","v","w","r","s","t","p","q","m","n"};MathTestFunc[vars_List,1]:=With[ a=If[RandomInteger[4]>0,RandomChoice[vars], RandomChoice[numberslib]], a];MathTestFunc[vars_List,n_Integer]:=With[ {f=RandomChoice[Join[libBinary,libUnary]]}, If[MemberQ[libBinary,f], f@@MathTestFunc[vars,n-1], MathTestFunc[vars,n-1], f@MathTestFunc[vars,n-1]]];
An example of a random depth 3 function generated through the use of this function has been shown below:
The next thing to do was to create functions that could repeatedly call the random function generator and generate lists of values that were in the form of
question->answer rules. As we reached functions of higher depths, there were often also functions that Mathematica failed to integrate or took too long (longer than 4 seconds) to integrate. Those were filtered out and stored in a separate list.
The function below shows how this was done to generate functions with a depth of 3. The same process was repeated to generate functions with a depth of 1-5.
question->answer rules. As we reached functions of higher depths, there were often also functions that Mathematica failed to integrate or took too long (longer than 4 seconds) to integrate. Those were filtered out and stored in a separate list.
The function below shows how this was done to generate functions with a depth of 3. The same process was repeated to generate functions with a depth of 1-5.
The output above shows the form of output that was obtained from this function. Based on the values present, these could be filtered out into separate lists. The function below shows how this was done for functions with a depth of 3.
The values obtained can be seen by running the code cell below
Similar functions were used to obtain and filter data of functions of depths 1-5.
Quantitative Symbolic Vectorization
Quantitative Symbolic Vectorization
After curating all the data using the two methods explained above, the next step was to find a way to convert the integrands (questions) and integrals (answers) into vectors that contain numbers. This is because it would be easier to train a Neural Network using such numerical vectors.
The vectorization method I used involved an analysis of the integration answers for many of the problems and based on the results seen, I curated a set of 84 different Wolfram Functions that could show up in the answers (note: it is possible that some integrals give us other functions that are not in this list but were just missed out because they were not obtained during our random function generation and Wolfram|Alpha problem generator data collection methods). One of the functions that I added myself is called powerExpVar which signifies an exponential function in which the variable is present in the exponent (and not in the base).
Extraction of Mathematical Functions from Expressions
Extraction of Mathematical Functions from Expressions
When we run this function, we also separate the Associations for the questions (integrands) and answers because separate operations will be conducted on these later.
A summary of the vectorization algorithm above is that it identifies the different kinds of Mathematical Functions present in a given expression and the number of times that they occur. These are then represented in the form of an association.
One might notice that all different numeric values appear separately, however, when these associations are converted to numerical vectors in the next step, that will be taken care of.
One might notice that all different numeric values appear separately, however, when these associations are converted to numerical vectors in the next step, that will be taken care of.
Converting association of extracted functions into numerical vectors
Converting association of extracted functions into numerical vectors
The first step was to create a list called funcsList that contained a set of all the 84 functions that were defined in our function vectorize in the previous section.
The next step is to convert the associations of extracted functions into numerical vectors that can be sent to the neural network. In order to do this, we create a vector of length 84 with all its values starting at 0. Then, for each association that we want to convert to a numeric vector, we take each function present in the association and find its position pos in funcList. Now, if we are quantifying a question (integrand), we replace the value at the position pos in our quantified vector with the number of times the function was found in the original question. On the other hand, if we are quantifying an answer, we replace the value of the position pos in our quantified vector with the number 1.
This different process being applied to the answer associations might seem strange but this has been explained in a later part of this essay.
This different process being applied to the answer associations might seem strange but this has been explained in a later part of this essay.
We will first generate the Numerical vectors for depth 3 questions and then for depth 3 answers. We will store these in dep3QuesNumer and dep3AnsNumer respectively.
This process was repeated with thousands of questions and answers of all 5 depths to obtain a large dataset.
Since we now have vectors with purely numerical data that represent both our functions and our answers, our next step is to move onto visualizing this data after which we can start with the Machine Learning.
Data Visualization
Data Visualization
My hypothesis was that the uniformity of the transformation that would need to be applied to an integrand of a given depth to obtain the answer post integration decreases with an increase in the depth of the integrand. In order to test this hypothesis, I used the FeatureSpacePlot function to generate separate plots for each of the 5 depths. Each point was a pair of an integrand of the specified depth and its corresponding integral.
As can be seen from the plots above, the close grouping of depth 1 data suggests that the transformations required to integrate depth 1 functions are relatively similar. In contrast, the spaced-out and seemingly more random nature of the depth 5 data suggests that as we increase the depths, the transformations required for integrating become less uniform, which confirms my hypothesis.
Machine Learning
Machine Learning
Now that we have converted all of our symbolic mathematical data into numerical vectors, we must convert our data back into a format that is structured in the form of a list of question->answer
After repeating this procedure for all of our data for all depths, we get our training data for the neural network.
Next, we define the structure of a basic neural network:
We divide all of our data into completeTrainingData, which contains approximately 80% of the data, and completeTestData, which contains the remaining 20% of the data. Due to the large sizes of these datasets, training time was taking extremely long, so for the purpose of this notebook, only approximately 9000 examples of the training data were used to train this model. The results of the same have been depicted in the output below.
I now ran a small set of code to get an estimate of the accuracy of this neural network. It is worth noting that the majority of the data used for calculating the accuracy was from depths of 1-3. The accuracy for functions of depths 4 and 5 was much lower.
While the Neural Network depicted above is the one I will use to obtain an estimate of the accuracy of the Neural Network , there were a few simpler networks that I trained using smaller portions of the data (before all the data had been collected). The most notable results from those intermediate models were:
◼
Integrals with a depth of 1 could be predicted with an accuracy of approximately 0.987 (98.7%).
◼
Integrals with a depth of 2 could be predicted with an accuracy of approximately 0.945 (94.5%).
The accuracies were much lower for higher depths. This does give us an idea about the effectiveness of the vectorization method I used. These will be discussed later in the essay.
The predictions received from the neural network were in the form of 84-length vectors, so I also had to design a function to convert that vector back to an answer:
Creating a pipeline for the user
Creating a pipeline for the user
Given that many of the functions defined in this essay are for the purpose of vectorization and other processes that are not actively used while predicting the functions present in the answer to an integral, I decided to create a basic pipeline with replicas of the functions used in prediction (I also changed the names of the functions slightly from the names used earlier in this essay):
The function that the user will use:
Some practical examples
Some practical examples
I decided to run my function predIntegral for a few integrals and to compare my results with the results returned by the Integrate function provided by the Wolfram|Language.
NoteL powerExpVar is an addition class I added which indicates an exponential function with a variable in the power (for example: e^x).
Conclusions
Conclusions
There are several conclusions that can be drawn from the work done in this project:
◼
There do seem to be several merits behind the vectorization of the symbolic mathematical notation since neural networks struggle to work well with mathematical notation without doing so.
◼
Many of the predictions made by the model seem to be very accurate representations of the actual answers obtained from the Wolfram Integrate function.
◼
The decrease in the uniformity of the transformations applied to an integrand to get to the answer with an increase in depth is very apparent.
◼
One drawback that does seem to appear from the vectorization method used is that it does not preserve the order of operations of the original integrand, so my function would output the same values for integrals of both Sin[Log[x]] and Log[Sin[x]].
In general, the work done also seems to have a lot of potential applications in the real world. For example, physicists could use a function like this to predict whether or not an integral can be calculated. Also, the ability to obtain estimates on the kinds of functions present upon integrating complicated integrals would also be beneficial to mathematicians and anybody else who makes use of integral calculus regularly.
That being said, there is a lot of scope for future work with this project:
That being said, there is a lot of scope for future work with this project:
◼
I intend to spend more time to design a more intricate Neural Network that can achieve an even higher accuracy than the current model.
◼
I also intend to attempt an implementation of a different vectorization technique that preserves the order of operations of the integral and tries to reconstruct an entire output based on that.
◼
Another potential addition would be an attempt to correctly predict the depth of the answer and the number of times each function occurs. This will help since it will enable one to run through all permutations of the functions outputted and to then calculate the derivatives of all the permutations to find the one that is closest to the original function.
Hopefully, this method of quantitative vectorization which enables neural networks to train on mathematical data will inspire others to come up with other, more advanced vectorization methods that will boost our abilities in predictive mathematics.
Keywords
Keywords
◼
Machine Learning
◼
Calculus
◼
Mathematics
◼
Vectorization
◼
Symbolic Mathematical Notation
◼
Integration
Acknowledgements
Acknowledgements
Thank you to my mentor, Peter Barendse, for all the help and guidance. Also, a big thank you to Stephen Wolfram and everyone else associated with the Summer Camp since this work would not have been possible without your support.