Keywords are vital component of programming languages . They are predefined words that have special functions when fed to the compiler . By identifying the keywords of a snippet of code, we can get a good feel of the style of the code written. This project aims to be able to identify keywords from a snippet of code, given the language the code is written in. Data and Examples in the project are sourced from RosettaCode.org
Introduction
Introduction
Since the start of computer programming, keywords have been important to the functionality of a language. We may not notice it, but the role they play significantly improves our quality of life while programming. By shortening entire commands to mere words, programmers can use these functions efficiently and hassle free. Keywords are in almost every programming languages, but each programming language has its own set of syntax for these keywords. Although different, many languages have an overlap with their syntax, allowing the same few words to stretch across a sea of languages. The identification of keywords is very tricky, as when a code snippet is presented in string form, there is very little to distinguish keywords from other normal words. Though difficult, this project uses filters, blacklists, and frequency to effectively distinguish keywords from their normal counterparts.
Stripping Comments and Strings
Stripping Comments and Strings
To obtain the keywords in a snippet of code, we must first remove any comments and strings that may contain words that could be mistaken for keywords.
We must separate the comments into two types of comments: EOL(End of Line) Comments and Wrapper Comment.
We must separate the comments into two types of comments: EOL(End of Line) Comments and Wrapper Comment.
End Of Line Comments
End Of Line Comments
EOL comments are comments that begin from the comment start syntax and end at the end of the line. By nature Wolfram does not use EOL comments, but many other popular languages such as Java, Python, and C++ all utilize this commenting syntax.
Wrapper Comments
Wrapper Comments
Wrapper comments are comments that begin from the comment start syntax but instead end at the comment end syntax. Wolfram uses this syntax of commenting. Because programming languages can have either one or both of the syntaxes mentioned above, we must account for both. Our syntax will be sourced from http://rigaux.org/language-study/syntax-across-languages.html
I first pulled the syntax of each programming language for both Wrapper and EOL commenting and placed it in an association. This way when the language of the code snippet is entered, the function will be able to search for the right type of syntax.
I first pulled the syntax of each programming language for both Wrapper and EOL commenting and placed it in an association. This way when the language of the code snippet is entered, the function will be able to search for the right type of syntax.
In[]:=
wrappersyntax=<|"Beta"->"(**)","F#"->"(**)","Mathematica"->"(**)","Modula"->"(**)","OCaml"->"(**)","Pascal"->"(**)","SML"->"(**)","Matlab"->"%(%)","REXX"->"/**/","Dylan"->"/**/","Io"->"/**/","Oz"->"/**/","SQL99"->"/**/","Pascal"->"{}","Rebol"->"{}","Haskell"->"{--}","Lisp"->"#||#","Scheme"->"#||#","Julia"->"#==#","Perl6"->"#[]","Lua"->"--[[]]","Smalltalk"->"\"\"","B"->"/**/","C"->"/**/","C#"->"/**/","C++"->"/**/","CSS"->"/**/","GNU"->"/**/","Go"->"/**/","Java"->"/**/","JavaScript"->"/**/","Mercury"->"/**/","PHP"->"/**/","Pike"->"/**/","PL/I"->"/**/","YCP"->"/**/","Yorick"->"/**/","XML"->"<!---->","HTML"->"<!---->","Forth"->"()","CoffeeScript"->"###\n###\n"|>;singleLineSyntax=<|"Awk"->"#","BourneShell"->"#","CoffeeScript"->"#","E"->"#","FishShell"->"#","GNU"->"#","Icon"->"#","Io"->"#","Julia"->"#","Maple"->"#","merd"->"#","Perl6"->"#","PHP"->"#","Perl"->"#","PHP"->"#","Pliant"->"#","Python"->"#","Ruby"->"#","Tcl"->"#","YAML"->"#","BCPL"->"//","C#"->"//","C++"->"//","C99"->"//","Dylan"->"//","F#"->"//","Go"->"//","Io"->"//","Java"->"//","JavaScript"->"//","PHP"->"//","Pike"->"//","Scilab"->"//","YCP"->"//","Yorick"->"//","PHP"->"//","PHP"->"//","Ada"->"--","Cecil"->"--","Eiffel"->"--","Haskell"->"--","Lua"->"--","Sather"->"--","Simula"->"--","SQL92"->"--","Assembler"->";","Lisp"->";","Logo"->";","MUMPS"->";","Rebol"->";","Scheme"->";","Erlang"->"%","Matlab"->"%","Mercury"->"%","Oz"->"%","Postscript"->"%","Prolog"->"%","Tex"->"%","Basic"->"rem","Visual Basic"->"'","Vimscript"->"\"","Forth"->"\\","Assembler"->"!","J"->"NB."|>;
As you can see, since the two types of syntax function differently, we need two different associations. This way when we are checking for comments in the code snippet later, we will be able to effectively check for both types of syntax.
Strings
Strings
Additionally, I pulled the syntax for strings as well from each programming language and put it in a different association.
In[]:=
stringsyntax=<|"Beta"->"''","BourneShell"->"''","CoffeeScript"->"''","CSS"->"''","JavaScript"->"''","Lua"->"''","Matlab"->"''","Pascal"->"''","Perl"->"''","Perl6"->"''","PHP"->"''","FishShell"->"''","Python"->"''","Ruby"->"''","Smalltalk"->"''","Vimscript"->"''","YAML"->"''","Awk"->"\"\"","C"->"\"\"","C#"->"\"\"","C++"->"\"\"","REXX"->"\"\"","Lisp"->"\"\"","CSS"->"\"\"","Dylan"->"\"\"","E"->"\"\"","Eiffel"->"\"\"","F#"->"\"\"","Go"->"\"\"","Haskell"->"\"\"","Io"->"\"\"","Java"->"\"\"","JavaScript"->"\"\"","Lua"->"\"\"","Maple"->"\"\"","Mathematica"->"\"\"","Modula"->"\"\"","Ada"->"\"\"","SQL92"->"''","MUMPS"->"\"\"","OCaml"->"\"\"","Oz"->"\"\"","Pike"->"\"\"","Python"->"\"\"","Rebol"->"\"\"","Scheme"->"\"\"","SML"->"\"\"","YAML"->"\"\"","YCP"->"\"\"","Awk"->"\"\""|>;
Now, we can utilize these associations to determine the syntax for strings and comments of each language so we can search the snippet of code for these and remove them.
Before we do that though, we need to split the wrapper comment syntax into two parts, one for the start of the comment and one for the end.
Before we do that though, we need to split the wrapper comment syntax into two parts, one for the start of the comment and one for the end.
Syntax Splitting
Syntax Splitting
If we have this syntax split in two, it allows us to search for the beginning of the comment and the end of the comment and then remove everything in between. The same applies for strings. We can create a second association to associate the syntax to the index where the string containing the syntax needs to be split.
In[]:=
startindexoflanguage=<|"(**)"->{1,2},"%(%)"->{1,2},"/**/"->{1,2},"{}"->{1,1},"{--}"->{1,2},"#||#"->{1,2},"#==#"->{1,2},"#[]"->{1,2},"--[[]]"->{1,4},"\"\""->{1,2},"/**/"->{1,2},"<!---->"->{1,4},"()"->{1,1},"###\n###\n"->{1,5}|>;endindexoflanguage=<|"(**)"->{3,4},"%(%)"->{3,4},"/**/"->{3,4},"{}"->{2,2},"{--}"->{3,4},"#||#"->{3,4},"#==#"->{3,4},"#[]"->{3,3},"--[[]]"->{5,6},"\"\""->{3,4},"/**/"->{3,4},"<!---->"->{5,6},"()"->{2,2},"###\n###\n"->{6,10}|>;stringstartindexsplitter=<|"\"\""->{1,1},"''"->{1,1}|>;stringendindexsplitter=<|"\"\""->{2,2},"''"->{2,2}|>;
After we have created all of our second associations. We can start constructing our functions.
Function Construction
Function Construction
We will need 3 different functions to accomplish this task, one for EOL, one for Wrapper, and one for Strings. Our functions will take an input of a snippet of code and a key, which will take the form of a string. It will output the code void of whatever it was created to remove.
Wrapper Function
Wrapper Function
For our wrapper function, we will first check if the code needs syntax removed. If it does, then it will move past the initial if statement and split the syntax of the wrapper function into two parts, the start and the end syntax. Then it will use StringDelete to search through the string and delete anything that starts with a start syntax AND ends with an end syntax.
In[]:=
removewrappercomments[code_String,language_String]:=If[MemberQ[Keys[wrappersyntax],language],With[{startsyntax=StringTake[wrappersyntax[language],startindexoflanguage[wrappersyntax[language]]],endsyntax=StringTake[wrappersyntax[language],endindexoflanguage[wrappersyntax[language]]]},StringDelete[code,startsyntax~~Except[Alternatives@@Characters[endsyntax]]..~~endsyntax]],code];
In[]:=
removewrappercomments["static public double noise(double x, double y, double z) { int X = (int)Math.floor(x) & 255, /* FIND UNIT CUBE THAT*/ Y = (int)Math.floor(y) & 255, /* CONTAINS POINT.*/ Z = (int)Math.floor(z) & 255;","Java"]
Out[]=
static public double noise(double x, double y, double z) { int X = (int)Math.floor(x) & 255, Y = (int)Math.floor(y) & 255, Z = (int)Math.floor(z) & 255;
End of Line Function
End of Line Function
For our EOL function, it will follow the same approach of initially checking if the code needs syntax removed. If it does however, instead of splitting the syntax of the function, we can just directly take the syntax and input it because there is no end syntax, as it ends with EOL ( /n). We then use RegularExpression to find expressions that begin with the syntax of the language and end with the new line operator. Lastly we will use StringReplace to replace the expression found with RegularExpression with just a new line.
In[]:=
removesinglelinecomments[code_String,language_String]:=If[MemberQ[Keys[singleLineSyntax],language],StringReplace[code,RegularExpression [singleLineSyntax[language]<>"[^\n]+\n"]->"\n"],code];
Here is an example of the code working with a snippet of the Perlin noise Java snippet:
In[]:=
removesinglelinecomments[" static public double noise(double x, double y, double z) { int X = (int)Math.floor(x) & 255, // FIND UNIT CUBE THAT Y = (int)Math.floor(y) & 255, // CONTAINS POINT. Z = (int)Math.floor(z) & 255;","Java"]
Out[]=
static public double noise(double x, double y, double z) { int X = (int)Math.floor(x) & 255, Y = (int)Math.floor(y) & 255, Z = (int)Math.floor(z) & 255;
String Function
String Function
For our String function, it will follow the same approach as the wrapper function, splitting the String syntax into the beginning and ending String syntax. Then using StringDelete it will search through the code and delete anything that starts with a start syntax AND ends with an end syntax.
In[]:=
removestrings[code_String,language_String]:=If[MemberQ[Keys[stringsyntax],language],With[{startsyntax=StringTake[stringsyntax[language],stringstartindexsplitter[stringsyntax[language]]],endsyntax=StringTake[stringsyntax[language],stringendindexsplitter[stringsyntax[language]]]},StringDelete[code,startsyntax~~Except[Alternatives@@Characters[endsyntax]]..~~endsyntax]],code]
Here is an example of the code working using a small part of the Perlin noise Python snippet:
In[]:=
removestrings["if __name__ == '__main__': print(\"%1.17f\" % perlin_noise(3.14, 42, 7))","Python"]
With these 3 functions completed, we are now able to clean our code snippets of any excess text that may not contain keywords.
More Tidying
More Tidying
Once our code has been cleaned of unnecessary text, we need to remove all symbols and strip the snippet down to its simplest form: Words.
We can do this through the StringSplit function, which helps us split the string into a list, splitting where the symbols are. Then we use DeleteCases to remove all the white spaces leftover. This leaves us with a list of all the words.
We can do this through the StringSplit function, which helps us split the string into a list, splitting where the symbols are. Then we use DeleteCases to remove all the white spaces leftover. This leaves us with a list of all the words.
Putting it All Together
Putting it All Together
Now that we've defined all of the individual functions that go into the overall function, we can put it all together. Essentially what is happening is the code snippet is being spit out through functions, put into a variable, then that variable is put into the next function and the cycle repeats.
The Algorithm
The Algorithm
Now, for the algorithm determining keywords, there was only one way to do it: Frequency. This wasn't the method I really wanted to use as there are many flaws with using the frequency method, but given the scope of this project, this was the only option. We can also increase the accuracy of the frequency method by blacklisting certain words.
Below is a list of blacklisted word types:
◼
Numbers
◼
Words < 2 Characters
◼
Words > 10 Characters
◼
Words that are 2 characters and double capitalized
◼
Words not listed in the Wolfram Dictionary
Now that we have completed our function, we can test it with some snippets of various languages.
Results
Results
For our testing, we will be using the perlinNoise code snippets obtained from Rosettacode.org
Here I have processed the imported string that was obtained from the website.
The list of supported languages of this program can be generated by using the Union function on each of the syntax associations.
Here we have have called Union on the keys(languages in this case) of single and double line syntax:
We can then use this to see which languages from the website are usable with our function
Here we are taking the common Keys in the supported languages and the Perlin noise snippets to see which languages we can use for this specific snippet of code:
Lastly we can feed this set of languages to keyword finder and see how our results panned out.
Here we are printing out the number of each word for a single language:
Here is our final list of keywords in order of frequency:
We can also create WordClouds out of the results, with the largest words being the ones that were used the most. We can see that out of those largest words, in all 3 languages, the majority of them are keywords of each respective language.
Conclusion
Conclusion
Although frequency may not be the most optimal method of identifying keywords, with some tweaks and modifications, it can serve as a good estimator of the keywords in a certain snippet of code. Another major trend I've noticed with using frequency is the overall lumping of keywords with normal words, and subsequently, the increased difficulty in discerning keywords from normal words. This leads to the tradeoff of sacrificing a couple keywords for decreased amounts of clutter, or inversely, increased amounts of clutter for a couple more keywords.
Constraints
Constraints
Due to the time constraints I had, I was unable to create a classifier for each programming language as it would have required more knowledge about pattern structures. Time was a major constraint as it held me back from applying the highest level of complexity for this algorithm.
Further Study
Further Study
In the future, I would like to implement a weighing system that weights the first x amount of words in the code snippet heavier as keywords because generally many keywords happen only once or twice and they happen at the start of the snippet for class/variable/object declaration. Additionally, I would like to potentially create a white list of words that may be unintentionally blacklisted.
References
References
Perlin Noise. Perlin noise - Rosetta Code. (n.d.). Retrieved July 21, 2022, from http://www.rosettacode.org/wiki/Perlin_noise
Syntax across languages. (n.d.). Retrieved July 21, 2022, from http://rigaux.org/language-study/syntax-across-languages.html
Big thanks to Isabel for helping me with my project!