Abstract
In this project I attempt to better understand the rate of change of languages over time by assigning an estimated volatility score to each language through a series of functions and comparing these languages according to various metrics to see if we can find reason for their differing scores.
Introduction
I wanted to computationally classify languages by volatility and use this to better understand what makes languages more or less volatile. Being able to classify languages by their volatility allows us to better understand the underlying moving forces of natural language.
Initial Observations
At the beginning of this project, I went through a different words manually using TimeSeries of WordFrequencyData and saw noticeable changes throughout the 300 years of data. However, I noticed that older words have much lower word frequency values even when they were at the height of their use. (Note: I use MovingAverage[] on all graphs in this presentation to make data more fluid. This might mean that interpretations of data can be inaccurate at times.)
Graphs (“cat” vs. “catt”)
We see that the “death” of the Middle English noun “catt” corresponds with the rise of the Modern English noun “cat”. We also see that “catt” was used about 25x less than “cat” even at its height of usage. After further research, I found a number of reasons for why this could be the case. Most notably, unconventional spelling in English. Conventional spelling only started to be implemented in English in the 1500’s, and even then it generally only applied to a small minority of the population who both could read and speak Latin. These spellings also often tweaked English words slightly to sound more like Latin. An example is, the spelling of “Falcon” instead of the then used “Faucon”. Today, we use the word “Falcon” and pronounce it with the “L”, but many of these changes were unprecedented and confusing at the time. This lead them to being largely disregarded even into the late 1700’s and early 1800’s. For this reason, it’s very likely that there were many different spellings and ways of saying “cat” in the 1700’s, which would explain what appears to be a period of time where people just didn’t write about cats that much. Another explanation could just be that people didn’t write about cats much in the 1700’s. I did notice this pattern in most words I compared, so it’s an important thing to keep in mind. Taking this into account and other logistical reasons, I decided to not implement old versions of words in any language for my investigation.
Another thing to keep in mind is WordFrequencyData’s way of displaying time data. For words with constant data from 1701-2008, the time data is expressed as six values that instruct the function to create its own internal list with Unix Times when it needs to be graphed. Initially, I used UnixTime[] to find these values (as of this writing UnixTime[] is buggy), but I then found that I can fetch the data using [“Times”]. Keep this in mind if you plan on using WordFrequencyData.
Another thing to keep in mind is WordFrequencyData’s way of displaying time data. For words with constant data from 1701-2008, the time data is expressed as six values that instruct the function to create its own internal list with Unix Times when it needs to be graphed. Initially, I used UnixTime[] to find these values (as of this writing UnixTime[] is buggy), but I then found that I can fetch the data using [“Times”]. Keep this in mind if you plan on using WordFrequencyData.
Choosing Word Lists
Choosing words to check volatility score is an important job, because I want a large volume of words to test, and to not bias particular languages. For this reason, I attempted to avoid cultural or product related words. I do recognize that having many different words makes the chance of bias decrease. I chose 151 words that I saw as somewhat constant to the human experience. Of course, my choices are subjective as to what I think are constant to the human experience, so I do not claim to say that these words are the ideal choice. I sorted these words into groups by what context I think it might be used. For example, although the word “war” is technically an abstract concept that doesn’t represent a particular object, I did not put it in the “abstract conceptual” group. Instead I put it in the “human experience” group, because I thought it might be used more often alongside references to civilization, than concepts like happiness and time. In a future iteration I would like to find a way to group words computationally, instead of by assumption. Below is my list of words and their groups.
DISCLAIMER: You can’t choose to look at the difference between particular sorted groups. You can select a particular word group to use for `wordlist` and then run the implemented function.
DISCLAIMER: You can’t choose to look at the difference between particular sorted groups. You can select a particular word group to use for `wordlist` and then run the implemented function.
Word Lists
Calculating Volatility
First thing’s first, getting the word data. This function gets the data from WordFrequencyData and then we can assign it to a variable.
In[]:=
getworddata[word_,language_]:=WordFrequencyData[word,"TimeSeries",IgnoreCaseFalse,Languagelanguage]
In[]:=
worddata=getworddata["gauss"];
Now I take the list of times and corresponding frequencies, and calculate the moving average for the frequencies and times to make the graph more fluid.
In[]:=
times=worddata["Times"];
In[]:=
freqs=worddata["Values"];
getfulldata[worddata_,maval_]:=MovingAverage[worddata,maval];timesMAcalc[times_,maval_]:=MovingAverage[times,maval];freqMAcalc[freqs_,maval_]:=MovingAverage[freqs,maval];
In[]:=
timesMA=timesMAcalc[times,29];
In[]:=
freqsMA=freqMAcalc[freqs,29];
Now we start on getting the Practical Derivative.
finalcalc[] will format our data in a pretty way that allows us to take the derivatives for time and frequency.
The output will be {{time_1, freq_1}, {time_2, freq_2}, ......}
Dividem[] is a simple slope-finding function. It takes two points and finds the slope between them. Each slope that Dividem[] outputs will be a point in our final list of derivative values.
DiviX[] and the functions following it will select the correct value for each position in the Dividem[] function from final.
Then fullfunction[] will implement Dividem[] and the position functions so that it can be used on our data values.
Runthrough[] will run fullfunction[] on the entire list, and output our final list of derivatives.
finalcalc[] will format our data in a pretty way that allows us to take the derivatives for time and frequency.
The output will be {{time_1, freq_1}, {time_2, freq_2}, ......}
Dividem[] is a simple slope-finding function. It takes two points and finds the slope between them. Each slope that Dividem[] outputs will be a point in our final list of derivative values.
DiviX[] and the functions following it will select the correct value for each position in the Dividem[] function from final.
Then fullfunction[] will implement Dividem[] and the position functions so that it can be used on our data values.
Runthrough[] will run fullfunction[] on the entire list, and output our final list of derivatives.
In[]:=
finalcalc[timesMA_,freqsMA_]:=Transpose[{timesMA,freqsMA}];Dividem[x_,x1_,y_,y1_]:=(y1-y)/(x1-x)DiviX[num_,final_]:=final[[num,1]]DiviX1[num_,final_]:=final[[num+1,1]]DiviY[num_,final_]:=final[[num,2]]DiviY1[num_,final_]:=final[[num+1,2]]fullfunction[index_,final_]:=Dividem[DiviX[index,final],DiviX1[index,final],DiviY[index,final],DiviY1[index,final]]
In[]:=
final=finalcalc[timesMA,freqsMA];
In[]:=
Runthrough[timesMA_,final_]:=Map[fullfunction[#,final]&,Drop[Range[Length[timesMA]],-1]]
In[]:=
Runthrough[timesMA,final];
Since we now have our derivative values, we can start to analyze what they mean. This part of the code will check for zeroes in the derivative.
It will do this by assigning each derivative value in the list into its own list with its neighbor. I.E.: {a,b,c,d,e...} will have new neighbors list {{a,b},{b,c},{c,d},{d,e}.....}.
Then CheckMinMax[] will check to see the sign of each member of each list. This relies on the Intermediate Value Theorem, which states that if the signs of two points on a continuous function are different, there is a zero value between them. All lists where both elements are positive or both negative will be recorded as “nan”. All lists where the first value is positive and the second is negative are recorded as “1”, which signifies a maximum in the original data at that point, and all lists where the opposite is true are recorded as “-1” which signifies a minimum at that point.
CheckALL[] will then map CheckMinMax[] to every sublist and output a list we can use to see where the minimums and maximums are.
It will do this by assigning each derivative value in the list into its own list with its neighbor. I.E.: {a,b,c,d,e...} will have new neighbors list {{a,b},{b,c},{c,d},{d,e}.....}.
Then CheckMinMax[] will check to see the sign of each member of each list. This relies on the Intermediate Value Theorem, which states that if the signs of two points on a continuous function are different, there is a zero value between them. All lists where both elements are positive or both negative will be recorded as “nan”. All lists where the first value is positive and the second is negative are recorded as “1”, which signifies a maximum in the original data at that point, and all lists where the opposite is true are recorded as “-1” which signifies a minimum at that point.
CheckALL[] will then map CheckMinMax[] to every sublist and output a list we can use to see where the minimums and maximums are.
In[]:=
CompDerivData[timesMA_,maDERval_,final_]:=MovingAverage[Transpose[{Drop[timesMA,-1],Runthrough[timesMA,final]}],maDERval]Divi1Y[Derivdata_,num_]:=Derivdata[[num,2]]Divi1Y1[Derivdata_,num_]:=Derivdata[[(num+1),2]]
In[]:=
derivdata=CompDerivData[timesMA,60,final];
In[]:=
NegPosLstF0[Derivdata_]:=Map[Divi1Y[Derivdata,#]&,Range[Length[Derivdata]-1]]NegPosLstF1[Derivdata_]:=Map[Divi1Y1[Derivdata,#]&,Range[Length[Derivdata]-1]]getderivcompare[Derivdata_]:=Transpose[{Flatten[NegPosLstF0[Derivdata]],Flatten[NegPosLstF1[Derivdata]]}]
In[]:=
derivcompare=getderivcompare[derivdata];
In[]:=
CheckMinMax[derivcompare_,x_]:=Module[{ListCheck={}},Which[Or[And[Sign[derivcompare[[x,1]]]1,Sign[derivcompare[[x,2]]]1],And[Sign[derivcompare[[x,1]]]-1,Sign[derivcompare[[x,2]]]-1],And[Sign[derivcompare[[x,1]]]0,Sign[derivcompare[[x,2]]]0]],Append[ListCheck,"nan"],And[Sign[derivcompare[[x,1]]]-1,Sign[derivcompare[[x,2]]]1],Append[ListCheck,-1],And[Sign[derivcompare[[x,1]]]1,Sign[derivcompare[[x,2]]]-1],Append[ListCheck,1]]]
In[]:=
CheckALL[derivcompare_]:=Flatten[Map[CheckMinMax[derivcompare,#]&,Range[Length[derivcompare]]]]
In[]:=
checkall=CheckALL[derivcompare]
Out[]=
{nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,1,nan,nan,nan,nan,nan,nan,nan,nan}
Now that we have the locations of all the zeroes, we can use them to find out where our function is increasing and decreasing. Unfortunately at the moment our function does not take into account the difference between decreases and increases or minimums and maximums when it is calculating volatility and just counts them both as “trend changes that indicate volatility”. However, there is a small way this feature is implemented (even though it can’t be used in the calculation) that you will see later. Hopefully, at some point these can be considered more heavily when calculating volatility.
getallzeroes[] grabs the location of all maximums and minimums and puts them in a list. Zero is always at the beginning, and the Length[] of the derivdata is always the end value. This will allow us to choose lists to take the means of in the future, by indicating that we want the list to be separated into groups {{0, zero_1},{zero_1, zero2}, ... {zero_i, Length[derivdata]}}. We do this in getydata.
Now that we have our separated lists, we can separate lists further by checking for clusters in the data with FindClusters[].
Finally, we get the size of the clusters so we know the intervals of all the groups. This is necessary in the case that FindClusters[] did split the data into more parts after finding zeroes.
getallzeroes[] grabs the location of all maximums and minimums and puts them in a list. Zero is always at the beginning, and the Length[] of the derivdata is always the end value. This will allow us to choose lists to take the means of in the future, by indicating that we want the list to be separated into groups {{0, zero_1},{zero_1, zero2}, ... {zero_i, Length[derivdata]}}. We do this in getydata.
Now that we have our separated lists, we can separate lists further by checking for clusters in the data with FindClusters[].
Finally, we get the size of the clusters so we know the intervals of all the groups. This is necessary in the case that FindClusters[] did split the data into more parts after finding zeroes.
In[]:=
getallzeroes[derivdata_,checkall_]:=Join[{0},Sort[Join[Flatten[Position[checkall,1]],Flatten[Position[checkall,-1]]]],{Length[derivdata[[All,2]]]}];
In[]:=
allzeroes=getallzeroes[derivdata,checkall]
Out[]=
{0,38,47}
In[]:=
getydata[derivdata_,allzeroes_]:=TakeList[derivdata[[All,2]],Differences[allzeroes]];
In[]:=
ydata=getydata[derivdata,allzeroes];
In[]:=
MeanShiftFindPath[ydata_,x_]:=MeanShiftFilter[x,IntegerPart[Length@ydata/10],MedianDeviation@x,MaxIterations10]FindMeans[ydata_]:=MeanShiftFindPath[ydata,#]&/@ydata
In[]:=
FC[x_]:=Flatten[FindClusters[x]];FindCllusters[ydata_]:=FC[#]&/@FindMeans[ydata];getsizeclusters[clusters_]:=Accumulate[Length[clusters[[#]]]&/@Range[Length[clusters]]];
In[]:=
clusters=FindCllusters[ydata];
In[]:=
sizeclusters=getsizeclusters[clusters]
Out[]=
{38,47}
Now we take the mean of all the values in each cluster. This will allow us to find where functions are increasing, decreasing, or dead.
A “dead” interval is considered to be any interval which is less than or equal to wordDeathVal times the mean of all other means. A small issue is that the mean of all other means does not consider the sign of the other means. So the calculated value if the other two means were would be 0. This is an error because the intention is to detect on average how far from zero the other means are (absolute value). This is not a huge error, because dead intervals are very uncommon as compared to decreasing or increasing intervals anyway, but should be taken into account.
An increasing interval is any interval which is not dead and the mean is positive while a decreasing interval is any interval which is not dead and its mean is negative.
Intervals with means of zero are not taken into account because they would be dead by definition.
A “dead” interval is considered to be any interval which is less than or equal to wordDeathVal times the mean of all other means. A small issue is that the mean of all other means does not consider the sign of the other means. So the calculated value if the other two means were
1.73179×and-1.73179×
-16
10
-16
10
An increasing interval is any interval which is not dead and the mean is positive while a decreasing interval is any interval which is not dead and its mean is negative.
Intervals with means of zero are not taken into account because they would be dead by definition.
In[]:=
meanclusterscalc[clusters_]:=Map[Mean[#]&,clusters]
In[]:=
meanclusters=meanclusterscalc[clusters]
Out[]=
{1.53139×,-2.63058×}
-16
10
-17
10
In[]:=
DerivWordState[x_,meanclusters_,wordDeathVal_]:=If[ Length[meanclusters]1, If[ Sign[meanclusters[[x]]]1, " increasing", " decreasing"], Which[ Abs[meanclusters[[x]]]≤Times[wordDeathVal,Mean[Drop[meanclusters,{x}]]], " dead", Sign[meanclusters[[x]]]1, " increasing", Sign[meanclusters[[x]]]-1, " decreasing" ]];
Now we generate a string, called statement, that will tell us which intervals are dead, increasing and decreasing. The locations of derivative values are not mapped to time yet, so it is only a rough indication to the user where something is increasing and where it is decreasing. statementClusters[] can print out a statement for the user. To view the output of statementClusters[] in RunWord[], use the commented line for the final line as RunWord[]. We use this statement to determine the volatility score of that word. A future feature would be to implement a output function that will show intervals with corresponding timestamp information.
Finally, the volatility is calculated by counting how many times the words “decreasing” and “increasing” are used in statement. “dead” intervals are not counted in the volatility score.
A future iteration might take into account how large of an increase or decrease is within the intervals and the absolute value of their means.
Another future implementation would be expanding further on the statement function within volatility.
RunWord[] runs all necessary calculations to fetch the volatility value of a particular word.
A future iteration might take into account how large of an increase or decrease is within the intervals and the absolute value of their means.
Another future implementation would be expanding further on the statement function within volatility.
RunWord[] runs all necessary calculations to fetch the volatility value of a particular word.
Below are my global variables.
word is only applicable to RunWord[]. It cannot be used by CollectList[],TranslateWords[], or TotalVolatility[].
maval refers to the variable of moving average applied to raw time and frequency data.
maDERval refers to the variable of moving average applied to derived time and frequency data.
wordDeathVal allows us to decide at what relative point to its sister clusters a mean on an interval represents a “dead” interval
language decides the language we will be analyzing the derivative of.
wordlist is my list of 151 words I decided might have somewhat consistent use across languages. Feel free to use the sublists I indicated in the introduction, or create your own.
maval and maDERval both work functionally, so long as they are near or above 30. If they are significantly below, volatility scores begin to grow significantly and become somewhat arbitrary, only really indicating the amount of data (or lack thereof), not its actual volatility.
wordDeathVal was an arbitrary value I chose based on what I thought signified word movement death, and can be optimized. Please let me know if you know more about this/have suggestions.
language, word, and wordlist can be changed to the user’s liking.
word is only applicable to RunWord[]. It cannot be used by CollectList[],TranslateWords[], or TotalVolatility[].
maval refers to the variable of moving average applied to raw time and frequency data.
maDERval refers to the variable of moving average applied to derived time and frequency data.
wordDeathVal allows us to decide at what relative point to its sister clusters a mean on an interval represents a “dead” interval
language decides the language we will be analyzing the derivative of.
wordlist is my list of 151 words I decided might have somewhat consistent use across languages. Feel free to use the sublists I indicated in the introduction, or create your own.
maval and maDERval both work functionally, so long as they are near or above 30. If they are significantly below, volatility scores begin to grow significantly and become somewhat arbitrary, only really indicating the amount of data (or lack thereof), not its actual volatility.
wordDeathVal was an arbitrary value I chose based on what I thought signified word movement death, and can be optimized. Please let me know if you know more about this/have suggestions.
language, word, and wordlist can be changed to the user’s liking.
Finally, here are our functions that will allow us to summarize estimated volatility for an entire language.
CollectList[] creates a list of all outputted volatility scores from RunWord[] for a certain list of words in a particular language. It will not be able to translate words if it is run on its own, and would need to be inputted a list with words in the particular language the user would like to use.
TranslateWords[] creates a list of translated words from an English list of words, into another language of the user’s choosing. If the chosen language is English, it will not run the translation.
DISCLAIMER: This function uses service credits. If you plan to calculate volatility for any language that is NOT English, even if you translated the wordlist by yourself, do not use TotalVolatility[]. It will automatically attempt to translate any list whose language was not indicated as English.
The reason I use TextTranslation[] (which costs service credits, as it relies on Microsoft data) instead of WordTranslation[] (which does not, it relies on Wolfram data) is because WordTranslation[] was inaccurate and after testing it with a few of the words in my wordslist, I decided against using it at this time. An example is shown below, where it suggests the translations for “hair” might be “decapitate”, “guillotine”, or one of the actual translations for “hair”. In general, I have seen that oftentimes, even when the function does choose the correct translations, they are less commonly used translations of the word (for example, poil as seen below is considered to be a less common word for hair in French), which would throw off the volatility in comparison to the volatility chosen for its corresponding word in English, in turn making the total volatility score incomparable to the one chosen for English, because uncommon words tend to have different derivative graphs than common words. Meanwhile, TextTranslation[] appears to consistently chose an accurate word that is the most used in its given language. I hope that in the future, this WordTranslation[] is improved upon so my code can be run in other languages without using service credits.
TotalVolatility[] runs TranslateWords[], then inputs them into CollectList[], which runs each word in TranslateWords[] in its corresponding language in RunWord[]. Then TotalVolatility[] takes the mean of all volatilities collected in CollectList[]. This will give the user a total “estimated volatility score” for the given language.
CollectList[] creates a list of all outputted volatility scores from RunWord[] for a certain list of words in a particular language. It will not be able to translate words if it is run on its own, and would need to be inputted a list with words in the particular language the user would like to use.
TranslateWords[] creates a list of translated words from an English list of words, into another language of the user’s choosing. If the chosen language is English, it will not run the translation.
DISCLAIMER: This function uses service credits. If you plan to calculate volatility for any language that is NOT English, even if you translated the wordlist by yourself, do not use TotalVolatility[]. It will automatically attempt to translate any list whose language was not indicated as English.
The reason I use TextTranslation[] (which costs service credits, as it relies on Microsoft data) instead of WordTranslation[] (which does not, it relies on Wolfram data) is because WordTranslation[] was inaccurate and after testing it with a few of the words in my wordslist, I decided against using it at this time. An example is shown below, where it suggests the translations for “hair” might be “decapitate”, “guillotine”, or one of the actual translations for “hair”. In general, I have seen that oftentimes, even when the function does choose the correct translations, they are less commonly used translations of the word (for example, poil as seen below is considered to be a less common word for hair in French), which would throw off the volatility in comparison to the volatility chosen for its corresponding word in English, in turn making the total volatility score incomparable to the one chosen for English, because uncommon words tend to have different derivative graphs than common words. Meanwhile, TextTranslation[] appears to consistently chose an accurate word that is the most used in its given language. I hope that in the future, this WordTranslation[] is improved upon so my code can be run in other languages without using service credits.
TotalVolatility[] runs TranslateWords[], then inputs them into CollectList[], which runs each word in TranslateWords[] in its corresponding language in RunWord[]. Then TotalVolatility[] takes the mean of all volatilities collected in CollectList[]. This will give the user a total “estimated volatility score” for the given language.
Although it takes a while to run, TotalVolatility[] gives us a really interesting number we can use to make assumptions about a particular language! I plan to also implement TotalVolatility[] showing the user the standard deviation of the items in CollectList[].
Below is the volatility score for English, but I recommend testing this on your own to see what you get in different languages!
Below is the volatility score for English, but I recommend testing this on your own to see what you get in different languages!
Conclusion/Future Work
The potential to check for language volatility across different languages opens a whole new view into computational linguistics, allowing us to think harder about how to quantify the driving forces of language change, and how many different factors including population of speakers, cultural context, and other factors might play a role in the change of languages in addition to quantifiable natural variations. I hope that this project will excite other linguists and coders and help lead to future studies on different volatilities between languages. As data is collected, it will be added to this notebook and updated.
In the future there are many different features I’d like to be able to apply to my function, many of which I mentioned as annotations in my code. For one, I’d like to be able to make this function more accessible by adding dynamic functions and making it more user friendly. I think by adding user friendly interface, my function might be able to reach people who aren’t necessarily coders but are still interested in the subject, or inspire someone to take up doing a similar project. I’d also like to make the derivation of volatility more mathematically based with more complex equations that will give more insight into the true volatility score, and have variables (like wordDeathVal and my wordslist) decided by actual research and study, instead of just intuition. Finally, one thing that continually inspired me was the simple idea on this thread https://mathematica.stackexchange.com/questions/16723/time-series-decomposition-in-mathematica/16845#16845 of decomposing data into trend, seasonal, and random. Currently my data only really considers trend, but I’d love to be able to utilize the users methods above to find seasonal trends across words.
In the future there are many different features I’d like to be able to apply to my function, many of which I mentioned as annotations in my code. For one, I’d like to be able to make this function more accessible by adding dynamic functions and making it more user friendly. I think by adding user friendly interface, my function might be able to reach people who aren’t necessarily coders but are still interested in the subject, or inspire someone to take up doing a similar project. I’d also like to make the derivation of volatility more mathematically based with more complex equations that will give more insight into the true volatility score, and have variables (like wordDeathVal and my wordslist) decided by actual research and study, instead of just intuition. Finally, one thing that continually inspired me was the simple idea on this thread https://mathematica.stackexchange.com/questions/16723/time-series-decomposition-in-mathematica/16845#16845 of decomposing data into trend, seasonal, and random. Currently my data only really considers trend, but I’d love to be able to utilize the users methods above to find seasonal trends across words.
Acknowledgements
A big thank you to my mentor, Megan Davis, who was always there to help when I was stuck on something or went too far down a rabbit hole.
Thank you to Jessica Shi, who helped me organize my (verifiably messy) code.
I learned so much doing this project, and I’m so excited to help it improve in the future!
Thank you to Jessica Shi, who helped me organize my (verifiably messy) code.
I learned so much doing this project, and I’m so excited to help it improve in the future!