Introduction
Introduction
How do you find insights from large, complex constitutional texts of two hundred and thirty countries in just two weeks?
...You employ a Large Language Model.
...You employ a Large Language Model.
This project utilizes Wolfram’s Large Language Model (LLM) capabilities to investigate how features of a country’s constitution might help us understand its outputs in terms of its performance on key socio-economic/development variables, including GDP, Global Peace Index, Human Development Index, and Democracy Index. Using few-shot learning approach of Generative Pre-trained Transformer , we asked Open AI’s GPT-3.5 to classified countries into governance types and then assess to what extent their constitutions emphasized values such as democracy, religion, economic development, equality, state authority and sovereignty. We got inspiration for some of the classification from an analysis of the World Values Survey by Ronald Inglehart and Christian Welzel whose Inglehart–Welzel Cultural Map puts countries into one of two cultural dimensions:
◼
Traditional values versus Secular-rational values and
◼
Survival values versus Self-expression values
We then then use exploratory data analysis in the Wolfram Language to establish relationships between how a country’s constitution is structured and how that affects or enhances peace,democracy and the socio-economic advancements of its citizens. Specifically, we compared GPT-3.5’s ratings of countries with GDP Per Capita (PPP) as well as indices on peace, development, and democracy.
Some of the key questions we will explore include:
Some of the key questions we will explore include:
◼
Can LLMs efficiently classify countries based on their constitutions?
◼
How are the constitutions of various countries interrelated ?
◼
To what extent does a country’s constitution promote egalitarianism, democracy, development, etc.?
◼
Is there a correlation between GDP/development and a country’s constitutional framework?
Getting the Data
Getting the Data
Let’s start by getting our data (GDP,the peace index, the democracy index and the human development index) from Wikipedia:
In[]:=
gdpPerCapitaPPPURL="https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(PPP)_per_capita";globalPeaceIndexURL="https://en.wikipedia.org/wiki/Global_Peace_Index";democracyIndexURL="https://en.wikipedia.org/wiki/The_Economist_Democracy_Index";developmentIndexURL="https://en.wikipedia.org/wiki/List_of_countries_by_Human_Development_Index";
In[]:=
gdpWiki=;peaceIndexWiki=;democracyIndexWiki=;developmentIndexWiki=;
Some data cleaning is definitely required. The Wikipedia data tables are quite dirty: they contain many unwanted strings, characters and white-spaces that the following codes should help address.
In[]:=
gdpPerCapitaPPPcl1=gdpWiki/.{(b_String/;StringContainsQ[b,","]):>StringReplace[b,","""],(a_String/;StringEndsQ[a," *"]):>StringTake[a,{1,-3}],s_String:>StringReplace[s,RegularExpression["\\[n\\s*\\d*\\]"]->""]};gdpPerCapitaPPPcl2=Cases[gdpPerCapitaPPPcl1/.(i_String/;StringStartsQ[i," "]):>ToExpression[StringReplace[i," """]],{__,_?NumericQ},∞];gdpPerCapita=gdpPerCapitaPPPcl2[[All,{1,-2}]];
We have our GDP data from Wikipedia.
List of countries
List of countries
We shall analyze the +200 countries in the GDP Per Capita Index Table on Wikipedia. We shall also get all of the texts in the constitutions of countries in the world from Wolfram’s ResourceData Repository. The Wolfram Repository contains the constitutional texts of one hundred and forty five countries.
In[]:=
worldConstitutions=ResourceData["World Constitutions"];countryList=EntityValue[Normal[worldConstitutions[All,1]],"Name"];
In[]:=
SeedRandom[1230];selectedCountriesconst=RandomSample[countryList,130];exampleCountries=Complement[countryList,selectedCountriesconst];valCountries=RandomSample[selectedCountriesconst,10];
Here, we exclude a few countries that we shall use for training the LLM about what to do:
In[]:=
selectedCountries=Complement[gdpPerCapita[[All,1]],exampleCountries];
Some more data cleaning. Yes, It’s true that data pre-processing constitutes a significant part of most data science tasks.
In[]:=
selectedCountries[[{44,55,160,175}]]={"Republic of Congo","Congo","Saint Helena","Sint Maarten"};
In[]:=
selectedCountries=Delete[selectedCountries,{{65},{163}}];
In[]:=
gdpPerCapitaFinal=Cases[gdpPerCapita,{Alternatives@@selectedCountries,___}];
The following codes cleans the Wikipedia data on Peace, Democracy and Human Development:
Here is the cleaned Peace Index Table:
In[]:=
peaceIndex=Cases[peaceIndexWiki,{_,_,_,_,_,_,_,_,_?NumericQ},∞]/.{(d_String/;StringEndsQ[d,"="]):>ToExpression[StringTake[d,{1,-2}]],(h_String/;StringEndsQ[h," "]):>StringTake[h,{1,-2}],"Türkiye"->"Turkey"};peaceIndexFinal=peaceIndex[[All,{1,3}]];
and then the Global Democracy Index Table:
In[]:=
getPos=FirstPositiondemocracyIndexWiki,;democracyIndex=ReplaceAlldemocracyIndexWiki[[First[getPos]]][[2;;]]/.(g_String/;StringEndsQ[g," "]):>StringTake[g,{1,-2}],[[All,2;;4]];
and finally, the Human Development Index.
In[]:=
getPos2=FirstPositiondevelopmentIndexWiki,;developmentIndexRaw=developmentIndexWiki[[First[getPos2]]][[3;;,2;;]];developmentIndexRaw2=Map[If[Length[#]>2,Take[#,{2,3}],#]&,developmentIndexRaw];developmentIndex=Module[{prev={}},Map[If[MemberQ[#,_String?(StringContainsQ[#,"%"]&)],Replace[#,{x_String/;StringContainsQ[x,"%"]:>If[Length[prev]>=2,prev[[2]],x]},{1}],prev=#;#]&,developmentIndexRaw2]]/.(h_String/;StringContainsQ[h," "]):>StringTake[h,{1,-2}];
Putting it all together...
Putting it all together...
Socio-economic metrics
Socio-economic metrics
Let’s combine all the Wikipedia tables .
First, we shall update the various tables into a Wolfram Dataset which can be quite powerful to work with.
First, we shall update the various tables into a Wolfram Dataset which can be quite powerful to work with.
In[]:=
gdpSet=Dataset[gdpPerCapitaFinal];
In[]:=
keysGDP={"country","gdpPPP"};gdpUpdated=AssociationThread[keysGDP->#]&/@gdpSet;
In[]:=
peaceIndexSet=Dataset[peaceIndexFinal];keysPeaceIndex={"country","peaceIndex"};peaceIndexUpdated=AssociationThread[keysPeaceIndex->#]&/@peaceIndexSet;
In[]:=
demoIndexSet=Dataset[democracyIndex];keysDemoIndex={"country","category","democracyIndex"};demoIndexUpdated=AssociationThread[keysDemoIndex->#]&/@demoIndexSet;
In[]:=
devIndexSet=Dataset[developmentIndex];keysDevIndex={"country","developmentIndex"};devIndexUpdated=AssociationThread[keysDevIndex->#]&/@devIndexSet;
Now, we are going to join all the four tables using JoinAcross. We will use a “Left” join to make sure we are dealing with only the specified list of countries in the GDP Table.
In[]:=
sMetricTable=JoinAcross[JoinAcross[gdpUpdated,peaceIndexUpdated,"country","Left"],JoinAcross[devIndexUpdated,demoIndexUpdated,"country","Left"],"country","Left"];
Constitutional Metrics
Constitutional Metrics
We want to generate data about the features of the constitutions of countries from Open AI. It might be cheating but we shall assume already knows a lot --more than the smartest human-- about the constitutions of most countries. Another approach would be to provide the actual texts to the AI or using OpenAIEmbeddings to extract and parse relevant sections of the constitutions. We are too lazy for either of those approaches plus we have very limited resources for the OpenAI API requests.
Prompt Engineering
Prompt Engineering
We will use the following prompt to get GPT-3.5 to provide us with the information we require:
Prompt: “You are an expert in world constitutions who has been asked to assess countries based on the texts of their constitutions. Your ratings and classifications of countries should be based solely on the constitutional texts of the country, no need to consider any political or socio-economic contexts.
1. Classify a country into democracy, authoritarian,monarchy,hybrid or other based solely on its constitution
2. On a scale of one to five, to what extent does country constitution emphasizes democratic values
3. On a scale of one to five, to what extent does country constitution emphasizes the importance of God
4. On a scale of one to five, to what extent does country constitution emphasizes the importance of religion
5. On a scale of one to five, to what extent does country constitution emphasizes racial equality
6. On a scale of one to five, to what extent does country constitution emphasizes gender equality
7. On a scale of one to five, to what extent does country constitution emphasizes the economic well-being of the country and its citizens
8. On a scale of one to five, to what extent does country constitution uphold or protect the right to abortion?
9. On a scale of one to five, to what extent does country constitution uphold or protect LGBTQ rights?
10. On a scale of one to five, to what extent does country constitution emphasizes importance of state authority
11. On scale of one to five, to what extent does country constitution emphasizes national sovereignty
12. On a scale of one to five,to what extent does country constitution make provisions for regulating technology and innovation”
1. Classify a country into democracy, authoritarian,monarchy,hybrid or other based solely on its constitution
2. On a scale of one to five, to what extent does country constitution emphasizes democratic values
3. On a scale of one to five, to what extent does country constitution emphasizes the importance of God
4. On a scale of one to five, to what extent does country constitution emphasizes the importance of religion
5. On a scale of one to five, to what extent does country constitution emphasizes racial equality
6. On a scale of one to five, to what extent does country constitution emphasizes gender equality
7. On a scale of one to five, to what extent does country constitution emphasizes the economic well-being of the country and its citizens
8. On a scale of one to five, to what extent does country constitution uphold or protect the right to abortion?
9. On a scale of one to five, to what extent does country constitution uphold or protect LGBTQ rights?
10. On a scale of one to five, to what extent does country constitution emphasizes importance of state authority
11. On scale of one to five, to what extent does country constitution emphasizes national sovereignty
12. On a scale of one to five,to what extent does country constitution make provisions for regulating technology and innovation”
Let’s train our Large Language Model using Wolfram’s new LLMExampleFunction:
In[]:=
constClassifier=LLMExampleFunction;
We shall now define and evaluate the function and then iconize the results to make sure it is always stored in the notebook.
consTable=Map[constClassifier,gdpPerCapitaFinal[[All,1]]]
Iconize[consTable,"LLM Results"]
Time to add country column to our constitution data and store the results.
In[]:=
combinedTables=gdpPerCapitaFinal[[All,1]],;tableKeys={"country","GovernanceType","EmphasisonDemocracy","EmphasisOnGod","emphasisOnReligion","empahsisEqualityRace","emphasisGenderEquality","emphasisEconomy","rejectsAbortion","rejectsHomosexuality","emphasisAuthority","emphasisNationalPride","emphasisTechnology"};
In[]:=
llmConstitutionRatings=Transpose[{combinedTables[[1]],Sequence@@Transpose[combinedTables[[2]]]}];llmConstitutionRatingsFinal=Dataset[AssociationThread[tableKeys,#]&/@llmConstitutionRatings];(*WehavealsoaddedcolumnnamesandconvretedtheconstituiondataintoaDataset*)
We will also rename the variables to make them readable:
In[]:=
tableKeysII={"Country","Governance Type","Emphasis on Democracy","Emphasis on God","Emphasis on Religion","Empahsis on Equality of Race","Emphasis on Gender Equality","Emphasis on Economy","Protects Right to Abortion","Protects LGBTQ Rights","Emphasis on Authority","Emphasis on National Pride","Emphasis on Innovation and Technology"};
The following codes iconize the final constitution data.
Iconize[Prepend[llmConstitutionRatings//Sort,tableKeysII],"LLMConsResults"];Iconize[Prepend[Values/@Normal[sMetricTable]//Sort,Keys[Normal[sMetricTable][[1]]]],"Socioeconomic Data"];
Our data still contains an unwanted value (the global averages/index the data called “World”). We need to remove it.
Let’s also remove the Missing[Unwanted] from the data.
Now that we have all the data we need, in the format we want, let’s explore.
So what did we find?
So what did we find?
Relationships between constitutional features and socio-economic development
Relationships between constitutional features and socio-economic development
To establish the relationship between countries, we shall create a MatrixPlot a showing the correlations among and within the constitutional and socio-economic indexes.
The following code computes the various correlations:
For our Correlation MatrixPlot, we will need the labels of all variables . This is provided as an association with the following code :
We find several interesting correlations including the following:
◼
The Human Development Index of is positively correlated with a number of constitutional indicators including emphasis on racial equality, technology/innovation and patriotism.
◼
A negative correlation between emphasis on God or religion in a constitution and the GDP Per Capita of the country. This suggest that some of the most religious countries tend to also be the poorest.
◼
We also found a positve correlation between emphasis on democracy in the constitution and peace.
In which countries are certain values emphasized in the constitution?
In which countries are certain values emphasized in the constitution?
It’s also possible to find the most popular words in the constitutions of most countries.
It’s also possible to find the most popular words in the constitutions of most countries.
To do this, we shall extract the texts of the world' s constitutions available in the ResourceFunction repository .
and there is our WordCloud of the most popular words in constitutional texts .
We might also wish to see how frequently certain words appear in constitutions of the world .The most popular words can be visualized in a WordCloud as follows:
Look, the word “MEN” is the most frequent of the keywords in world constitutions! Shall is the most frequent word, overall.
Graphs and Networks: Which countries share similar constitutions?
Graphs and Networks: Which countries share similar constitutions?
The following code creates a Network Graph of countries based on their GPT-3 ratings on the various indicators.
...and here is how all the countries cluster based on their GPT-3.5 classification across the various indicators
...and here is how all the countries cluster based on their GPT-3.5 classification across the various indicators
Communities of countries with similar constitutions emerge. It’s possible to check them out using the following code and setting VertexLabel to Automatic:
A close examination of the charts shows that countries like the United States, Finland, Vietnam, Rwanda and Egypt have very unique constitutions as shown by the isolated edges in the figure above.
A close examination of the charts shows that countries like the United States, Finland, Vietnam, Rwanda and Egypt have very unique constitutions as shown by the isolated edges in the figure above.
We can also show the countries with similar constitutions based on the various indicators as follows:
Countries are clustered based on how little or how much they emphasize certain values in their constitutions..
Conclusion
Conclusion
We have explored the machine learning capabilities of the Wolfram Language to demonstrate how to extract useful information from huge volumes of text. In this case we found that OpenAI’s GPT-3.5 did a decent job of providing an assessment of the values of two hundred and ten countries based on how they are emphasized in the constitution. We found quite interesting insights from the data generated by the Model including a negative correlation between Emphasis on God/religion in their constitutional texts and its per Capital Gross Domestic Product at Purchasing Power Parity.
Key Words
Key Words
◼
Large Language Model
◼
Generative Pre-trained Transformer
◼
OpenAI
◼
Constitution
Acknowledgements
Acknowledgements
I am incredibly indebted to the staff of Wolfram Research and the Wolfram Summer School especially Mads and Yi for a thoroughly inspiring Summer School experience . I also wish to particularly thank my Summer School mentors, Daniel George (Ph.D) and Daniel Robinson who provided useful guidance and tips throughout. I also would like to thank Jofre Espiguel, Christian Pasquel, Alejandra Ortiz and Mark Greenberg for their support with creating and/or debugging some of the Mathematica code used in this project.
Stephen Wolfram provided really useful ideas during the project meeting that helped shape the overall direction of this project. I offer my sincere gratitude.
Stephen Wolfram provided really useful ideas during the project meeting that helped shape the overall direction of this project. I offer my sincere gratitude.
References
References
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Using large language models to generate insights from world constitutions
by Joel Abdulai Kallon
Wolfram Community, STAFF PICKS, July 12, 2023
https://community.wolfram.com/groups/-/m/t/2959735
by Joel Abdulai Kallon
Wolfram Community, STAFF PICKS, July 12, 2023
https://community.wolfram.com/groups/-/m/t/2959735