In this experiment, we’re going to analyse rare and NER words in Wikipedia articles using Large Language Models through LLMFunctions in Wolfram Language.
Introduction
Introduction
Rare and NER words are extremely important in the field of Natural Language Processing (NLP). Rare words are particularly important to understand the sentiment of a sentence, most likely classical English literature and then to create tokenizers that are able to handle them properly and deflect the problem of incorrect tokenization.
Afterwards, NER (Named Entity Recognition) words are typically locations, organisations, brand names and other point of interest words. In NLP, Artificial intelligence labs spend a huge amount of resources and time to find NER words and create Neural Network models that can easily find NER words from large corpus of texts and contents. There are a lot of Competitions organised by different universities and platform where AI labs such as Amazon, Google, Baidu and others participate in these competitions and create state of the art Neural Network models that break previous benchmarks.
Currently, the most obvious method of finding rare and NER words is a time-consuming and computationally expensive process. With the recent advent of Large Language Models especially GPT models and papers that followed where text generation capabilities of GPT being analysed in layer by layer level, we are perfectly in the position of using large language models for finding rare and NER words in few seconds without any human annotation in the loop or overhead of training a fairly complicated model with intermediate amounts of computational power.
Afterwards, NER (Named Entity Recognition) words are typically locations, organisations, brand names and other point of interest words. In NLP, Artificial intelligence labs spend a huge amount of resources and time to find NER words and create Neural Network models that can easily find NER words from large corpus of texts and contents. There are a lot of Competitions organised by different universities and platform where AI labs such as Amazon, Google, Baidu and others participate in these competitions and create state of the art Neural Network models that break previous benchmarks.
Currently, the most obvious method of finding rare and NER words is a time-consuming and computationally expensive process. With the recent advent of Large Language Models especially GPT models and papers that followed where text generation capabilities of GPT being analysed in layer by layer level, we are perfectly in the position of using large language models for finding rare and NER words in few seconds without any human annotation in the loop or overhead of training a fairly complicated model with intermediate amounts of computational power.
Objective
Objective
We want to find rare words using Large Language Model and compare our results against traditional method
Description
Description
Traditional Method: Traditional Method in our experiment means, analysing a passage of text and remove common words and stop words using built-in wolfram language functions and then sort the words based on their rarity using the rarity function in Wolfram and finally remove any Numerical numbers to receive only words.
LLM Method: In LLM method, we use LLM functions built-in in Wolfram Language and a combination of carefully created prompt to find rare and NER words in given passage/dataset.
Preparation of Dataset: To create our dataset, we resorted to WikipediaData function in Wolfram Language and used a combination of list of articles and looped through the list of articles and stored the content of those articles in (.txt) file. Further we have split every single article into portions of 400 words. Because Wolfram Functions and OpenAI API can handle 400 words in per iteration.
LLM Method: In LLM method, we use LLM functions built-in in Wolfram Language and a combination of carefully created prompt to find rare and NER words in given passage/dataset.
Preparation of Dataset: To create our dataset, we resorted to WikipediaData function in Wolfram Language and used a combination of list of articles and looped through the list of articles and stored the content of those articles in (.txt) file. Further we have split every single article into portions of 400 words. Because Wolfram Functions and OpenAI API can handle 400 words in per iteration.
Crawling Wikipedia articles across 3 different genres, collecting 10 articles per genre and compiling a dataset (.txt format) to analyse it for rare words and NER (Name Entity Recognition) in those articles.
Crawling Philosophy articles
Crawling Philosophy articles
In[]:=
articleNames={"Metaphysics","Epistemology","Ethics","Logic","Philosophy_of_mind","Philosophy_of_science","Philosophy_of_language","Political_philosophy","Aesthetics","Existentialism"};
In[]:=
articleData=AssociationThread[articleNames,WikipediaData[#,"ArticlePlaintext"]&/@articleNames];
In[]:=
(*Getthedirectorypathofthenotebook*)notebookDirectory=NotebookDirectory[];
(*Saveeacharticleinaseparate.txtfile*)MapThread[Export[StringJoin[notebookDirectory,"/",#1,".txt"],#2,"Text"]&,{articleNames,Values[articleData]}];
In[]:=
MapThread[Export[StringJoin[notebookDirectory,"/Philosophy/",#1,".txt"],#2,"Text"]&,{articleNames,Values[articleData]}];
Crawling Scientific Articles
Crawling Scientific Articles
In[]:=
articleScience={"Scientific_method","Newton's_laws_of_motion","General_relativity","Quantum_mechanics","DNA","Big_Bang","Evolution","Cell_theory","Periodic_table","Climate Change"};
In[]:=
articleData=AssociationThread[articleScience,WikipediaData[#,"ArticlePlaintext"]&/@articleScience];
In[]:=
notebookDirectory=NotebookDirectory[];
In[]:=
MapThread[Export[StringJoin[notebookDirectory,"/Science/",#1,".txt"],#2,"Text"]&,{articleScience,Values[articleData]}];
Crawling Cognitive Science Articles
Crawling Cognitive Science Articles
In[]:=
articleCognitive={"Cognitive_science","Cognition","Artificial_intelligence","Neuroscience","Psychology","Perception","Memory","Language_acquisition","Attention","Decision-making"};
In[]:=
articleData=AssociationThread[articleCognitive,WikipediaData[#,"ArticlePlaintext"]&/@articleCognitive];
In[]:=
notebookDirectory=NotebookDirectory[];
In[]:=
MapThread[Export[StringJoin[notebookDirectory,"/Cognitive Science/",#1,".txt"],#2,"Text"]&,{articleCognitive,Values[articleData]}];
(*Setthedirectorypathwherethe.txtfilesarelocated*)directoryPath="F:\\CS\\Wolfram Fellowship\\Science";
(*Getalistofall.txtfilesinthedirectory*)fileList=FileNames["*.txt",directoryPath];
(*Createanewdirectorytostoretheslicedfiles*)newDirectoryPath=FileNameJoin[{directoryPath,"Science Chuck"}];CreateDirectory[newDirectoryPath];(*Loopthrougheach.txtfile*)Do[(*Readthecontentofthefile*)fileContent=Import[file,"Text"];(*Splitthecontentinto400-wordchunks*)chunkList=Partition[StringSplit[fileContent," "],400];(*Loopthrougheachchunkandsaveasseparate.txtfiles*)Do[(*Generatethenewfilename*)newFileName=StringJoin[FileBaseName[file],"_",ToString[i],".txt"];(*Jointhewordsinthechunkandsaveasanewfile*)Export[FileNameJoin[{newDirectoryPath,newFileName}],StringJoin[Riffle[chunkList[[i]]," "]]],{i,Length[chunkList]}],{file,fileList}]
Which is why, we performed the split.
Traditional Method
Like we discussed before, in traditional method, we used a combination of functions to remove common words, duplicates and stop words and finally, used rarity to find rare words in a given passage.
Like we discussed before, in traditional method, we used a combination of functions to remove common words, duplicates and stop words and finally, used rarity to find rare words in a given passage.
In[]:=
paragraph="The regulation of artificial intelligence is the development of public sector policies and laws for promoting and regulating artificial intelligence (AI); it is therefore related to the broader regulation of algorithms.The regulatory and policy landscape for AI is an emerging issue in jurisdictions globally. According to AI Index at Stanford, the annual number of AI-related laws passed in the 127 survey countries jumped from one passed in 2016 to 37 passed in 2022 alone.Between 2016 and 2020, more than 30 countries adopted dedicated strategies for AI.Most EU member states had released national AI strategies, as had Canada, China, India, Japan, Mauritius, the Russian Federation, Saudi Arabia, United Arab Emirates, US and Vietnam. Others were in the process of elaborating their own AI strategy, including Bangladesh, Malaysia and Tunisia.The Global Partnership on Artificial Intelligence was launched in June 2020, stating a need for AI to be developed in accordance with human rights and democratic values, to ensure public confidence and trust in the technology. Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher published a joint statement in November 2021 calling for a government commission to regulate AI. In 2023, OpenAI leaders published recommendations for the governance of superintelligence, which they believe may happen in less than 10 years.In a 2022 Ipsos survey, attitudes towards AI varied greatly by country; 78% of Chinese citizens, but only 35% of Americans, agreed that \"products and services using AI have more benefits than drawbacks\". A 2023 Reuters/Ipsos poll found that 61% of Americans agree, and 22% disagree, that AI poses risks to humanity. In a 2023 Fox News poll, 35% of Americans thought it \"very important\", and an additional 41% thought it \"somewhat important\", for the federal government to regulate AI, versus 13% responding \"not very important\" and 8% responding \"not at all important\". ";
In[]:=
words=StringSplit[ToLowerCase@StringReplace[paragraph,Except[WordCharacter]->" "]];
In[]:=
words=DeleteDuplicates[StringReplace[words,Except[WordCharacter]->""]];
In[]:=
functionWords=DeleteStopwords[paragraph];
In[]:=
rarityScores=WordFrequencyData[words];
In[]:=
rarityScores=ReplaceAll[rarityScores,Missing[___]->0];
In[]:=
threshold=0.1;(*Adjustthisvalueasperyourdefinitionofrarity*)rareWords=Select[words,rarityScores[#]<threshold&&!MemberQ[functionWords,#]&];
In[]:=
commonWords=WordList[];
In[]:=
rareWords=Complement[rareWords,commonWords];
In[]:=
rareWords=Select[rareWords,!StringMatchQ[#,NumberString]&];
In[]:=
Column[rareWords]
Out[]=
ai |
algorithms |
americans |
arab |
arabia |
attitudes |
bangladesh |
benefits |
broader |
canada |
chinese |
citizens |
countries |
daniel |
drawbacks |
elaborating |
emirates |
eric |
eu |
henry |
huttenlocher |
including |
india |
ipsos |
jumped |
june |
jurisdictions |
kissinger |
launched |
laws |
malaysia |
mauritius |
november |
openai |
others |
passed |
policies |
poses |
products |
promoting |
recommendations |
released |
responding |
reuters |
rights |
risks |
russian |
saudi |
schmidt |
stanford |
states |
stating |
strategies |
superintelligence |
tunisia |
vietnam |
LLM Function
In LLM Function we performed, the finding of rare words and NER through carefully creating a prompt and iterating that prompt over a passage and found rare and NER words.
In LLM Function we performed, the finding of rare words and NER through carefully creating a prompt and iterating that prompt over a passage and found rare and NER words.
Finding rare and NER (Named Entity Recognition) words using Large Language Model (e.g. GPT 3.5 Turbo)
Finding rare and NER (Named Entity Recognition) words using Large Language Model (e.g. GPT 3.5 Turbo)
Extension Work
Extension Work
We have performed an extension work by creating inflection table for exotic languages such as Latin and Hebrew
Future work
Future work
In the future, Large Language Models can be used to understand cultural shifts in languages and perform different linguistics analysis in Languages. On top of that, by analysing the layers in Large Language Models, we can further unlock how it learns and deciphers pattern in Constructed Languages such as High Valyrian. Finally, we propose to perform an in-depth analysis between Large Language Models and domain specific Neural Networks trained to find NER words and perform a benchmark to understand how capable Large Language models are in the domain of NER words as well as whether Large Language model’s NER finding capabilities are dependent on certain type data and whether synthetic datasets can aid in this field or not.
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Analysing rare and NER words in Wikipedia
by Tawsif Ahmed
Wolfram Community, STAFF PICKS, July 12, 2023
https://community.wolfram.com/groups/-/m/t/2958560
by Tawsif Ahmed
Wolfram Community, STAFF PICKS, July 12, 2023
https://community.wolfram.com/groups/-/m/t/2958560