Nowadays, play scripts are all over the internet, but they are often lengthy and difficult to navigate. These texts are often unstructured, so you may have a hard time quickly finding what you're looking for. Some play scripts are marked up in formats such as TEI and XML, making it slightly easier to traverse, but these texts are overwhelmed by tags and numerous descriptors. It takes a while to become accustomed to these formats and understand how everything is formatted. The goal of my project is to implement additional forms of text segmentation, specifically regarding plays. Given a marked up play, I am able to process it into a structured, hierarchical format, so that it may be easily interpreted by anyone. With this newly formatted text, I am able to conduct numerous analyses on different plays to find out information such as who the most prevalent characters are or what the main themes of each act are.
Introduction
Introduction
Text Encoding Initiative, or TEI, is a variation of the well-known XML format. It currently acts as the standard for maintaining texts digitally, and is organized in more of a semantic matter. In TEI, everything from acts to individual words have their own tag, so everything is accounted for.
The tags mentioned in this project are as follows:
The tags mentioned in this project are as follows:
Methodology
Methodology
Data Importing
Data Importing
The text used in this project is TEI-formatted and was downloaded from The Folger Shakespeare Library.
Importing the play Romeo and Juliet from the cloud.
In[]:=
Out[]//Short=
XMLObject[Document][{XMLObject[ProcessingInstruction][xml-model,href="https://raw.githubusercontent.com/TEIC/TEI-Simple/master/teisimple.rng" type="application/xml" schematypens="http://relaxng.org/ns/structure/1.0"]},XMLElement[TEI,{{http://ww…000/xmlns/,…}…},{1,XMLElement[text,{},1]}],{}]
Text Processing Rules
Text Processing Rules
By just looking at Romeo and Juliet in TEI format, it's difficult to figure out where acts begin and what lines are assigned to each character. In order to convert the play into a more legible and structured format, I created rules and patterns to process the data.
Word Processing
Word Processing
The wordProcessingRule deals with the most fundamental structure in our text — words. You can think of it as the base case to our recursive function. In a TEI-formatted document, a single sentence is broken down into words, punctuation characters, whitespace, and line beginnings. Each of these properties has it's own line, so you must read down several lines in order to fully read and understand a sentence. The following wordProcessingRule takes all of these attributes into account and allows us to label and sort through all of them in order to parse the text more easily later on.
Specifically, wordProcessingRule assigns the data into seven different categories, with the main categories being:
Specifically, wordProcessingRule assigns the data into seven different categories, with the main categories being:
◼
Word: Either parses the <c> tag into whitespace or keeps track of the words and punctuation in a sentence.
◼
Type: Stores and renames the tag type into a more understandable label.
◼
ID: Each tag has a unique number/ID associated with it.
◼
Number: Each section in the play is broken down such that each corresponding group of tags has the same number.
◼
Lemma: Stores the root word(s) of a given word in the play.
If a certain data point doesn't fall into some of the categories, it is given a label of "Missing" for that specific property.
In[]:=
wordProcessingRule=XMLElement[type:("w"|"pc"|"c"|"lb"),attrib_List,word_]:>Module[{attassoc=Association@@attrib,w=Replace[word,{ww_}:>ww]},<|"Word"->Which[MatchQ[type,"c"]," ",True,w],"Type"->Replace[type,{"w"->"Word","pc"->"Punctuation","c"->"Whitespace","lb"->"Line Beginning"}],"ID"->Lookup[attassoc,Key[{"http://www.w3.org/XML/1998/namespace","id"}],Missing["NotAvailable"]],"Number"->Lookup[attassoc,"n",Missing["NotAvailable"]],"Lemma"->Lookup[attassoc,"lemma",Missing["NotAvailable"]],"Analysis"->Lookup[attassoc,"ana",Missing["NotAvailable"]],"Rendition"->Lookup[attassoc,"rendition",Missing["NotAvailable"]]|>];
The following is an example of a sentence that we would apply the wordProcessingRule to. The sentence “Gregory, on my word we’ll not carry coals.” is broken down into several different lines. Each line represents one line beginning, character, whitespace, or punctuation mark from the sentence.
In[]:=
exampleSentence=Cases[playText,XMLElement["p",{{"http://www.w3.org/XML/1998/namespace","id"}"p-0015"},_],Infinity]
Out[]=
{XMLElement[p,{{http://www.w3.org/XML/1998/namespace,id}p-0015},{XMLElement[lb,{{http://www.w3.org/XML/1998/namespace,id}ftln-0015,n1.1.1},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002620,n1.1.1,lemmaGregory,ana#n1-nn},{Gregory}],XMLElement[pc,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002630,n1.1.1},{,}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002650,n1.1.1,lemmaon,ana#acp-p},{on}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002670,n1.1.1,lemmamy,ana#po},{my}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002690,n1.1.1,lemmaword,ana#n1},{word}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002710,n1.1.1,lemmawe|will,ana#pns|vmb},{we’ll}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002730,n1.1.1,lemmanot,ana#xx},{not}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002750,n1.1.1,lemmacarry,ana#vvi},{carry}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002770,n1.1.1,lemmacoal,ana#n2},{coals}],XMLElement[pc,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002780,n1.1.1},{.}]}]}
Applying the wordProcessingRule to previous sentence.
In[]:=
exampleSentence/.wordProcessingRule
Out[]=
{XMLElement[p,{{http://www.w3.org/XML/1998/namespace,id}p-0015},{Word{},TypeLine Beginning,IDftln-0015,Number1.1.1,LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],WordGregory,TypeWord,IDfs-rom-0002620,Number1.1.1,LemmaGregory,Analysis#n1-nn,RenditionMissing[NotAvailable],Word,,TypePunctuation,IDfs-rom-0002630,Number1.1.1,LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordon,TypeWord,IDfs-rom-0002650,Number1.1.1,Lemmaon,Analysis#acp-p,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordmy,TypeWord,IDfs-rom-0002670,Number1.1.1,Lemmamy,Analysis#po,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordword,TypeWord,IDfs-rom-0002690,Number1.1.1,Lemmaword,Analysis#n1,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordwe’ll,TypeWord,IDfs-rom-0002710,Number1.1.1,Lemmawe|will,Analysis#pns|vmb,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordnot,TypeWord,IDfs-rom-0002730,Number1.1.1,Lemmanot,Analysis#xx,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordcarry,TypeWord,IDfs-rom-0002750,Number1.1.1,Lemmacarry,Analysis#vvi,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordcoals,TypeWord,IDfs-rom-0002770,Number1.1.1,Lemmacoal,Analysis#n2,RenditionMissing[NotAvailable],Word.,TypePunctuation,IDfs-rom-0002780,Number1.1.1,LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable]}]}
As we can see, the text looks a lot cleaner, which each attribute being properly labeled. If we remove all of the “Missing” characteristics, we can get a clearer view of the attributes and have a better idea of how the rule is working.
In[]:=
noMissing=DeleteCases[exampleSentence/.wordProcessingRule,Missing[_],Infinity]
Out[]=
{XMLElement[p,{{http://www.w3.org/XML/1998/namespace,id}p-0015},{Word{},TypeLine Beginning,IDftln-0015,Number1.1.1,WordGregory,TypeWord,IDfs-rom-0002620,Number1.1.1,LemmaGregory,Analysis#n1-nn,Word,,TypePunctuation,IDfs-rom-0002630,Number1.1.1,Word ,TypeWhitespace,Wordon,TypeWord,IDfs-rom-0002650,Number1.1.1,Lemmaon,Analysis#acp-p,Word ,TypeWhitespace,Wordmy,TypeWord,IDfs-rom-0002670,Number1.1.1,Lemmamy,Analysis#po,Word ,TypeWhitespace,Wordword,TypeWord,IDfs-rom-0002690,Number1.1.1,Lemmaword,Analysis#n1,Word ,TypeWhitespace,Wordwe’ll,TypeWord,IDfs-rom-0002710,Number1.1.1,Lemmawe|will,Analysis#pns|vmb,Word ,TypeWhitespace,Wordnot,TypeWord,IDfs-rom-0002730,Number1.1.1,Lemmanot,Analysis#xx,Word ,TypeWhitespace,Wordcarry,TypeWord,IDfs-rom-0002750,Number1.1.1,Lemmacarry,Analysis#vvi,Word ,TypeWhitespace,Wordcoals,TypeWord,IDfs-rom-0002770,Number1.1.1,Lemmacoal,Analysis#n2,Word.,TypePunctuation,IDfs-rom-0002780,Number1.1.1}]}
Although we would not remove any of the Missing values when working with the text in practice, this view of the sentence allows us to clearly see how each line is broken down.
Stage Directions
Stage Directions
A play script consists of numerous stage directions, and each direction informs us about two main things—what characters are involved and a description of how the characters must proceed within a scene. In a TEI-formatted play, we have a <stage> tag to indicate that whatever information follows will be part of a stage direction.
One level beneath the <stage> tag, we will find our word processing tags (<w>, <pc>, <c>, <lb>). Thus, once we process our stage tag, we must go one level down and call the wordProcessingRule to fully process the given text. Whenever we need to go a couple of levels down and call a rule recursively, we place the referenced rule in the "Contents" attribute.
One level beneath the <stage> tag, we will find our word processing tags (<w>, <pc>, <c>, <lb>). Thus, once we process our stage tag, we must go one level down and call the wordProcessingRule to fully process the given text. Whenever we need to go a couple of levels down and call a rule recursively, we place the referenced rule in the "Contents" attribute.
The stageDirectionRule processes any tag that begins with the tag <stage>.
In[]:=
stageDirectionRule=XMLElement[type:"stage",attrib_List,outType_]:>Module[{attassoc=Association@@attrib},<|"ID"->Lookup[attassoc,"id",Missing["NotAvailable"]],"Number"->Lookup[attassoc,"n",Missing["NotAvailable"]],"Type"->Lookup[attassoc,"type",Missing["NotAvailable"]],"Characters"->Lookup[attassoc,"who",Missing["NotAvailable"],StringSplit],"Contents"->(outType//.wordProcessingRule)|>];
Headings
Headings
Another tag we use is called the <head> tag. Essentially, this tag acts as a heading and oftentimes contains information about what scene or act is coming up.
The headRule searches for all of the <head> tags and begins processing from that level.
In[]:=
headRule=XMLElement[type:"head",attrib_List,outType_]:>Module[{attassoc=Association@@attrib},<|"Type"->Lookup[attassoc,"type",Missing["NotAvailable"]],"ID"->Lookup[attassoc,Key[{"http://www.w3.org/XML/1998/namespace","id"}],Missing["NotAvailable"]],"Contents"->(outType//.wordProcessingRule)|>];
Lines
Lines
Next, we have the lineRule, which accounts for the <l> tag. This tag represents each line in a play. If a certain character has multiple lines to speak in a row, each line will be broken down into <l> tags, and each <l> is further broken down into the word processing tags.
The lineRule processes all lines that include the <l>, or line, tag.
Speakers
Speakers
The <speaker> tag in a play keeps track of the character(s) who perform next. A stage direction or a sentence usually follows a speaker rule, so to account for both of these possibilities, the speakerRule leads into both the stage direction and word processing rules.
The speakerRule starts processing from the "speaker" rule, and works its way down from there.
Paragraphs
Paragraphs
In a prose play, the <p> tag divides speech into paragraphs. Each paragraph is filled with sentences, so we must call the wordProcessingRule next to drill down further.
Searching for each element beginning with a <p>, the pRule goes about processing the data.
Speech
Speech
The next rule searches for the <sp> tag, which stands for speech. Along with paragraphs and lines, a single speech block contains stage directions and speakers. To account for all of these conditions, we make sure to call the line, speaker, paragraph, and stage direction rules after we process the information in the speech tag.
Here, we can see the spRule looking for all the <sp> tags and processing the data following this pattern.
Main Rule
Main Rule
Finally, we have the mainRule. All of the previous rules culminate to this main rule, because it will process any tag in a play. We can apply this rule to any block of text and it will process it into a clean, readable format. The highest level tag in a play is the <div> tag, and the prologue, along with each act, each have this tag. Below this tag, we have more tags ranging from speech to stage directions to a heading, so those rules are called recursively in the "Contents" section.
The final, main rule that will take care of processing that play.
To see the main rule in action, we will navigate to Act 4 of the play and see how it breaks down each individual component.
Once again, if we view Act 4 without the Missing elements, we have a clear picture of what is going on.
Cast Rule
Cast Rule
We can also create another rule to grab the character list from the play.
Exploration
Exploration
Now that the play is nicely formatted, it is a lot easier to analyze the text. I generated a created a couple of new functions and conducted some analyses about the text itself along with characters in the text.
Functions and Text Analysis
Functions and Text Analysis
The following DisplayPrologue function can be used to display the prologue of any given TEI-formatted play. We can simply search for the "Contents" and "Word" labels under the "prologue" section in the processed text, and the prologue will be displayed.
We can also easily find out the character list in the play. This is done by first applying the castRule to the unprocessed text.
Then, the CastList function grabs each character from castListData and clearly prints out all the characters and extra people in the play
I can even find all of a certain character's lines in a specific scene. First, we must store our acts and scenes in variables so that we can easily access them later on in the function.
Next, the FindLines functions searches through all of a character's speech blocks in the specified act and scene. The lines are displayed following this process.
Here, I can find all of Romeo' s lines in Act 2 Scene 4
And all of Juliet' s lines in Act 5 Scene 3
Visuals
Visuals
With the newly structured text, I was able to generate graphs and charts to get a better understanding of what is happening in each act and scene.
The following image shows a word cloud of each act in the play.
The words in each cloud represent various characteristics, including the key characters and themes of each act. We can see how important each character is to an act. The surrounding words in each cloud also help us visualize the transition from Romeo and Juliet being a happy couple to everything ending in death, a tragedy.
The next visual allows us to have a better idea of how important each character is in a scene. Each graph is a separate act, and we can view how many lines each character says in a given scene. We are now able to see more about the play, and see who is the star in each scene of the play.
The final two visuals show the number of lines the twelve most prominent characters speak in each act. The first visual contains the information of the top six characters with the most words.
The following plot shows the results for the next six characters with the most speaking time.
Further Study
Further Study
In the future, it would be interesting to conduct more analyses on the text to better understand the characters and the overall formatting of the play. It would also be nice to incorporate machine learning, so that we can figure out who each character is speaking to. Using machine learning, given a line, we could identify who said it.
Keywords
Keywords
◼
Text segmentation
◼
Text processing
◼
Shakespeare
◼
Plays
◼
XML-formatting
◼
TEI-formatting
◼
Natural Language Processing
Acknowledgements
Acknowledgements
I'd like to thank my mentor, Aaron Enright, for his guidance and support throughout the entire program. I'd also like to thank everyone at camp for making all of this possible.