Chats Analysis
Chats Analysis
In[]:=
ClearAll["Global`*"]
Before Using!!
Before Using!!
◼
Make sure your data has the 24h format for the messages. You can get it by changing your phone’s clock to 24h and the export the chat again.
◼
Make sure your .txt file is in the same folder of this notebook
Data Processing
Data Processing
Get Data
Get Data
Use the directory of the notebook to store your WhatsApp file.
In[]:=
NotebookDirectory[]
In[]:=
rawData=Import[StringJoin[NotebookDirectory[],"chatFile.txt"]];
Process
Process
We have two types of information:
◼
Messages
◼
“Events” : When someone is added or removed from a group, or when its name is changed
We want to create two lists, one containing the messages and another containing the event. Let’s go!
Get dates and time
In[]:=
datesAndTime=StringSplit/@StringCases[rawData,"["~~x:(DigitCharacter..~~"/"~~DigitCharacter..~~"/"~~DigitCharacter..~~Whitespace~~DigitCharacter..~~":"~~DigitCharacter..~~":"~~DigitCharacter..)~~"]"x]
Get messages
In[]:=
messages=StringSplit[rawData,"["~~DigitCharacter..~~"/"~~DigitCharacter..~~"/"~~DigitCharacter..~~Whitespace~~DigitCharacter..~~":"~~DigitCharacter..~~":"~~DigitCharacter..~~"]"~~Whitespace..];
We should have the same length for datesAndTime and messages, since we used the same criteria to split the texts. Let’s compare them:
In[]:=
Length@messagesLength@datesAndTime
Separate Sender from message
In[]:=
messagesAndSender=StringSplit[#,":",2]&/@StringReplace[messages,"\n"" "]
Now, let’s put it all together:
In[]:=
together=Flatten/@Transpose[Append[Transpose@datesAndTime,messagesAndSender]]
We can see how many messages and events (when someone was added or the group name was changed) we have:
In[]:=
Tally[Length/@together]
Let’s divide messages from events
In[]:=
messagesList=Select[together,Length@#>3&];messagesList=Rest@messagesList;(*<--COMMENTIFNOTGROUP*)eventsList=Select[together,Length@#==3&];(*Last"corrections"*)(*Wewantdateasyy/mm/ddandwehavedd/mm/yy*)messagesList[[All,1]]=Reverse/@Map[ToExpression@#&,StringSplit[#,"/"]&/@messagesList[[All,1]],{2}];(*timeash,m,s*)messagesList[[All,2]]=Map[ToExpression@#&,StringSplit[#,":"]&/@messagesList[[All,2]],{2}];
When the group is created, the group “sends a message” saying that the group is being create. If this is the case, we must account for this. This is the purpose of the Rest function above. If you are working with a chat, you should comment it.
In[]:=
Tally@eventsList[[All,3]]//Column
Play
Play
Member’s participation
Member’s participation
In[]:=
PieChart[First/@GroupBy[Tally[messagesList[[All,3]]],FirstLast],ChartStyle"Pastel",ChartLabelsMap[StringRiffle[#,": "]&,Map[ToString,Tally[messagesList[[All,3]]],{2}]]]
Time Analysis
Time Analysis
Time Line
Time Line
Create {Time, # messages (from anyone)}
In[]:=
messPerDayRaw={DateObject@#[[1]],#[[2]]}&/@Tally[messagesList[[All,1]]];
Let’s take the days in which no conversation took place
In[]:=
missingDays=Complement[Range[QuantityMagnitude[Max[messPerDayRaw[[All,1]]]-Min[messPerDayRaw[[All,1]]]]+1],QuantityMagnitude[messPerDayRaw[[All,1]]-Min[messPerDayRaw[[All,1]]]]+1];
Now, let’s fill the missing days and assign the value of 0, since no message were sent during this time.
In[]:=
messPerDay=TimeSeriesResample[messPerDayRaw,"Day"];messPerDay[[missingDays,2]]=0;
In[]:=
DateListPlot[messPerDay,DateTicksFormat{"Month","/","Year"},PlotRangeAll,LabelStyleDirective[FontSize12],FrameLabel{None,"[Messages / day]"},PlotTheme"Detailed",ImageSizeMedium]
Let’s smooth this plot by applying a moving average of 7 days. These will reflect the messages per week
In[]:=
DateListPlot[MovingAverage[TimeSeries[messPerDay],7],DateTicksFormat{"Month","/","Year"},PlotRangeAll,LabelStyleDirective[FontSize12],FrameLabel{None,"[Messages / day]"},PlotLabel"Moving Average of 7 days",PlotTheme"Detailed",ImageSizeMedium]
At what time do we communicate?
At what time do we communicate?
In[]:=
messPerHourMembers=Map[First,GroupBy[messagesList[[All,{3,2}]],FirstLast],{2}];
In[]:=
Histogram[Values@messPerHourMembers,ChartLegendsKeys@messPerHourMembers,ChartLayout"Stacked",Ticks{Range[0,23],Automatic},FrameTrue,FrameLabel{"Hour of the Day","Frequency"}]
Funny Words
Funny Words
Used by all
Used by all
In[]:=
funnyWordsAll={"lol","hahah","sos"}
In[]:=
Length@StringCases[ToLowerCase@rawData,#]&/@funnyWordsAll
In[]:=
dataFunnyWordsAll=SortBy[{#,Length@StringCases[rawData,#]}&/@funnyWordsAll,Last]
In[]:=
BarChart[Last/@dataFunnyWordsAll,ChartLabelsPlaced[First/@dataFunnyWordsAll,Below,Rotate[#,Pi/2.4]&],PlotLabel"Funny words Frequency",ChartStyle"DarkRainbow"]
Individual word analysis
Individual word analysis
Here, you can see the number of words (different from the number of messages above) sent by each person
In[]:=
wordsPerMember=TextWords/@Map[StringRiffle[#," "]&,GroupBy[messagesList[[All,-2;;]],FirstLast]];
In[]:=
PieChart[Length/@wordsPerMember,ChartStyle"Pastel",PlotLabel"Words Sent",ChartLabelsKeyValueMap[StringRiffle[{#1,#2},": "]&,Map[ToString,Length/@wordsPerMember]]]
Word cloud per person!
Word cloud per person!
Here, I create a list of connectors in Spanish, I want to exclude them from the analysis.
In[]:=
connectorsSpanishAllLower=StringSplit["la el lo los les las un una es ser te o en si de a pero es ver gif omitida que me al qué mi este esta ese unos unas le y omitido con ya por nos del mas más para esto se su como ni"];connectorsSpanishFirstUpper=StringReplacePart[#,ToUpperCase@StringPart[#,1],{1,1}]&/@connectorsSpanishAllLower;connectorsSpanish=Join[connectorsSpanishAllLower,connectorsSpanishFirstUpper];
To delete these cases, I will find their position and then drop them.
In[]:=
{wordsPerMember//Keys,Range@Length@wordsPerMember}//Transpose
Take a look into the frequency of the words used!
In[]:=
ReverseSortBy[Tally@Select[wordsPerMember[[1]],!MemberQ[connectorsSpanish,#]&],Last][[1;;10]]
To delete these cases, I will find their position and then drop them:
In[]:=
WordCloud@Select[wordsPerMember[[7]],!MemberQ[connectorsSpanish,#]&]