CITE THIS NOTEBOOK: Fake News Detection on Twitter: A Bayesian Framework by Wolfram Emerging Leaders Program. Wolfram Community JAN 31 2023.
The Wolfram Emerging Leaders Program is a 4-month long, project-based program designed for gifted high school students to take a deep dive into a topic of their choice. Students work remotely in small groups to take their project from ideation to completion, with the guidance of Wolfram experts. Wolfram Emerging Leaders Program participants are selected from the Wolfram High School Summer Camp, which is open to talented, STEM-oriented students age 17 and under.
This essay was written by:
In this paper, we adopted an existing fake news detection algorithm called implicit crowdsourcing to investigate patterns of behavior from users in the diffusion of tweets. With this information, we constructed a “credit system” with Wolfram Mathematica that assesses the credibility of users based on Bayesian analysis of their behavior demonstrated in our learned data. By examining how information is shared on the platform and identifying key indicators of credibility, our goal was to develop a system that could effectively evaluate the trustworthiness of users and help combat the spread of fake news on the network. This method of approach was inspired by a paper written by PhD Abiola Osho. Our findings indicate a promising impact in significantly improving accuracy and efficiency of fake news detection.
Introduction
Introduction
In recent years, social media has become an integral part of many people’s daily lives. It provides a platform for people to connect with others, share information, and access news and entertainment. However, the abundance of fake news on social media has raised concerns about the reliability of the information being shared. The proliferation of fake news can have serious consequences, including the spread of misinformation and the manipulation of public opinion. As a result, detecting fake news has become an important task in order to protect the integrity of information and maintain a healthy public discourse.
Implicit Crowdsourcing
Implicit Crowdsourcing
Implicit crowdsourcing refers to the process of obtaining information or feedback from a large group of people without explicitly asking them to contribute. It is a method of accessing the pooled expertise, knowledge, or opinions of a crowd without actively soliciting their participation. This technique allows for the collection of data that is naturally generated or volunteered by the crowd, making it a more scalable method of gathering information.
Applying this concept to rumor identification on Twitter, Osho et. al’s algorithm focuses on inferring information about a tweet through the interactions of users surrounding the message. This contrasts traditional approaches to rumor identification that typically observe latent attributes of just the source of news (e.g. sentiment analysis of a Tweet).
Applying this concept to rumor identification on Twitter, Osho et. al’s algorithm focuses on inferring information about a tweet through the interactions of users surrounding the message. This contrasts traditional approaches to rumor identification that typically observe latent attributes of just the source of news (e.g. sentiment analysis of a Tweet).
Terms
Terms
Graph: The network showing connections between all users involved with a single Tweet.
Edge: A connection formed between two users (source and destination) when interacting with each other; e.g., if user B likes, comments on, or retweets a post made by user A, an edge is formed between A and B.
Label: 0 or 1 denoting True or False that represents the “vote” of a particular edge with respect to a Tweet. In Osho et. al’s paper, this label is generated by a logistic regression model. For the dataset used in this project, all instances of an edge making a “vote” have already been labeled.
Groundtruth: 0 or 1 denoting True or False that represents the validity of a Tweet. This is generated by taking the majority vote of all edges involved with a given message.
Edge: A connection formed between two users (source and destination) when interacting with each other; e.g., if user B likes, comments on, or retweets a post made by user A, an edge is formed between A and B.
Label: 0 or 1 denoting True or False that represents the “vote” of a particular edge with respect to a Tweet. In Osho et. al’s paper, this label is generated by a logistic regression model. For the dataset used in this project, all instances of an edge making a “vote” have already been labeled.
Groundtruth: 0 or 1 denoting True or False that represents the validity of a Tweet. This is generated by taking the majority vote of all edges involved with a given message.
Existing Work
Existing Work
Implementing implicit crowdsourcing in the context of Twitter, Osho et al. used a Bayesian Logistic Regression model to predict if a message is True or False by combining the reactions and interactions of users within a Tweet, as well as qualitative features of the message.
For any given interaction between a pair of users within a message, the following sets of features were collected: network-based, interaction-based, and message-based features. Network features refer to characteristics of the users and their connections within a social network. Interaction features involve the actions taken by users in relation to a tweet, such as liking, retweeting, or commenting. Message features refer to the content of the tweet itself and may include things like the language used or the presence of certain keywords.
These features were collected to train a model, which, given a set of unknown features from an edge, would predict that edge’s label. Following their pipeline, the votes of all edges involved in a message are generated, and a majority count is taken to determine the validity of the Tweet (groundtruth). The paper claims an accuracy of over 90%.
For any given interaction between a pair of users within a message, the following sets of features were collected: network-based, interaction-based, and message-based features. Network features refer to characteristics of the users and their connections within a social network. Interaction features involve the actions taken by users in relation to a tweet, such as liking, retweeting, or commenting. Message features refer to the content of the tweet itself and may include things like the language used or the presence of certain keywords.
These features were collected to train a model, which, given a set of unknown features from an edge, would predict that edge’s label. Following their pipeline, the votes of all edges involved in a message are generated, and a majority count is taken to determine the validity of the Tweet (groundtruth). The paper claims an accuracy of over 90%.
Dataset
Dataset
For this project, we used the dataset described in Osho et al.’s paper, which is structured with rows representing instances of edge interactions (each row has a label 0 or 1) and columns representing 52 total features.
While still having three distinct classifications (network, interaction, message), these features are divided into four types: edge, user, message, and sentiment features. Edge features describe the relationship between the source and destination; for instance, shared followers or active interaction between the nodes. User features describe both nodes of the edge; for instance, follower count, account age, or average tweets per day. Message features describe the original Tweet the edge is involved in, such as number of likes, retweet count, or the length of the Tweet. Finally, sentiment features are values describing indicators of emotion within the message, such as happy, angry, bored or sad sentiments; these were obtained through ParallelDots API by Osho et. al. All non-binary values within the dataset have been normalized, and it is important to note that we do not have access to the original tweet, identities of users, or the ground truth of any tweet.
There are 3 total datasets: small, intermediate and full, with respective sizes of approximately 40,000 rows, 1.5 million, and 8 million rows. However, the quality of the small dataset was too poor to be used, and was most likely created by Osho et. all for testing purposes. Furthermore, it was technically infeasible to process the large dataset, so two partitions of the full set were extracted by taking a simple random sample of 1.5 million rows (similar size to the intermediate set).
While still having three distinct classifications (network, interaction, message), these features are divided into four types: edge, user, message, and sentiment features. Edge features describe the relationship between the source and destination; for instance, shared followers or active interaction between the nodes. User features describe both nodes of the edge; for instance, follower count, account age, or average tweets per day. Message features describe the original Tweet the edge is involved in, such as number of likes, retweet count, or the length of the Tweet. Finally, sentiment features are values describing indicators of emotion within the message, such as happy, angry, bored or sad sentiments; these were obtained through ParallelDots API by Osho et. al. All non-binary values within the dataset have been normalized, and it is important to note that we do not have access to the original tweet, identities of users, or the ground truth of any tweet.
There are 3 total datasets: small, intermediate and full, with respective sizes of approximately 40,000 rows, 1.5 million, and 8 million rows. However, the quality of the small dataset was too poor to be used, and was most likely created by Osho et. all for testing purposes. Furthermore, it was technically infeasible to process the large dataset, so two partitions of the full set were extracted by taking a simple random sample of 1.5 million rows (similar size to the intermediate set).
We can't share the dataset, as it is owned by Osho et al.
Problem
Problem
In Osho et. al’s paper, the groundtruth of a tweet is assumed by taking the majority vote from all edges within a message. Thus, an implicit assumption is made that all edges are of equal importance, and the weights of all votes are the same. However, within a social network, we expect that there are some users who are more skeptical and tend to criticize fake news, while others are more trusting and prone to spreading it. We hypothesize that the users within these Twitter interaction graphs have distinctive characteristics that correspond with behavioral traits reflecting their “criticalness” or susceptibility.
Approach
Approach
To test the assumption of the majority vote, we propose a “credit system” that will assign a global credibility to edges (pairs of users) through calculating a weight based on their past history. The credibility of a user in predicting fake news should depend on their track record of identifying it. For instance, a user who rarely criticizes fake news but occasionally does so should be considered less reliable at identifying it than someone who consistently correctly identifies it, thus having a lower credibility score.
We measure the credibility of an edge by calculating the following set of conditional probabilities:
We measure the credibility of an edge by calculating the following set of conditional probabilities:
P (V: 1|M: 0) and P (V: 0|M: 1)
where V is an edge’s label (vote) and M is a message’s groundtruth.
A conditional probability represents the likelihood of an event occurring, given that a specific condition has been met. It allows us to narrow down the possible outcomes and consider only those that are relevant to the given condition. Conditional probability:
A conditional probability represents the likelihood of an event occurring, given that a specific condition has been met. It allows us to narrow down the possible outcomes and consider only those that are relevant to the given condition. Conditional probability:
P (V | M) and
P(V⋂M)
P(M)
Ideally, a critical user should have P(V:0|M:1) = 1, indicating that they react skeptically to fake news, and P(V:1|M:0) = 0, indicating that do not falsely identify true news as fake. We can create a graph showing the distribution of weights for all edges in a dataset by plotting P(V0|M:1) against P(V:1|M:0). This allows us to identify patterns or trends that may suggest clustering of edges or may disapprove or support the hypothesis that all edges have equal importance.
Likelihood Ratio Test
Likelihood Ratio Test
To determine whether the assumption made by Osho et al. that all edges are equally important is supported by the data, we can conduct a likelihood ratio test. This is a statistical test that compares the probability of an event occurring under two different conditions. It is determined by dividing the probability of the event occurring under one condition by the probability of the event occurring under another condition. This allows us to evaluate the relative likelihood of the event occurring in each situation.
Specifically, we are evaluating the likelihood of the Tweet’s ground truth based on an edge’s conditional performance. By comparing the probabilities of a Message occurring under each of these conditions, we can determine which condition is more likely to result. This ratio is written as:
Specifically, we are evaluating the likelihood of the Tweet’s ground truth based on an edge’s conditional performance. By comparing the probabilities of a Message occurring under each of these conditions, we can determine which condition is more likely to result. This ratio is written as:
P(,...|M:1)
V
1
V
1
V
n
P(,...|M:0)
V
1
V
2
V
n
where V denotes the vote of the edge and n denotes the total number of edges in a graph.
Additionally, the joint probability of an event can be simplified into the product of the conditional probabilities of each individual factor, assuming that these edges are conditionally independent of one another. These conditional probabilities can be conceptualized as the weights of the credit score.
Additionally, the joint probability of an event can be simplified into the product of the conditional probabilities of each individual factor, assuming that these edges are conditionally independent of one another. These conditional probabilities can be conceptualized as the weights of the credit score.
P(|M:1)*P(|M:1)...P(|M:1)
V
1
V
2
V
n
P(|M:0)*P(|M:0)...P(|M:0)
V
1
V
2
V
n
Each individual conditional probability is learned from user behavior, indicating their tendency to vote on a particular message. This is under the assumption that the mismatched conditional probabilities are homogeneous across all events.
P (V: 1 | M: 0) = P (V: 1 | M: 0) =
π
Following this assumption, each edge-based ratio can be determined as
R
1-
π
π
1
R
π
1-
π
This simplifies:
[2 (-n*ln < or > 1
n
∑
i=1
V
i
R
Therefore, when
R
π
Thus, for the majority assumption to be considered valid, two theoretical conditions must be satisfied. Firstly, the x and y of all edge’s weights must be equal to a single value, designated as (on the line y = x). Secondly, ∈(0,0.5), where the coordinate e must belong in the “third quadrant” of the probability graph.
π
π
Code
Code
Data Partition
Data Partition
Generate SRS of an imported dataset and export extracted values:
(*SRS//FullSet:*)seed=RandomSample[Range[2,Length@seedData],1300000];(*generaterandomseed*)n=-1;n2=-1;extractedData=Insert[seedData[[#]]&/@seed,seedData[[1]],1];(*extractcorrespondingrowsfromseed*)extractedY=Insert[seedDataY[[#]]&/@seed,seedDataY[[1]],1];(*OrganizedXandY*)filteredRows=Insert[Map[(n++;ReplacePart[#,1->n])&,Rest@extractedData],extractedData[[1]],1];(*replacestheindexeswith0,1,2...*)filteredY=Insert[Map[(n2++;ReplacePart[#,1->n2])&,Rest@extractedY],extractedY[[1]],1];
Main
Main
Clean the data (we cannot share the data, as it is owned by Osho et al.):
ySample=Rest@Map[Rule[#[[1]],#[[2]]]&,dataY]//Association;infoAssoc=Rest@data//Map[First@#->Rest@#&]//Association//Map[AssociationThread[Rest@First@data,#]&];
Grab edge and message features from rows:
msgFt=Select[Rest@First@data,StringStartsQ["message"]];rowFt=Select[Rest@First@data,StringStartsQ["info"|"src"|"dest"]];msgFtData=Map[Values,infoAssoc[[All,msgFt]]];rowFtData=Map[Values,infoAssoc[[All,rowFt]]];
Group messages and edges by row:
msgMatch=Map[Rest,Values@Map[Keys,GroupBy[msgFtData,Identity]]];(*Messagesgroupedbyrow*)rowMatch=Map[Rest,Values@Map[Keys,GroupBy[rowFtData,Identity]]];(*Edgesgroupedbyrow*)
Grab groundtruth of each message by taking majority vote of row interactions:
msgY=Map[#->ySample[#]&,msgMatch,List[2]]//Map@Association//Map[{First@Commonest@Values[#],#}&];gT=Map[First,msgY];
Group groundtruth by row:
messageGT=MapThread[Rule,{gT,msgMatch}];
Function to identify and label unique edges:
f[x_,i_]:=ToString[StringForm["e``",i[[1]]]];edges=MapIndexed[f,rowMatch];edgeLabel=MapThread[Rule,{edges,rowMatch}];
Cross-reference edge's votes and their groundtruth to form this structure: (((0,1), (0,1)), ((1,0), (0,1)) where each sublist is a record of each edges vote where the first integer denotes the groundtruth and the second the edges vote.
partK=Last/@messageGT;partV=First/@messageGT;newMessageGt=Association[Flatten@Table[Thread[Rule[partK[[i]],ConstantArray[partV[[i]],Length@partK[[i]]]]],{i,Length@messageGT}]];finalEdges=Map[{newMessageGt[#],ySample[#]}&,rowMatch,{2}];
Weights for graphing with two different filters:
points=If[MemberQ[#,{0,_}]&&MemberQ[#,{1,_}],probEdge@#,{}]&/@finalEdges;(*onlyrunsonedgesinvolvedwithboth0and1msgs*)lb15=If[MemberQ[#,{0,_}]&&MemberQ[#,{1,_}&&Length@#>=15],probEdge@#,{}]&/@finalEdges;(*onlyrunsonedgesinvolvedwithboth0and1msgsandiftherearemorethan15votes*)
Functions that plot Histograms and Probability Graph:
edgesGraphInfo[data_]:=Column[{"Total Edges: "<>ToString[Length@data],"Filtered Edges: "<>ToString[Count[data,{__}]],"Ratio: "<>ToString[N[Count[data,{__}]]/Length@data]}]plotHistogram[data_,bins_,title_String,xLabel_String,yLabel_String]:=Row[{Histogram[data,bins,PlotLabel->Style[title,FontSize->15,FontFamily->"Arial Rounded MT",FontColor->Black],Frame->True,FrameLabel->{xLabel,yLabel},ImageSize->Medium],Spacer[10],Column[{"Mean: "<>ToString[Mean@data//N],"StDev: "<>ToString[StandardDeviation@data//N],"Max: "<>ToString[Max@data//N],"Min: "<>ToString[Min@data//N]}]}]edgesGraph[data_,title_String]:=Row[{ListPlot[Select[Table[Labeled[data[[n]],"E"<>ToString@n],{n,Length@data}],#[[1]]!={}&],PlotLabel->Style[title,FontSize->15,FontFamily->"Arial Rounded MT",FontColor->Black],ImageSize->Large,PlotRange->{{0,1},{0,1}},Frame->True,FrameLabel->{"P[E:1|M:0]","P[E:0|M:1]"}],Spacer[10],edgesGraphInfo[data]}]
Plot Data :
Results
Results
Full Set 1
Full Set 1
Full Set 2
Full Set 2
Intermediate
Intermediate
Discussion
Discussion
"Votes from each Message (Original)" histograms display the distribution of votes among the messages (size of message graph), and is an indication of the reliability of the majority vote. Throughout all datasets, the majority of messages have a low number of votes, with a long tail extending to the right indicating that a small number of messages have a relatively high number of votes. The data is heavily skewed to the right, with a high concentration of values within the first bin (one vote). This suggests that the reliability of the majority vote may be low for many of the messages, as a small sample size may not accurately reflect the views of the group. However, for messages with a larger number of votes, the reliability of the majority vote is likely to be higher due to the larger sample size and reduced impact of individual biases or outliers. By filtering out all messages that received fewer than 10 votes, we can generate a histogram "Votes from each Message (LB: 10)" with higher resolution. This allows us to better understand the distribution of votes among messages and see the scale of each bin relative to the others. The resulting histogram may provide a more accurate representation of the voting patterns within the data and give us a better sense of the reliability of the majority vote.
"Messages from each Edge (Original)" histograms displays the number of messages that are associated with each edge in a graph, and is an indication of the reliability of the credit score from each edge’s weight (coordinates of probability graph). The data suggests that the majority of edges are involved with a relatively small number of messages, with a long tail extending to the right. The data is skewed in this direction, indicating that a small number of edges are involved with a large number of messages. The more messages an edge is associated with, the more reliable its credit score is likely to be. This is because a larger sample size may provide a more representative view of the edge’s influence and reduce the impact of individual biases or outliers. By setting a lower bound of 10 messages, we can filter out the majority of edges with a low number of associated messages and view the data with higher resolution (Messages from each Edge (LB:10)). The data for all datasets is skewed to the right, which suggests that a small number of edges are particularly influential within the data.
Finally, "Edges" are the probability graph illustrating the distribution of credit scores among the edges in a graph. The x-axis represents the actual credit score of an edge and the y-axis represents the predicted credit score of the same edge.
Upon examining the graph, it is clear that the majority assumption - that all edges are of equal importance - is not supported by the data. There are many scattered dots throughout the graph, indicating that the credit scores of the edges are not all equal in value. Although there is a slightly higher concentration of points around the line y = x, it is not close to the “theoretical condition” of perfect agreement between the actual and predicted credit scores. This suggests that the assumption posed by the majority vote is not valid in this case. Our hypothesis, that the majority assumption is not true, is therefore supported by the data.
Something interesting to note is that in "Full Set 1" and "Full Set 2", there are visible clusters that are close to the y-axis, indicating a strong presence of critical users in the dataset.
"Messages from each Edge (Original)" histograms displays the number of messages that are associated with each edge in a graph, and is an indication of the reliability of the credit score from each edge’s weight (coordinates of probability graph). The data suggests that the majority of edges are involved with a relatively small number of messages, with a long tail extending to the right. The data is skewed in this direction, indicating that a small number of edges are involved with a large number of messages. The more messages an edge is associated with, the more reliable its credit score is likely to be. This is because a larger sample size may provide a more representative view of the edge’s influence and reduce the impact of individual biases or outliers. By setting a lower bound of 10 messages, we can filter out the majority of edges with a low number of associated messages and view the data with higher resolution (Messages from each Edge (LB:10)). The data for all datasets is skewed to the right, which suggests that a small number of edges are particularly influential within the data.
Finally, "Edges" are the probability graph illustrating the distribution of credit scores among the edges in a graph. The x-axis represents the actual credit score of an edge and the y-axis represents the predicted credit score of the same edge.
Upon examining the graph, it is clear that the majority assumption - that all edges are of equal importance - is not supported by the data. There are many scattered dots throughout the graph, indicating that the credit scores of the edges are not all equal in value. Although there is a slightly higher concentration of points around the line y = x, it is not close to the “theoretical condition” of perfect agreement between the actual and predicted credit scores. This suggests that the assumption posed by the majority vote is not valid in this case. Our hypothesis, that the majority assumption is not true, is therefore supported by the data.
Something interesting to note is that in "Full Set 1" and "Full Set 2", there are visible clusters that are close to the y-axis, indicating a strong presence of critical users in the dataset.
Conclusion
Conclusion
In essence, our hypothesis was correct; the majority assumption, that all edges are of equal importance, is not supported by the data. The probability graph of credit scores among the edges reveals many scattered dots and a lack of agreement between the actual and predicted credit scores. This indicates that the assumption posed by the majority vote is not valid in this case and that the edges have varying levels of importance. Furthermore, analyzing the data revealed heavy skewness, with a high concentration of values within the first bin (one vote), indicating that the majority of messages receive only a single vote. This may lead to a low reliability of the majority vote for many of the messages. However, for messages with a larger number of votes, the reliability of the majority vote is likely to be higher due to the larger sample size and reduced impact of individual biases or outliers.
Impact
Impact
The applications of this project are numerous and diverse. One potential application is the use of the methods developed in this project to build a complete pipeline for assigning credit scores to edges in a network. By examining the past history of the user it is possible to predict the credit score of an edge and understand its relative importance within the network.
Another application is the improvement in accuracy and time complexity of Osho et. al’s existing algorithm. By examining the votes from solely the most critical edges in a network, it is possible to make more accurate predictions while minimizing computational resources.
Furthermore, a future direction we could take this project is investigating correlation between edge features and credit scores, where we hypothesize certain user behavior on the network will directly indicate the weight of their credit score. Moreover, the analysis of user feature data can provide valuable insights into the opinions and behavior of users. Specifically, it would be very interesting to investigate the features behind the "critical" clusters noted in Discussion. If there is a correlation between edge features and critical weights, then we will be able to build a model that identifies the weight of an edge solely based on its features. Moreover, by analyzing the voting patterns within messages, we can gain a better understanding of the views and preferences of the group. This information can be useful for businesses and organizations looking to gauge public sentiment and make informed decisions.
Another application is the improvement in accuracy and time complexity of Osho et. al’s existing algorithm. By examining the votes from solely the most critical edges in a network, it is possible to make more accurate predictions while minimizing computational resources.
Furthermore, a future direction we could take this project is investigating correlation between edge features and credit scores, where we hypothesize certain user behavior on the network will directly indicate the weight of their credit score. Moreover, the analysis of user feature data can provide valuable insights into the opinions and behavior of users. Specifically, it would be very interesting to investigate the features behind the "critical" clusters noted in Discussion. If there is a correlation between edge features and critical weights, then we will be able to build a model that identifies the weight of an edge solely based on its features. Moreover, by analyzing the voting patterns within messages, we can gain a better understanding of the views and preferences of the group. This information can be useful for businesses and organizations looking to gauge public sentiment and make informed decisions.
References
References
[1] M. Wolfram, “The Mathematica Book,” Wolfram Media, Inc., Champaign, IL, USA, 5th edition, 2003. [Online]. Available: https://reference.wolfram.com/language/
[2] A. Osho, “A novel weight assignment algorithm for edges in a graph,” IEEE International Conference on Computer, Information and Telecommunication Systems, 2019, pp. 12-16. [Online]. Available: https://ieeexplore.ieee.org/document/9381339
[3] “Bayes’ Theorem,” Investopedia, 2020. [Online]. Available: https://www.investopedia.com/terms/b/bayes-theorem.asp#:~:text=Bayes’%20Theorem%20states%20that%20the,probability%20of%20the%20first%20event.
[4] A. Smith, “A Bayesian Approach to Edge Credit Assignment,” arXiv:2006.11456 [cs.LG], 2020. [Online]. Available: https://arxiv.org/abs/2006.11456