CITE THIS NOTEBOOK: FinBERT and FinRoBERTa models for financial sentiment analysis by Jürgen Kanz. Wolfram Community JAN 18 2023.
CITE THIS NOTEBOOK: FinBERT and FinRoBERTa models for financial sentiment analysis by Jürgen Kanz. Wolfram Community JAN 18 2023.
Introduction
Introduction
FinBert is an open source Natural Language Processing (NLP) model that has been trained on Financial data and outperforms almost all other NLP techniques for financial sentiment analysis.
Financial data sentiment analysis is more complicated than normal use cases. Let’s take a look at an example:
“Although the global market is experiencing a downturn, food companies are doing well thanks to Covid.”
Normal NLP techniques will not be able to determine whether the above sentence is good news or has a positive sentiment for food companies. This is because normal NLP techniques, like word2vec, look at each word separately and do not provide context for the words. In order to understand the sentiment of the above sentence, we would need to be aware of the context.
Context awareness is enabled by BERT for sentences. BERT is the acronym for Bidirectional Encoder Representation from Transformer. It is one of the most popular state-of-the-art text embedding models published by Google.
Financial data sentiment analysis is more complicated than normal use cases. Let’s take a look at an example:
“Although the global market is experiencing a downturn, food companies are doing well thanks to Covid.”
Normal NLP techniques will not be able to determine whether the above sentence is good news or has a positive sentiment for food companies. This is because normal NLP techniques, like word2vec, look at each word separately and do not provide context for the words. In order to understand the sentiment of the above sentence, we would need to be aware of the context.
Context awareness is enabled by BERT for sentences. BERT is the acronym for Bidirectional Encoder Representation from Transformer. It is one of the most popular state-of-the-art text embedding models published by Google.
Domain Specific Fine-Tuning of BERT
Domain Specific Fine-Tuning of BERT
Many domain specific models have emerged using BERT as the base and are being used for NLP tasks. Some of them are: FinBERT for Finance, BioBERT for Biomedical, VideoBERT for Video captioning categorization, ClinicalBERT for hospitals, and many more continue to evolve.
BERT, RoBERTa and the Wolfram Neural Net Repository
BERT, RoBERTa and the Wolfram Neural Net Repository
BERT is available in the Wolfram Neural Net Repository [1]. Furthermore, Wolfram provides domain specific fine-tuned versions of BERT like BioBERT, DistilBERT, Pre-trained Distill BERT, and SciBERT [2].
FinBERT is currently not available as a Wolfram Resource.
“The better BERT” is RoBERTa. RoBERTa is a robustly optimized BERT pretraining approach [3]. While BERT is trained on BookCorpus and Wikipedia Data, it is RoBERTa that is already trained on BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories Datasets.
Consequently, the author decided to train BERT and RoBERTa on the enhanced Financial Phrase-bank.
FinBERT is currently not available as a Wolfram Resource.
“The better BERT” is RoBERTa. RoBERTa is a robustly optimized BERT pretraining approach [3]. While BERT is trained on BookCorpus and Wikipedia Data, it is RoBERTa that is already trained on BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories Datasets.
Consequently, the author decided to train BERT and RoBERTa on the enhanced Financial Phrase-bank.
Datasets
Datasets
Financial PhraseBank
Financial PhraseBank
For training of the networks, the sentiment analysis dataset used is the Financial Phrase-bank [4] from Malo et al. (2014) [5]. The “Financial Phrase-bank” consists of, 4845 English sentences selected randomly from financial news found on the LexisNexis database. These sentences then were annotated by 16 people with background in finance and business. The annotators were asked to give labels according to how they think the information in the sentence might affect the mentioned company stock price. The dataset also includes information regarding the agreement levels on sentences among annotators. The distribution of agreement levels and sentiment labels can be seen on the following table. Table 1: Distribution of sentiment labels and agreement levels in Financial Phrase-bank
FiQA - Financial Opinion Mining and Question Answering
FiQA - Financial Opinion Mining and Question Answering
On top, 997 selected sentences from the “FiQA - Financial Opinion Mining and Question Answering” challenge [6] have been used for training purposes.
Validation Measurements
Validation Measurements
On the repository webpage [10] in section “Train a classifier model with the subword embeddings”, we can find information about Validation Measurements for different net models.The following results are based on calculations with “BERT Trained on BookCorpus and Wikipedia Data”, “GloVe 300 - Dimensional Word Vectors Trained on Wikipedia and Gigaword 5 Data”, and finally “RoBERTa Trained on BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories Datasets”. As input the FinancialPhrase-bank was used.It becomes clear, that the given BERT net model does not deliver the best results regarding Loss and ErrorRate. Nevertheless, BERT is more or less the current standard transformer, which can be observed in the Python [7] [8] and Matlab community [9].The best results are provided by RoBERTa regarding Loss and Error Rate. Therefore, I decided to train both BERT versions simultaneously.
Training
Training
The training process itself is not part of this notebook. Nevertheless, some general remarks:
◼
The Wolfram resource “BERT Trained on BookCorpus and Wikipedia Data” [10] is used to create FinBERT.
◼
To become FinRoBERTa, the Wolfram resource “RoBERTa Trained on BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories Datasets”[11] is used.
◼
A balanced training set has been used. Which means that there are the same number of positive, neutral, and negative sentiments applied.
◼
The training has been completed with sentences with and without sentence punctuation.
Load FinBERT and FinRoBERTa
Load FinBERT and FinRoBERTa
Import FinBERT and FinRoBERTa from the WolframCloud
Import FinBERT and FinRoBERTa from the WolframCloud
In[]:=
finbert=Import["https://www.wolframcloud.com/obj/juergen.kanz/FinBERT.WLNET"]
Out[]=
NetChain
In[]:=
finroberta=Import["https://www.wolframcloud.com/obj/juergen.kanz/FinRoBERTa.WLNET"]
Out[]=
NetChain
Import local copies of FinBERT and FinRoBERTa
Import local copies of FinBERT and FinRoBERTa
Let’s assume that this notebook, as well as FinBERT and FinRoBERTa are stored in the same directory.
In[]:=
SetDirectory[NotebookDirectory[]]
Out[]=
e:\Users\juerg\Documents\Mathematica\NLP
In[]:=
finbert=Import["FinBERT.WLNET"]
Out[]=
NetChain
In[]:=
finroberta=Import["FinRoBERTa.WLNET"]
Out[]=
NetChain
Sentiment Analysis
Sentiment Analysis
A simple example to show the application of both networks.
In[]:=
sentence="The company said that sales in the three months to the end of March slid to EUR86 .4 m US$ 113.4 m from EUR91 .2 m last year";
In[]:=
sentimentFinBERT=finbert[sentence]
Out[]=
negative
In[]:=
sentimentFinRoBERTa=finroberta[sentence]
Out[]=
negative
The answers are acceptable for both Sentiment Analysis.
Comparison with other BERTs
Comparison with other BERTs
Let’s look at a prepared dataset for this step. To avoid further file import, the dataset is compressed.
In[]:=
ds=Uncompress["1:eJztXFtz28YZVe+Z6Uwv05k8f51m6hfJI8lJbKvtpLo6cixLI8p23tolsCA2XGCR3QUpZPq3+95zFqRsJbYjy4ouVl84XBIAwcPv+875Lss/D91h8YuFhYXwGzxsqaiCjsXf+cIv8fDEhFj8iqvf4mE9BJcZFY2ri5/PDzlsrR58hCcDXUddZ3rAs4/M+MiNJXN1MLn2QZ64One1qDoXF0vtxbosXSlI4byUWuXftspHHPu9a/8OTw66WOLsHVNvbB8eDXivtW6jV/Y1B++paNXwDAf/qT+41BXuJFNnOOPjH55x6HiOOnXSuyL2D2JvQq07ydoYROUSGl3nph4Jv7bK9NABTlWZXEbeTfnG0HWZi3FNXgwe/zhkH6XbG+FWJvrHMXvL0W8C7S2nvAW1U2e9K2y7NDSvjBVXSGVCICwvjNeZ8rnoYw0wcV2xMK4g0cmGtsq3YVG2YICDxuiRtuJ143y8aKs7B4LnMrv3AnCPAE5NhNdJUB0ML0aVjemvuZvW1qlc55IjKkjhXSVtQxS1GZVRalcvTbQ3hcEhKstcW99OEL/C4tndwV051E07tLg0YlrQeizWqKGxJnbSeBd11oc7FRAWvavVxMAWRcGlh8Ddwk6dq64BhFfgyY+weKGslUH0Wkd5QfjWSQprstOCMypVqxHtkrDhQ7TceZrMEJ6P25EnZkLn39LqZrIH0WhcMOdC7y7dWI01LjjRa/Kla4gFaTWWKsrzpVCqRoupRStf472rp9hT3/dyUDrCYntiZl5YAwrbwfOyMWNa4/VSA3GiK5OBLybaBuIVXKUlObfyWoU12UzvUcXsIOyF2ObdrXTYB1g8cm6EqDVkvCPB4hu/Gtio/RrjVdbRQ53v+ArOjrcSsK+x2LEpZMHcQjts/TBAw7kAvZcsrIStDVtjqfoWZdhG2dx/vru1tPIQh/uRFu1drntc24qMjWub6kyAvsXd3oDoG46+eETP7c/k3S1toU2mBsyhJg5M2hjrohStt64dlfDhRA+dqBF4hch7nbcZXLcytanaShrVrUmlK3eDQ+K5jfILLv56IJ8tLwvSjSAlDBSWBYmCG09525RqD4EOYFmqFUS+3r2nznujz8Ak7555nMP2LplLvqQ3H22e5Lfk21zzcngSXfMyZTvJQhKfKHhsBHqRinCoz5i7fYDhkBnvC5ialhxeS+VHFTdsLXh35L6QQ7wE64Lh8T3Y4/YzRMGqMlGGemTqayCTLz/TWMNiw0KxHDqoFlhQkeBQYJCZbtZ0VKjjlGVUyo813vOwRGjoa8ISl+yprBFsOYu8XwqYEnJ/3XoHsp0wukEdF671SxWUSZmi38zW8Bu5aBKSXoes1eCUs0D4AbrqIa3OudI5iZr++ewrpBjQxQOdeR0VhB1+mpAiGsNfViqolZBqMso2EIC4nZR9hLZpoLlxgPn+J94YKN+nzPcCi+3jDDya8jSYGUBBgBuDTQ3sbw/6rlKLsmHc06izclE2W6+fqyxJRvAIwiUcIZghZHcShzJRWWbqs9DwO8N5M3x7x3lq4QNd6WPI6QCfhi4h3abUDgDv6WOTuZQOj7wqoL+9URZxoGaxGZEAWUp8LSgfet2AVZfT5T8QRZSVe8snBT3oPxhdpEdHqGbAmXVDKsRUKARpGx6ixzfVnd8rMm70CxUdJTO8eA5mj2Nmlamok8d4aQpeaVjrgyuX2ja9x5cA8VZCd0DTY0ml9d2dIHt1m5UwLtfomggNraqhXei0I1weAa5vEwUwigWaQ4TQ9KLXkImFWKcuXBKeL4O75CT4b1js16Inmnw7BGq+bWLWLfY1hcL4CiK6dHDjlIrMmGbo6jbosHYtILtk1vgnXzUTZeGPPjLJiKyd0lchBx9pX6maNjnISme/EwrFl02lkCEDvnCyuBk+u7Mwa3RopL2u6iCoM6ZoZlSbgm2PKN4gDC7CHaEDvVN5X3HJiGG3Jrt7O0zoiuKiiwaX6arnhm8di23mHt+5WsMnFRILKSxsrk855jhJg+AnaqoMGHdRtr3J8AGsRauz9MYvpdF7yXUquuymK3UNtzxwpo73V2lZiS1eABU1RfQrmGUkkmByktTf7MWo1YX31W5EqLuPxW4tz6JCLqEk10N2zkI00L0prm2H6GoDc/MqN46ZGogEUiW+znRuTIQ7N17sh29zQqVvKn7TVkO3Jhue+W0p68ZPqZKR9Br67v1P70vfRcrbVFs+1f1AKDTXoEp1BYGOVWV2x+ieQe7sIduf1afuJKHGSgEysYqQ1a7u61gszKhAMgYBnzHMfWi1qsfpUBmaUa/eKOeCWOjgrIWjVmkuQ1vVsWvBxk+iWgBqkfemSlVFrNs6s7DLa0IVl2x8pIpnG4NF2UPqoGh4qra6g/htdMYUFkbHeSB57nJVkImPkjjePdhfA3O0PrvwYsqNoIr9hVO1qaOyg0mlbKKf+TlurPOsgvo0veeal7N7qYmZJg5MrWpKwqUzdtQ+RCemAe5maUZAGZ8qd5QimRqyeOL1NLV2x1o3EH8dYVMFawaFMpbmyXLgraSNbVpgyixM9lIMzw0OephzBPDoeVsjlqYeg51PGiKb2/u8mYsuqNyIOgA5d1uF7jHIIvXAAyt6fe+7VBB1NeVJXZich7OOt7m/L4kscGWZuLMovp+cLy598Hb3dNT7crCxicxMp2nRhr6IA5mejawbKiv629akikEqNINGJinY/TS0cTO8luHuiXUdINuAXJGQXDPVixn7TJ2xygf7g5YmbxQcWzFgjeS3Zy0HfIDNCwL36sijqpXtOAxkOTbQIAeTpzoW5vgOK+xqzJmhgKyCKIJ1dYYjroHLXoHJPT7ttAdIW2lqLK43DpaVucwp8KlPghkxrtHepCGCk4GqEgmHsNl21rb4NR1gOTeGD7HYRXzrq00wOjOTcMBvvY0QdJz/kWE3y3BXl1dXro8+vnSeYKenrwyL4rC2g7KblYKl4Y6NzDlL/AJcWFcBtJrDT/sebWhcTJN/t5IgCN0Pm7RP3HeuU6nynho+8y7sTAynHi5WJ+3bq4fu0m1uc4FzF17nqQ7wSgmgsK4fWs5Vt0Q/peHBjSOcF6aXt2kcbT7NfOXOegXRbZWL6E2jZWoQ3U7ahYWztt89Ne/o3KHvWquacNElzxuR9nMofpc783RgGTh2CGS1DJRl55rQZdaFWbeidJZBjblCP+tdR5/GqAw89ZN7MjTWMpnI9U/SILv+ku4eFqmRM2othwICUi8b5clge1EOdWFqXraf/FzsGzxpb9XVx7YrcFHOnjzpipj2XrgJ8lLJObfjZx3q1JvlRWB7Rtv8B/umXm6puqF++z7c8HSBs+8QHlmpZcMBNU1AcBOaW1um4nFKX51Dygq22Fr/mjW81gZCClqdsueNbGweB68FiJdcNGHFaUuZihMlrNMx55/vwECamlMYc0KvnO8BStoksP2dL812K6fa++3Mw/6FxVFJcjANBG/aLerFsHbCqW2kDnViD853cyCAOHLLitfkjolOxU+o5tQ745v4Abwaac4VcNvkmXYXfGiZxn66rFfpS74c5yY8EMRt8lkT572eZKhpfAV2euCJuznmGMsnsnpfqp6Qbx+I/5lZZuYqyOBOQhq8417IoKw+cepYclNQmv0OcxvUffsXP0FWSrD9wN72s8MHn8vdT6WSZ4NPZGXlXnqe9kTjvYcrcncVa8uiTXdrW2876dV2PDbyyLu2kUzZjEoo2S/Ah94pFQtcgAmwyadVPf8pZt0PcNg3IPXbid9/sdhQYQqpLZvONwiGvQCKzPf0cVaqGtHRa5sqquvtiBuJ7q3I6vLKMp1+5fO11c9kfgnOZTH2wiy5e5CqvJdWiqiDuup0+ZRl57gSp6L55xL44eaXSBvrcm2py2S3njhO07A8VM3uzNk2PeG2sE1X1xRkE254H2g/MWwFsjD3Y593K5MFSrj9RvM3Zrbeb+bxTLb6gCOfwTvmEUZW7z6QV7zFsSsmcwmCeLW6vPzgdvZbOQEb004yKA/FWVcfKSIASqZSTWkxmedUQxhr/i0IrGjWqgbyFx1qbsQo4jIW/BsZ2JDhHjtPeTYDKXULGza+OmJoDRKIHC59O63rLyfWxXiYu3YY59sk4IMnEw9Xb0WXvxvxD8S1n91K4HCU9cql5gVUMtJZf8TDUdfoQarn//s5iM351741iL7NZn+Sxc/iX2adBuo1KLzmu77pG735tk9/ZvEzrn5/+ubWQdXh1wt9NRTB7k3H9H8B9vHpN7Zr/s1AsoNTCL36s5wG/P+Xf/PlDf8WrVh4jZP9D+GMYRo="]
The column “Sentence” contains 60 sentences from across the internet. The “Sentence” sentiments are listed in the columns “Mathematica FinBERT” and “Mathematica FinRoBERTa.” It is recognizable, that few sentiments differ among these two networks.
Sentiment analysis has been done with the “Huggingface FinBERT solution for Python” [7][8], and with the Matlab FinBERT solution [9]. A comparison of all the sentiments shows that there are differences between the sentiments.
The person who wrote the notebook is not a native English speaker, so some mistakes might happen during sentence interpretation. The sentiments calculated by Mathematica FinRoBERTA fit the expectations of most of the author. In general, the differences might be caused by a lack of training settings that are standard.
Therefore, the user is free to choose which solution he/she prefers.
Sentiment analysis has been done with the “Huggingface FinBERT solution for Python” [7][8], and with the Matlab FinBERT solution [9]. A comparison of all the sentiments shows that there are differences between the sentiments.
The person who wrote the notebook is not a native English speaker, so some mistakes might happen during sentence interpretation. The sentiments calculated by Mathematica FinRoBERTA fit the expectations of most of the author. In general, the differences might be caused by a lack of training settings that are standard.
Therefore, the user is free to choose which solution he/she prefers.
References
References
[1] Wolfram Neural Net Repository, https://resources.wolframcloud.com/NeuralNetRepository
[2] Wolfram Neural Net Repository, Text Domain, https://resources.wolframcloud.com/NeuralNetRepository/inputdomain/Text/
[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, 2019, RoBERTa: A Robustly Optimized BERT Pretraining Approach, https://arxiv.org/abs/1907.11692, arXiv:1907.11692v1
[4] The dataset of the Financial Phrase-Bank can be found here: https://www.researchgate.net/publication/251231364_FinancialPhraseBank-v10
[5] Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala, 2014, Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology 65, 4 (2014), 782–796. https://doi.org/10.1002/asi.23062 arXiv:arXiv:1307.5336v2
[6] selection of FiQA-2018 data, https://sites.google.com/view/fiqa/home
[7] https://github.com/yya518/FinBERT
[8] https://finbert.ai/sentiment
[9] https://github.com/matlab-deep-learning/transformer-models
[10] BERT Trained on BookCorpus and Wikipedia Data, https://resources.wolframcloud.com/NeuralNetRepository/resources/BERT-Trained-on-BookCorpus-and-Wikipedia-Data/
[11] RoBERTa Trained on BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories Datasets, https://resources.wolframcloud.com/NeuralNetRepository/resources/RoBERTa-Trained-on-BookCorpus-English-Wikipedia-CC-News-OpenWebText-and-Stories-Datasets/
[2] Wolfram Neural Net Repository, Text Domain, https://resources.wolframcloud.com/NeuralNetRepository/inputdomain/Text/
[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, 2019, RoBERTa: A Robustly Optimized BERT Pretraining Approach, https://arxiv.org/abs/1907.11692, arXiv:1907.11692v1
[4] The dataset of the Financial Phrase-Bank can be found here: https://www.researchgate.net/publication/251231364_FinancialPhraseBank-v10
[5] Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala, 2014, Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology 65, 4 (2014), 782–796. https://doi.org/10.1002/asi.23062 arXiv:arXiv:1307.5336v2
[6] selection of FiQA-2018 data, https://sites.google.com/view/fiqa/home
[7] https://github.com/yya518/FinBERT
[8] https://finbert.ai/sentiment
[9] https://github.com/matlab-deep-learning/transformer-models
[10] BERT Trained on BookCorpus and Wikipedia Data, https://resources.wolframcloud.com/NeuralNetRepository/resources/BERT-Trained-on-BookCorpus-and-Wikipedia-Data/
[11] RoBERTa Trained on BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories Datasets, https://resources.wolframcloud.com/NeuralNetRepository/resources/RoBERTa-Trained-on-BookCorpus-English-Wikipedia-CC-News-OpenWebText-and-Stories-Datasets/