I compared sentiment analysis using a built-in classifier, LSTM’s Neural Net model, and LLM. As expected, the LLM was the highest accuracy without training, but I also encountered various problems in my environment. These problems include: - Extremely poor response time compared to the other methods - Responses occasionally being rejected due to filtering policies (only in Azure OpenAI) - Responses sometimes not matching the specified label - High cost for large volume analysis
The key lies in understanding the characteristics of LLM and determining how we can best utilize it .
Out[]=
Method
Accuracy
Response
Cost
Built-in Classifier
0.63
0.003
s
free
NN model
0.84
0.287
s
free
NN model with GPU
0.84
1.841
s
free
LLM(gpt-35-turbo)
0.9
5.275
s
charged

testData


Built-in Classifier

Use the Built - in Classifier. Restrict the possible outputs to two, Positive and Negative, using ClassPriors.
In[]:=
c[txt_String]:=Classify["Sentiment",txt,ClassPriors<|"Positive"0.5,"Negative"0.5|>]
In[]:=
str=DateObject[];​​predicted=c/@testData[[All,1]];​​(*averageresponsetime*)​​Round[(DateObject[]-str)/Length[testData],0.001]
Out[]=
0.003
s
In[]:=
(*Accuracy*)​​actual=testData[[All,2]];​​tr=Transpose[{actual,predicted}];​​Round[Count[actual-predicted,0]/Length[testData],0.001]
Out[]=
0.63
In[]:=
(*PositiveAccuracy*)​​Round[Count[(actual-predicted)[[;;250]],0]/250,0.001]
Out[]=
0.504
In[]:=
(*NegativeAccuracy*)​​Round[Count[(actual-predicted)[[251;;]],0]/250,0.001]
Out[]=
0.756

Neural Net model

Use the LSTM's Neural Net model from the WOLFRAM NEURAL NET REPOSITORY.
In[]:=
sentimentNetModel[text_,device_:"GPU"]:=With[{output=NetModel["Sentiment Language Model Trained on Amazon Product Review Data"][text,NetPort[{"LastState","Output"}],TargetDevice->device]},If[Switch[Depth[output],2,output[[2389]],3,output[[All,2389]]]>0,"Positive","Negative"]]
In[]:=
str=DateObject[];​​predicted=sentimentNetModel/@testData[[All,1]];​​(*averageresponsetime*)​​Round[UnitConvert[(DateObject[]-str)/Length[testData],"Seconds"],0.001]
Out[]=
0.287
s
In the case of TargetDevice → "CPU", it is
1.841
s
.
In[]:=
(*Accuracy*)​​actual=testData[[All,2]];​​tr=Transpose[{actual,predicted}];Round[Count[actual-predicted,0]/Length[testData],0.001]
Out[]=
0.842
In[]:=
(*PositiveAccuracy*)​​Round[Count[(actual-predicted)[[;;250]],0]/250,0.001]
Out[]=
0.844
In[]:=
(*NegativeAccuracy*)​​Round[Count[(actual-predicted)[[251;;]],0]/250,0.001]
Out[]=
0.84

LLM

Use Azure OpenAI Service gpt-35-turbo model. Since LLMFunction and LLMExampleFunction do not yet support Azure OpenAI Service, use gpt-35-turbo model with Python.
sentimentLLM[txt_String]:=Block[{res},​​res=ExternalEvaluate["Python","import openaiopenai.api_type = \"azure\"openai.api_version = \"2023-06-01-preview\"openai.api_base = \""<>SystemCredential["Azure_OPENAI_ENDPOINT"]<>"\"openai.api_key = \""<>SystemCredential["Azure_OPENAI_API_KEY"]<>"\"​response = openai.ChatCompletion.create( engine=\"gpt-35-turbo\", temperature=.1, max_tokens=4000, top_p=0.5, frequency_penalty=0, presence_penalty=0, stop=None, messages=[ {\"role\": \"user\", \"content\":\"Please perform Sentiment Classification task.\\nGiven the sentence, assign a sentiment label from ['Negative', 'Positive']. Return label only without any other text.\\n\\nSentence: effective but too-tepid biopic .\\nLabel: Positive\\n\\nSentence: "<>sr@txt<>"\\nLabel: \"} ])response"];​​If[FailureQ@res,​​res,​​{res["choices"][[1]]["message","content"],res["usage"]["prompt_tokens"],res["usage"]["completion_tokens"]}​​]​​];​​​​sr[str_]:=StringReplace[str,{"\n"->"\\n","\""->"'"}];
In[]:=
str=DateObject[];​​predicted=sentimentLLM/@testData[[All,1]];​​(*averageresponsetime*)​​Round[UnitConvert[(DateObject[]-str)/Length[testData],"Seconds"],0.001]
Out[]=
5.275
s
In[]:=
(*Accuracy*)​​actual=testData[[All,2]];​​tr=Transpose[{actual,predicted}];Round[Count[actual-predicted[[All,1]],0]/Length[testData],0.001]
Out[]=
0.9
In[]:=
(*PositiveAccuracy*)​​Round[Count[(actual-predicted[[All,1]])[[;;250]],0]/250,0.001]
Out[]=
0.864
In[]:=
(*NegativeAccuracy*)​​Round[Count[(actual-predicted[[All,1]])[[251;;]],0]/250,0.001]
Out[]=
0.936
The cost is for the number of tokens for each prompt and completion. (pricing)
In[]:=
(*Cost*)​​Quantity[0.002,"USDollars"]*Total@Select[predicted[[All,2]],NumberQ]/1000+​​Quantity[0.0015,"USDollars"]*Total@Select[predicted[[All,3]],NumberQ]/1000
Out[]=
$
0.08
A closer look at the response of sentimentLLM shows that it is responding in a manner other than “Positive”, “Negative”.
In[]:=
predicted[[All,1]]//Counts
Out[]=
Positive229,Negative266,PythonError2,Neutral3
The response of sentimentLLM is “PythonError” because the LLM judges that there are inappropriate expressions in the sentence and rejects the response according to the filtering policies.
error=Position[predicted[[All,1]],"PythonError"];​​Extract[predicted,error]
This limitation is for the use of Azure OpenAI Service and does not occur when OpenAI is used . (Doesn' t mean OpenAI is better!)
At the prompt, the LLM was instructed to respond "Positive" or "Negative," but the LLM ignored it and responded "Neutral”. I think that this “fickleness” is one of the most troublesome problems in the LLM.

CITE THIS NOTEBOOK

Sentiment analysis by LLM​
by Kotaro Okazaki​
Wolfram Community, STAFF PICKS, August 15, 2023
https://community.wolfram.com/groups/-/m/t/2991010