In a project defined by Stephen Wolfram, we investigated the simulation of “drugging” effects on Large Language Models (LLMs). The initial idea, inspired by neuroscience, was to adjust LLM connectivity to mimic the effects of LSD on the brain. However, due to time constraints and the complexity of accurately simulating psychedelic effects, we altered our approach. Following Theodore Gray’s suggestion, we started by manipulating the LLM’s temperature to induce a “hallucination effect” on specific text strings, aiming to abstract language. We developed a function for ChatGPT to interpret this altered text, testing its ability to detect its own temperature changes. This set the stage for the concept of “drugging” language models. We also created embeddings for generated sequences to visualize meaning space to predict temperature and developed a psycholinguistic function to check text coherence. Our findings suggest that LLM’s coherence is greater when we have a meaningful psycholinguistic scale. Next steps include refining the coherence function, submitting it to the Wolfram Function Repository, and further exploring Xenolinguistics models to develop deeper and more meaningful psycholinguistic functions.
An initial attempt at guessing LLM’s Temperature
An initial attempt at guessing LLM’s Temperature
Our first attempt to “drug” Large Language Models (LLMs) involved a function named . We modified the temperature of a set of essays, each initiated by random words. We then used gpt-3.5-turbo to evaluate a different set of essays, also modified by random temperature changes. The aim was to compare the outcomes and extract insights from the process.
temperatureGuesser
Setting up the function (temperatureGuesser)
Setting up the function ()
temperatureGuesser
Here, we create a set of random words to generate some essays with varied temperatures that we may create a few-shot learning example to build our function.
temperatureGuesser
Here, we set a random seed and generate a set of 10 random words.
In[]:=
SeedRandom[1234];words=RandomWord[11];(*Generates11seedwords,fixedtoasetbytheSeedRandomfunction*)
We then use these words and a mapped set of temperatures to write a series of essays.
In[]:=
generatedText=MapThread[LLMSynthesize["Write 40 word essay about the word "<>#1,LLMEvaluator-><|"Model"->"gpt-3.5-turbo","Temperature"->#2|>,MaxItems->40]&,{words,Table[i,{i,0,2,.2}]}];(*GeneratesatabletextwithvariedTemperatures,from0to2insteps.2https://reference.wolfram.com/language/ref/LLMSynthesize.html*)
Once we have our essay data set, we map the corresponding temperatures to their respective strings to create a few-shot learning example.
In[]:=
fewShotExamples=MapThread[#1->#2&,{generatedText,Table[i,{i,0,2,.2}]}](*Mapsthetempraturestothetoourgeneratedtextinstringform*)
Now, with our few-shot examples, we can create the function using Mathematica’s and send it to gpt-3.5-turbo for processing.
temperatureGuesser
LLMExampleFunction
In[]:=
temperatureGuesser=LLMExampleFunction[{"Guess the token sampling temperature used to generate a piece of text using a GPT model:",fewShotExamples},LLMEvaluator-><|"Model"->"gpt-3.5-turbo"|>](*givesourmappedfewShotExamplesaalongwithMaintaskPromptforchatGPT*)
Out[]=
LLMFunction
Applying the function (temperatureGuesser)
Applying the function ()
temperatureGuesser
Now that we have our function, we need to create a test data set of essays. We’ll follow the previous steps to generate a new set of random words and temperatures, and then use these to create a set of essays for testing.
temperatureGuesser
SeedRandom[1234];testWords=Drop[RandomWord[51],11];(*Generates11seedwords,fixedtoasetbytheSeedRandomfunction*)
We use random integer to get 20 different temperatures.
SeedRandom[1234];testTemperatures=N[RandomInteger[{0,200},40]/100]
Using the same method, we did to generate the initial essays we generate a new set for testing purposes
tests=generatedText=MapThread[LLMSynthesize["Write 40 word essay about the word "<>#1,LLMEvaluator-><|"Model"->"gpt-3.5-turbo","Temperature"->#2|>,MaxItems->40]&,{testWords,testTemperatures}];
We can now apply our function to our sample essay set using the function.
temperatureGuesser
Map
In[]:=
testResults=Map[temperatureGuesser,tests]
Creating a that displays the difference between our and the (the original temperatures). To help evaluate the performance of our temperature prediction function.
BarChart
testResults
testTemperatures
In[]:=
BarChart[Map[ToExpression,testResults]-testTemperatures]
It seems our function is consistently overestimating temperatures?
Reworking the function (temperatureGuesser)
Reworking the function ()
temperatureGuesser
Our function is consistently overestimating temperatures, likely due to the ascending order of our training data. This ordering may cause gpt-3.5-turbo to expect each subsequent temperature to be higher. To address this, we can use Mathematica’s function to randomize our data, potentially improving the function’s accuracy. Here is an example of how works so you can see for yourself.
RandomSample
RandomSample
In[]:=
RandomSample[{"1"->1,2,3,4}]
Now, let’s apply the function to our function. By shuffling the examples and rewriting our function, we aim to resolve the issue of overestimation.
RandomSample
temperatureGuesser
In[]:=
SeedRandom[1234];temperatureGuesser2=LLMExampleFunction[{"Guess the token sampling temperature used to generate a piece of text using a GPT model:",RandomSample[fewShotExamples]},"Expression",LLMEvaluator-><|"Model"->"gpt-3.5-turbo"|>]
Out[]=
LLMFunction
We can now reapply the function using the previously mentioned data sets to generate our new predictions.
temperatureGuesser2
testResults2=Map[temperatureGuesser2,tests]
By plotting our results, it’s evident that the randomization has significantly improved the performance of our function.
In[]:=
BarChart[testResults2-testTemperatures]
A more nuanced approach, creating a Language Coherence Grader
A more nuanced approach, creating a Language Coherence Grader
Our function returns a numeric grade from 1-5 for each of the five psycholinguistic parameters, along with an explanation for each grade. We can format these results in a color-coded grid for easier reading and interpretation.
(The code for these graphics may be found in the notebooks appendix section).
We then did the same thing for interview text from Dr. Dre to compare and contrast and see if our function behaves somewhat stable.
(Once again, the code for these graphics may be found in the notebooks appendix section).
Our coherence grader seems to offers valuable insights across five categories: syntactic complexity, semantic complexity, cohesion, readability, and coherence. It’s reliable, consistent, has meaningful explanations, and lends itself to clear visualizations. Future enhancements include collecting more relevant datasets and refining the LLM function. If the same text is tested a number of times how much do the results vary? Currently, it’s a solid foundation and we’re considering submitting it to the Wolfram Prompt Repository or Wolfram Function Repository.
Concluding remarks
Concluding remarks
Although LLM Few-shot learning functions (alone) are not great for evaluating temperature, humans may be worse. However, randomizing learning examples, improves LLM functions. To visualize LLM high-dimensional embedding space, we plotted the trajectories of sequence of tokens and compared them at different temperatures. Using a basic psycholinguistic model we created a prompt with a one-shot learning example, that allows us to grade text coherence using an LLM.
In the future we would like to refine our few-shot learning examples for large language models (LLMs) and explore modifying the embedding layer, neural layer, attention layer, etc., modify activation functions. Compare our generated text against a variety of transcripts from patients on drugs. Consider submitting our work to the Wolfram Prompt Repository or Wolfram Function Repository. Enhance our understanding of psycholinguistic models to visualize “meaning-space”. In the long term, explore Xenolinguistics models to develop more profound psycholinguistic functions.
In the future we would like to refine our few-shot learning examples for large language models (LLMs) and explore modifying the embedding layer, neural layer, attention layer, etc., modify activation functions. Compare our generated text against a variety of transcripts from patients on drugs. Consider submitting our work to the Wolfram Prompt Repository or Wolfram Function Repository. Enhance our understanding of psycholinguistic models to visualize “meaning-space”. In the long term, explore Xenolinguistics models to develop more profound psycholinguistic functions.
Keywords
Keywords
#LLMS, Large Language Models, ChatGPT, Psychedelics, Drugs, one-shot learning, few-shot learning, Xenolinguistics, psycholinguistic, temperature, Randomizing
Acknowledgment
Acknowledgment
I would like to kindly acknowledge my mentor Jofre Espigule for guiding me through the entire process,Theodore Gray for clarity on the “hallucination effect”, Bob Nachbar for clarifying “what LLM’s are doing”, Mark Greenberg for visualizations, Aaron Carver for editing, and Stephen Wolfram for defining the project for me.
Appendix
Appendix
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
What Happens IF You "DRUG" LLMs?
by Michael Grey
Wolfram Community, STAFF PICKS, July 12, 2023
https://community.wolfram.com/groups/-/m/t/2959382
by Michael Grey
Wolfram Community, STAFF PICKS, July 12, 2023
https://community.wolfram.com/groups/-/m/t/2959382