Llama vs GPT
Llama vs GPT
Some days ago I posted a ways to access OpenAI's GPT, DALL-E and TTS apis via HTTPRequest and URLExecute. In this post I will introduce a function to access the Llama API my Meta (even though not the latest version). I will then look at a couple of examples and juxtapose the two LLMs.
First I will import the functionality I need for OpenAI.
In[]:=
Import["/Users/thiel/Desktop/GPT-via-API.wl"]
For this particular post, I should be able to run most of the examples with the two built-in Wolfram Language functions LLMSynthesize and LLMFunction, but I will still use my own implementation.
After importing the file I now have the functions:
◼
callTTS[myPrompt, myAPIKey]
◼
callDALLE[myPrompt, myAPIKey]
◼
callChatGPT[myPrompt, myAPIKey]
◼
callGPTVision[myPrompt, myImageURL, myAPIKey]
◼
callGPTVisionFromImage[myPrompt, myImage, myAPIKey]
The difference between the last two versions is that in the first case, the second argument is an image URL and in the second case it is the image itself.
At the time of writing this and to the best of my knowledge LLMSynthesize and LLMFunction cannot work with the TTS (text to speech) model or the gpt-4-vision models. Furthermore, I have not managed to use some of the recently released features such as a seed or settings for JSON output directly in the built in functions. I would strongly recommend to use the built-in functions if you do not need any of the unsupported features.
At the time of writing this and to the best of my knowledge LLMSynthesize and LLMFunction cannot work with the TTS (text to speech) model or the gpt-4-vision models. Furthermore, I have not managed to use some of the recently released features such as a seed or settings for JSON output directly in the built in functions. I would strongly recommend to use the built-in functions if you do not need any of the unsupported features.
Building an API link for Llama
Building an API link for Llama
We are now building a simple function to access Llama functions, see this website. GPT describes it like this:
“The Llama API is a cutting-edge tool designed to enhance the functionality of Large Language Models (LLMs) through a user-friendly API. Focusing on open-source models, it aims to improve the reliability and efficiency of these AI tools in various applications. The initiative not only provides access to these models but also curates and optimizes them, facilitating the development of more sophisticated AI assistants. Although the website offers an overview with resources like documentation and community platforms, it lacks detailed technical information, suggesting a need for further exploration for in-depth understanding.”
On the main website you can join a waitlist. Once admitted you can very easily get an API token from your account page, which is very straight forward.
“The Llama API is a cutting-edge tool designed to enhance the functionality of Large Language Models (LLMs) through a user-friendly API. Focusing on open-source models, it aims to improve the reliability and efficiency of these AI tools in various applications. The initiative not only provides access to these models but also curates and optimizes them, facilitating the development of more sophisticated AI assistants. Although the website offers an overview with resources like documentation and community platforms, it lacks detailed technical information, suggesting a need for further exploration for in-depth understanding.”
On the main website you can join a waitlist. Once admitted you can very easily get an API token from your account page, which is very straight forward.
An API call that currently works is:
In[]:=
callLlama[myPrompt_,myAPIKey_]:=Module[{myRequest,myResponse,jsonBody},jsonBody=ExportString[{"model"->"llama-13b-chat","messages"->{{"role"->"user",(*"temperature"->0.7*)"max_length"->2000,"content"->myPrompt}},"stream"->False(*Addotherfieldsasnecessary*)},"JSON"];myRequest=HTTPRequest["https://api.llama-api.com/chat/completions",<|Method->"POST","Body"->jsonBody,"ContentType"->"application/json","Headers"->{"Authorization"->"Bearer "<>myAPIKey}|>];myResponse=URLExecute[myRequest][[1,2,1,2,2,2,2]]]
Note that the long part function at the end [[1,2,1,2,2,2,2]] is a very awkward way of getting to the right bit of the API reply, which comes as list of rules and not as associations - similar to the functions I used for OpenAI’s API. I will assume that you have set the Llama key like so:
myLLamaKey=”<Your Llama Key Goes Here>”
myLLamaKey=”<Your Llama Key Goes Here>”
Let’s try this out:
In[]:=
callLlama["Who is Albert Einstein?",myLLamaKey]
Out[]=
🤔 Hey there! Albert Einstein was a renowned physicist who developed the theory of relativity and is considered one of the most influential scientists of the 20th century. He was born in Germany in 1879 and passed away in 1955. Einstein's work revolutionized our understanding of space, time, and gravity, and his famous equation E=mc² is still widely used today. He also received the Nobel Prize in Physics in 1921 for his explanation of the photoelectric effect. What else would you like to know about Albert Einstein? 😊
One of the interesting things to observe here is the use of emojis. Also, there are two different models available. The current one is 13B (A100 40gb) which comes at a cost of $0.0016/1K tokens at the time of writing this., and a second larger model 70B (A100 80gb) which comes at a cost of $0.0032/1K tokens. Let’s try the second one out.
In[]:=
callLlama[myPrompt_,myAPIKey_]:=Module[{myRequest,myResponse,jsonBody},jsonBody=ExportString[{"model"->"llama-70b-chat","messages"->{{"role"->"user",(*"temperature"->0.7*)"max_length"->2000,"content"->myPrompt}},"stream"->False(*Addotherfieldsasnecessary*)},"JSON"];myRequest=HTTPRequest["https://api.llama-api.com/chat/completions",<|Method->"POST","Body"->jsonBody,"ContentType"->"application/json","Headers"->{"Authorization"->"Bearer "<>myAPIKey}|>];myResponse=URLExecute[myRequest][[1,2,1,2,2,2,2]]]
In[]:=
callLlama["Who is Albert Einstein?",myLLamaKey]
Out[]=
Albert Einstein (1879-1955) was a renowned physicist who is widely regarded as one of the most influential scientists of the 20th century. He is best known for his theory of relativity, which revolutionized our understanding of space and time, and led to the famous equation E=mc². Einstein's work has had a profound impact on the development of modern physics and has led to many technological innovations, including GPS technology and nuclear power. He was awarded the Nobel Prize in Physics in 1921 for his explanation of the photoelectric effect. Einstein is also known for his advocacy of peace and human rights, and his name has become synonymous with genius and intellectual curiosity.
This model also often produces emojis though. So let’s try some better prompting.
Tweaking it a bit
Tweaking it a bit
Let’s try whether we can prompt the emojis away:
In[]:=
callLlama[myPrompt_,myAPIKey_]:=Module[{myRequest,myResponse,jsonBody},jsonBody=ExportString[{"model"->"llama-70b-chat","messages"->{{"role"->"system","content"->"Please do not use emojis in your reply. Produce a professional, formal and well structured reply. Before you reply, take a deep breath and think step by step."},{"role"->"user","max_length"->2000,"content"->myPrompt}},"stream"->False(*Addotherfieldsasnecessary*)},"JSON"];myRequest=HTTPRequest["https://api.llama-api.com/chat/completions",<|Method->"POST","Body"->jsonBody,"ContentType"->"application/json","Headers"->{"Authorization"->"Bearer "<>myAPIKey}|>];myResponse=URLExecute[myRequest][[1,2,1,2,2,2,2]]]
In[]:=
callLlama["Who is Albert Einstein?",myLLamaKey]
Out[]=
Albert Einstein (1879-1955) was a renowned theoretical physicist and mathematician who is widely regarded as one of the most influential scientists of the 20th century. Born in Germany to a Jewish family, Einstein's early passion for mathematics and physics led him to study at the Swiss Federal Polytechnic School, where he graduated in 1900. He later became a naturalized citizen of the United States and joined the faculty of Princeton University.Einstein's groundbreaking work in physics includes the development of the special theory of relativity, which posits that the laws of physics are the same for all observers in uniform motion relative to one another. He also proposed the famous equation E=mc², which represents the relationship between energy and mass. In 1915, Einstein completed his general theory of relativity, which introduced the concept of gravity as a curvature of spacetime caused by the presence of mass and energy.Einstein's work has had a profound impact on the fields of physics, mathematics, and engineering. He was awarded the Nobel Prize in Physics in 1921 for his explanation of the photoelectric effect, and he has been recognized for his contributions to the development of quantum mechanics, Brownian motion, and cosmology.In addition to his scientific accomplishments, Einstein was an advocate for social justice and peace. He was a vocal critic of racism and nationalism, and he spoke out against the rise of fascism in Europe in the 1930s. Einstein's legacy continues to inspire scientists, philosophers, and humanitarians around the world.Do you have any further questions about Albert Einstein?
Let’s compare that to gpt-4-turbo (which I have set as the default model in the Option Inspector).
In[]:=
LLMSynthesize["Who is Albert Einstein?"]
Out[]=
Albert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist who is one of the most influential scientists of the 20th century. He developed the theory of relativity, one of the two pillars of modern physics (alongside quantum mechanics). His work is also known for its influence on the philosophy of science.Einstein is best known for his equation E=mc^2, which states that energy (E) equals mass (m) times the speed of light (c) squared. This equation shows the relationship between energy and mass and is a fundamental principle in the field of nuclear physics.His 1905 paper on the photoelectric effect, which describes how light energy is emitted or absorbed by electrons in certain materials, won him the Nobel Prize in Physics in 1921, as his prediction was crucial in establishing quantum theory.Einstein's contributions to the understanding of the physical universe include:- Special theory of relativity (1905): This theory introduced a new framework for all of physics and proposed new concepts of space and time. It included the famous equation E=mc^2 as part of its framework. - General theory of relativity (1915): Generalizing special relativity and Newton's law of universal gravitation, this theory provided a unified description of gravity as a geometric property of space and time, or spacetime. This theory has important implications for the study of cosmology and black holes.- Photons and energy quanta: His work on the photoelectric effect, as mentioned, was influential in quantum theory's development.Beyond his scientific work, Einstein was an outspoken advocate for peace, civil rights, and Zionism. He was known for his pacifism during the First World War, and later for his advocacy for a Jewish homeland and his support of the establishment of the State of Israel. After he left Germany at the advent of the Nazi regime, he settled in the United States, where he became a professor at the Institute for Advanced Study in Princeton, New Jersey, and lived there until his death in 1955. Einstein was also a noted public figure and philosopher of science, whose wit and insightful quotes on a variety of topics have become widely known.
Note that the prompt for gpt-4 was simpler and I added a system prompt to Llama. The information in both outputs seems to be similar.
Comparison on “challenging questions”
Comparison on “challenging questions”
Jugs of water
Jugs of water
We now ask both systems a type of question where usually some thought is required, but I will use a question which has a really simple solution.
Llama
Llama
In[]:=
callLlama["You have one jug that holds 12 litres of water and a second one that holds 6 litres of wather. You want use them to measure exactly 6 litres. How do you use the jugs to find the required quantity?",myLLamaKey]
Out[]=
Thank you for asking! I'm happy to help you with your query. Measuring exactly 6 litres using two jugs with different capacities can be a bit tricky, but it's definitely doable. Here's a step-by-step guide on how to achieve this:1. Fill the 12-litre jug completely with water.2. Pour water from the 12-litre jug into the 6-litre jug until it's full, leaving 6 litres in the 12-litre jug.3. Empty the 6-litre jug.4. Pour the remaining 6 litres from the 12-litre jug into the 6-litre jug.5. Voila! You now have exactly 6 litres in the 6-litre jug.I hope this helps. Please let me know if you need further clarification or have any other questions!
GPT-4
GPT-4
That is interesting as in both cases this is not the straight forward solution. Let’s try something like a follow-up.
Llama
Llama
GPT-4
GPT-4
In a normal ChatGPT window I was able to convince GPT that there is a better solution.... It is also interesting that Llama uses my (British) spelling of litre whereas GPT answer with liter.
Let’s try some coding
Let’s try some coding
Llama
Llama
Let’s run that code
GPT
GPT
Let’s run that, too.
It appears that Llama does not only produce a wrong result; GPT-4’s code is correct and better structured as well as commented; it does print out the result twice. I do believe though that it is possible to find a simpler solution:
A more challenging problem
A more challenging problem
Suppose we are looking for a number abcdefghij, which is one number where a is the fist digit, b the second digit, etc. All digits are different. We are looking for this number, which the following contraints:
a is divisible by 1
ab (i.e. the number with first digit a and second digit b) is divisible by 2
abc is divisible by 3
abcd is divisible by 4
abcde is divisible by 5
abcdef is divisible by 6
abcdefg is divisible by 7
abcdefgh is divisible by 8
abcdefghi is divisible by 9
abcdefghij is divisible by 10
What is the number abcdefghij?
a is divisible by 1
ab (i.e. the number with first digit a and second digit b) is divisible by 2
abc is divisible by 3
abcd is divisible by 4
abcde is divisible by 5
abcdef is divisible by 6
abcdefg is divisible by 7
abcdefgh is divisible by 8
abcdefghi is divisible by 9
abcdefghij is divisible by 10
What is the number abcdefghij?
Llama
Llama
The code and the solution offered are both incorrect.
GPT-4
GPT-4
This code is not only correct, but it is also fast.
Let’s try a couple of riddles
Let’s try a couple of riddles
◼
Rachel goes to the supermarket and buys 10 tomatoes. Unfortunately, on the way back home, all but 9 get ruined. How many tomatoes are left in a good condition? (Answer 9)
◼
What 3 numbers give the same result when multiplied and added together? (Answer 1,2,3)
◼
Who makes it, has no need of it. Who buys it, has no use for it. Who uses it can neither see nor feel it. What is it? (Answer a coffin)
◼
What has hands but cannot clap? (Answer: a clock)
◼
What has a letter in it and starts and ends with an ‘e’. (Answer: envelope)
◼
What runs all around a backyard, yet never moves? (Answer: a fence)
◼
Two girls were born to the same mother, on the same day, at the same time, in the same month and year, and yet they’re not twins. How can this be? (Answer: The two babies are two of a set of triplets.)
Both models do rather well on these riddles, but Llama’s answer to the last question does not convince.
Conclusion
Conclusion
In this post we have developed a short function to interact with Llama. We have run a couple of performance tests. This is not a comprehensive way to compare the two models, and both got most of the problems right. Also, we have ignored functionality such as DALLE and TTS.
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Llama (another Large Language Model) and GPT comparison
by Marco Thiel
Wolfram Community, STAFF PICKS, November 17, 2023
https://community.wolfram.com/groups/-/m/t/3066344
by Marco Thiel
Wolfram Community, STAFF PICKS, November 17, 2023
https://community.wolfram.com/groups/-/m/t/3066344