Another interesting project that makes it easy to run local LLM models is Olhama at https://ollama.com/.
Here I did some tests with the model “Tinyllama”. It is running on my local machine, one of the smallest models.
Those local LLMs models can be installed at your local data center servers and network, or VSP in the cloud, for privacy and to integrate with your company data.
Those projects are very convenient for consulting, integrating Wolfram|Alpha data with private internal LLMs with all the convenience and Wolfram Language power.
Here I did some tests with the model “Tinyllama”. It is running on my local machine, one of the smallest models.
Those local LLMs models can be installed at your local data center servers and network, or VSP in the cloud, for privacy and to integrate with your company data.
Those projects are very convenient for consulting, integrating Wolfram|Alpha data with private internal LLMs with all the convenience and Wolfram Language power.
With the command line (shell) tool curl at Linux terminal, we can call the local WEB API with REST JSON format, using standard parameters from OpenAI:
curl http://localhost:11434/api/generate -d '{
"model": "tinyllama",
"stream": false,
"prompt":"Why is the sky blue?"
}'
"model": "tinyllama",
"stream": false,
"prompt":"Why is the sky blue?"
}'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
{"model":"tinyllama","created_at":"2024-08-09T03:23:32.68586328Z","response":"The sky blue is caused by different colors of light reflecting off the Earth's atmosphere in different wavelengths. Blue light, which makes up most of the visible spectrum of light, ranges from around 400 to 750 nanometers (nm). As you move closer to the blue end of the spectrum (around 400 nm), it gets brighter and brighter until it reaches a maximum intensity at about 656 nm, which is called infrared. This red light starts to get weaker and then fades away as you move further into the red end of the spectrum (around 750 nm). As a result, blue light is what we perceive as blue, and it makes up about 1% of all visible light in the sky.","done":true,"done_reason":"stop","context":[529,29989,5205,29989,29958,13,3492,526,263,8444,319,29902,20255,29889,2,29871,13,29966,29989,1792,29989,29958,13,11008,338,278,14744,7254,29973,2,29871,13,29966,29989,465,22137,29989,29958,13,1576,14744,7254,338,8581,491,1422,11955,310,3578,9432,292,1283,278,11563,29915,29879,25005,297,1422,10742,2435,4141,9499,29889,10924,3578,29892,607,3732,701,1556,310,278,7962,18272,310,3578,29892,20238,515,2820,29871,29946,29900,29900,304,29871,29955,29945,29900,23432,8328,29879,313,22882,467,1094,366,4337,17649,304,278,7254,1095,310,278,18272,313,11316,29871,29946,29900,29900,302,29885,511,372,4947,11785,261,322,11785,261,2745,372,22170,263,7472,26171,472,1048,29871,29953,29945,29953,302,29885,29892,607,338,2000,3041,25983,29881,29889,910,2654,3578,8665,304,679,8062,261,322,769,28747,29879,3448,408,366,4337,4340,964,278,2654,1095,310,278,18272,313,11316,29871,29955,29945,29900,302,29885,467,1094,263,1121,29892,7254,3578,338,825,591,17189,573,408,7254,29892,322,372,3732,701,1048,29871,29896,29995,310,599,7962,3578,297,278,14744,29889],"total_duration":60280165022,"load_duration":21142981,"prompt_eval_count":40,"prompt_eval_duration":320228000,"eval_count":170,"eval_duration":59896443000}
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 77 0 0 100 77 0 64 0:00:01 0:00:01 --:--:-- 64100 77 0 0 100 77 0 34 0:00:02 0:00:02 --:--:-- 34100 77 0 0 100 77 0 24 0:00:03 0:00:03 --:--:-- 24100 77 0 0 100 77 0 18 0:00:04 0:00:04 --:--:-- 18100 77 0 0 100 77 0 14 0:00:05 0:00:05 --:--:-- 14100 77 0 0 100 77 0 12 0:00:06 0:00:06 --:--:-- 0100 77 0 0 100 77 0 10 0:00:07 0:00:07 --:--:-- 0100 77 0 0 100 77 0 9 0:00:08 0:00:08 --:--:-- 0100 77 0 0 100 77 0 8 0:00:09 0:00:09 --:--:-- 0100 77 0 0 100 77 0 7 0:00:11 0:00:10 0:00:01 0100 77 0 0 100 77 0 6 0:00:12 0:00:11 0:00:01 0100 77 0 0 100 77 0 6 0:00:12 0:00:12 --:--:-- 0100 77 0 0 100 77 0 5 0:00:15 0:00:13 0:00:02 0100 77 0 0 100 77 0 5 0:00:15 0:00:14 0:00:01 0100 77 0 0 100 77 0 5 0:00:15 0:00:15 --:--:-- 0100 77 0 0 100 77 0 4 0:00:19 0:00:16 0:00:03 0100 77 0 0 100 77 0 4 0:00:19 0:00:17 0:00:02 0100 77 0 0 100 77 0 4 0:00:19 0:00:18 0:00:01 0100 77 0 0 100 77 0 4 0:00:19 0:00:19 --:--:-- 0100 77 0 0 100 77 0 3 0:00:25 0:00:20 0:00:05 0100 77 0 0 100 77 0 3 0:00:25 0:00:21 0:00:04 0100 77 0 0 100 77 0 3 0:00:25 0:00:22 0:00:03 0100 77 0 0 100 77 0 3 0:00:25 0:00:23 0:00:02 0100 77 0 0 100 77 0 3 0:00:25 0:00:24 0:00:01 0100 77 0 0 100 77 0 3 0:00:25 0:00:25 --:--:-- 0100 77 0 0 100 77 0 2 0:00:38 0:00:26 0:00:12 0100 77 0 0 100 77 0 2 0:00:38 0:00:27 0:00:11 0100 77 0 0 100 77 0 2 0:00:38 0:00:28 0:00:10 0100 77 0 0 100 77 0 2 0:00:38 0:00:29 0:00:09 0100 77 0 0 100 77 0 2 0:00:38 0:00:30 0:00:08 0100 77 0 0 100 77 0 2 0:00:38 0:00:31 0:00:07 0100 77 0 0 100 77 0 2 0:00:38 0:00:32 0:00:06 0100 77 0 0 100 77 0 2 0:00:38 0:00:33 0:00:05 0100 77 0 0 100 77 0 2 0:00:38 0:00:34 0:00:04 0100 77 0 0 100 77 0 2 0:00:38 0:00:35 0:00:03 0100 77 0 0 100 77 0 2 0:00:38 0:00:36 0:00:02 0100 77 0 0 100 77 0 2 0:00:38 0:00:37 0:00:01 0100 77 0 0 100 77 0 2 0:00:38 0:00:38 --:--:-- 0100 77 0 0 100 77 0 1 0:01:17 0:00:39 0:00:38 0100 77 0 0 100 77 0 1 0:01:17 0:00:40 0:00:37 0100 77 0 0 100 77 0 1 0:01:17 0:00:41 0:00:36 0100 77 0 0 100 77 0 1 0:01:17 0:00:42 0:00:35 0100 77 0 0 100 77 0 1 0:01:17 0:00:43 0:00:34 0100 77 0 0 100 77 0 1 0:01:17 0:00:44 0:00:33 0100 77 0 0 100 77 0 1 0:01:17 0:00:45 0:00:32 0100 77 0 0 100 77 0 1 0:01:17 0:00:46 0:00:31 0100 77 0 0 100 77 0 1 0:01:17 0:00:47 0:00:30 0100 77 0 0 100 77 0 1 0:01:17 0:00:48 0:00:29 0100 77 0 0 100 77 0 1 0:01:17 0:00:49 0:00:28 0100 77 0 0 100 77 0 1 0:01:17 0:00:50 0:00:27 0100 77 0 0 100 77 0 1 0:01:17 0:00:51 0:00:26 0100 77 0 0 100 77 0 1 0:01:17 0:00:52 0:00:25 0100 77 0 0 100 77 0 1 0:01:17 0:00:53 0:00:24 0100 77 0 0 100 77 0 1 0:01:17 0:00:54 0:00:23 0100 77 0 0 100 77 0 1 0:01:17 0:00:55 0:00:22 0100 77 0 0 100 77 0 1 0:01:17 0:00:56 0:00:21 0100 77 0 0 100 77 0 1 0:01:17 0:00:57 0:00:20 0100 77 0 0 100 77 0 1 0:01:17 0:00:58 0:00:19 0100 77 0 0 100 77 0 1 0:01:17 0:00:59 0:00:18 0100 77 0 0 100 77 0 1 0:01:17 0:01:00 0:00:17 0100 2038 100 1961 100 77 32 1 0:01:17 0:01:00 0:00:17 482
Out[]=
Success
Another way to call the Ollama is by ExternalEvaluate function:
In[]:=
ExternalEvaluate[{"Shell","ReturnType"->"Expression"},"curl http://localhost:11434/api/generate -d '{ \"model\": \"tinyllama\", \"stream\": false, \"prompt\":\"Distance from Boston to Chicago?\"}'"]
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
{"model":"tinyllama","created_at":"2024-08-09T23:19:20.830531018Z","response":"The distance between Boston and Chicago is approximately 2,345 miles (3780 km). It takes about 16 hours by car or 18 hours by flight to reach the two cities. The driving time would take into account traffic and road conditions, as well as any detours or stops for gas, rest, etc.","done":true,"done_reason":"stop","context":[529,29989,5205,29989,29958,13,3492,526,263,8444,319,29902,20255,29889,2,29871,13,29966,29989,1792,29989,29958,13,27469,515,12115,304,18168,1113,1484,29973,2,29871,13,29966,29989,465,22137,29989,29958,13,1576,5418,1546,12115,322,18168,1113,1484,338,14235,29871,29906,29892,29941,29946,29945,7800,313,29941,29955,29947,29900,2383,467,739,4893,1048,29871,29896,29953,6199,491,1559,470,29871,29896,29947,6199,491,16286,304,6159,278,1023,14368,29889,450,19500,931,723,2125,964,3633,12469,322,6520,5855,29892,408,1532,408,738,1439,2470,470,17726,363,10489,29892,1791,29892,2992,29889],"total_duration":35004847731,"load_duration":4066353411,"prompt_eval_count":42,"prompt_eval_duration":9164777000,"eval_count":72,"eval_duration":21731367000}
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 92 0 0 100 92 0 76 0:00:01 0:00:01 --:--:-- 76100 92 0 0 100 92 0 41 0:00:02 0:00:02 --:--:-- 41100 92 0 0 100 92 0 28 0:00:03 0:00:03 --:--:-- 28100 92 0 0 100 92 0 21 0:00:04 0:00:04 --:--:-- 21100 92 0 0 100 92 0 17 0:00:05 0:00:05 --:--:-- 17100 92 0 0 100 92 0 14 0:00:06 0:00:06 --:--:-- 0100 92 0 0 100 92 0 12 0:00:07 0:00:07 --:--:-- 0100 92 0 0 100 92 0 11 0:00:08 0:00:08 --:--:-- 0100 92 0 0 100 92 0 9 0:00:10 0:00:09 0:00:01 0100 92 0 0 100 92 0 9 0:00:10 0:00:10 --:--:-- 0100 92 0 0 100 92 0 8 0:00:11 0:00:11 --:--:-- 0100 92 0 0 100 92 0 7 0:00:13 0:00:12 0:00:01 0100 92 0 0 100 92 0 6 0:00:15 0:00:13 0:00:02 0100 92 0 0 100 92 0 6 0:00:15 0:00:14 0:00:01 0100 92 0 0 100 92 0 6 0:00:15 0:00:15 --:--:-- 0100 92 0 0 100 92 0 5 0:00:18 0:00:16 0:00:02 0100 92 0 0 100 92 0 5 0:00:18 0:00:17 0:00:01 0100 92 0 0 100 92 0 5 0:00:18 0:00:18 --:--:-- 0100 92 0 0 100 92 0 4 0:00:23 0:00:19 0:00:04 0100 92 0 0 100 92 0 4 0:00:23 0:00:20 0:00:03 0100 92 0 0 100 92 0 4 0:00:23 0:00:21 0:00:02 0100 92 0 0 100 92 0 4 0:00:23 0:00:22 0:00:01 0100 92 0 0 100 92 0 3 0:00:30 0:00:23 0:00:07 0100 92 0 0 100 92 0 3 0:00:30 0:00:24 0:00:06 0100 92 0 0 100 92 0 3 0:00:30 0:00:25 0:00:05 0100 92 0 0 100 92 0 3 0:00:30 0:00:26 0:00:04 0100 92 0 0 100 92 0 3 0:00:30 0:00:27 0:00:03 0100 92 0 0 100 92 0 3 0:00:30 0:00:28 0:00:02 0100 92 0 0 100 92 0 3 0:00:30 0:00:29 0:00:01 0100 92 0 0 100 92 0 3 0:00:30 0:00:30 --:--:-- 0100 92 0 0 100 92 0 2 0:00:46 0:00:31 0:00:15 0100 92 0 0 100 92 0 2 0:00:46 0:00:32 0:00:14 0100 92 0 0 100 92 0 2 0:00:46 0:00:33 0:00:13 0100 92 0 0 100 92 0 2 0:00:46 0:00:34 0:00:12 0100 1232 100 1140 100 92 32 2 0:00:46 0:00:35 0:00:11 236100 1232 100 1140 100 92 32 2 0:00:46 0:00:35 0:00:11 298
Out[]=
Success
Refactoring the function llama to call Ollama by WEB API REST JSON:
In[]:=
llama[prompt_]:=Module[{result=URLRead[HTTPRequest["http://localhost:11434/api/generate",<|"Body"->ExportString["model"->"tinyllama","stream"->False,"temperature"->1,"prompt"->prompt,"JSON"],"Headers"-><|"Content-Type"->"application/json","Authorization"->"Bearer no-key"|>|>]]},Dataset[ImportString[result["Body"],"RawJSON"]]];
We have:
Uau! This response has lots of hallucinations, mailing about going from NYC to Paris by train and car!! This is a tiny model any way...
Now with a function to highlight the AI generated text in orange:
In the classic fairytale "Alice's Adventures in Wonderland" by Lewis Carroll, the main character, Alice, encounters the enigmatic and mysterious White Rabbit while she wanders down the rabbit hole. The White Rabbit is a rabbit who has a habit of running through the garden, often at breakneck speed. The Rabbit's behavior and appearance are quite unusual for a rabbit, as they tend to move slowly, gracefully, and silently. Alice becomes fascinated by this uncharacteristic behavior and decides to follow the White Rabbit into Wonderland.
As she follows the Rabbit down the garden path, the White Rabbit's increasing speed begins to cause a bit of confusion for Alice. She notices that the Rabbit is running faster than ever before, almost as if he is about to "go all out." This causes Alice to feel a little uneasy and wonders why the White Rabbit is so fast.
The White Rabbit's sudden urgency and speed become increasingly frantic and anxious. He is clearly rushing through Wonderland, trying to get somewhere quickly but also being careful not to cause too much damage in the process. The White Rabbit is in a hurry, and Alice wonders why this behavior is so unusual for rabbits.
Despite her curiosity, Alice's initial fascination with the White Rabbit's running style eventually leads her down an unexpected path. She becomes lost in Wonderland due to the Rabbit's sudden, reckless speed, and she has to rely on herself and her wits to find her way back to the Ground.
As she follows the Rabbit down the garden path, the White Rabbit's increasing speed begins to cause a bit of confusion for Alice. She notices that the Rabbit is running faster than ever before, almost as if he is about to "go all out." This causes Alice to feel a little uneasy and wonders why the White Rabbit is so fast.
The White Rabbit's sudden urgency and speed become increasingly frantic and anxious. He is clearly rushing through Wonderland, trying to get somewhere quickly but also being careful not to cause too much damage in the process. The White Rabbit is in a hurry, and Alice wonders why this behavior is so unusual for rabbits.
Despite her curiosity, Alice's initial fascination with the White Rabbit's running style eventually leads her down an unexpected path. She becomes lost in Wonderland due to the Rabbit's sudden, reckless speed, and she has to rely on herself and her wits to find her way back to the Ground.
And now lest make the same question for GPT-4 model to compare...
Of course, GPT-4 gives a better response in comparison with tinyllama model.
But I expected for stuff like: "The rabbit was late for the tea party"
But I expected for stuff like: "The rabbit was late for the tea party"
Ai gerado de texto pode hallucinar bastante!
Ai gerado de texto: O texto gerado por um computador pode Hallucinar bastante!
No good! The text was translated to just Portuguese, and no good translation anyway, it would be such as:
"O texto gerado por IA pode alucinar bastante"
"O texto gerado por IA pode alucinar bastante"
Lest check this translation task by GPT-4
It is interesting to work with those tiny models, with appropriate use case.
To install Ollama on Linux:
Ollama has many LLM models, those bigger and more sophisticated, and others more specialized and simple.
References:
References:
◼
◼
◼
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Calling local LLM model tinyllama with Olhama from Wolfram Language
by Daniel Carvalho
Wolfram Community, STAFF PICKS, August 12, 2024
https://community.wolfram.com/groups/-/m/t/3245036
by Daniel Carvalho
Wolfram Community, STAFF PICKS, August 12, 2024
https://community.wolfram.com/groups/-/m/t/3245036