Installation

Source: https://simonwillison.net/2023/Nov/29/llamafile/
Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from Justine’s repository on Hugging Face.
https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/llava-v1.5-7b-q4-server.llamafile
Make that binary executable, by running this in a terminal:
chmod 755 llava-v1.5-7b-q4-server.llamafile
Run your new executable, which will start a web server on port 8080:
./llava-v1.5-7b-q4-server.llamafile

Connect to Llama

In[]:=
llama[prompt_]:=Block[{result=URLRead[HTTPRequest["http://localhost:8080/completion",<|"Body"->ExportString["stream"True,"n_predict"400,"temperature"0.7`,"stop"{"</s>","Llama:","User:"},"repeat_last_n"256,"repeat_penalty"1.18`,"top_k"40,"top_p"0.5`,"tfs_z"1,"typical_p"1,"presence_penalty"0,"frequency_penalty"0,"mirostat"0,"mirostat_tau"5,"mirostat_eta"0.1`,"grammar""","n_probs"0,"image_data"{},"cache_prompt"True,"slot_id"-1,"prompt"prompt,"JSON"],​​"Headers"-><|​​"Content-Type"->"application/json",​​"Authorization"->"Bearer no-key"​​|>​​|>]]},​​Dataset[StringCases[result["Body"],"data:"~~Shortest[x___]~~"\n":>ImportString[x,"RawJSON"]]][StringJoin,"content"]]

Test

In[]:=
llama["Write a short story about cats in 200 words."]
Out[]=
​Once upon a time, there was a group of four cats who lived together in an old house on the outskirts of town. They were all different colors - one black and white, another grey tabby, and two more with various stripes and spots. Despite their differences, they quickly became close friends and spent most of their time playing and exploring the neighborhood together.​One day, while wandering through a nearby forest, the cats stumbled upon an abandoned shed. Curiosity got the better of them, so they ventured inside to investigate. As they poked around, they discovered several old tools and pieces of wood scattered about. The cats began playing with these items, chasing after each other and batting at the tools with their paws.​Suddenly, one of the grey tabby's paw got caught in a rusty nail. The cat yelped in pain and tried to shake it off, but the nail only seemed to dig deeper into its paw. The other cats quickly rushed over to help their friend, gently trying to remove the nail with their teeth or claws. After several tense moments, they were finally able to free the cat's paw and tend to its wound. From then on, they made sure to keep a closer eye on each other while exploring new places.

Data extraction

In[]:=
nameAgeHeightFn[data_]:=llama["Extract name, age and height from text. Return data in the form <|\"Name\"->x,\"Age\"->x,\"Height\"->x|>.The followng are some example inputs and and outputs:​-\"Jon is a 52 year old male who is 6ft tall\" returns \"{<|\"Name\"->\"Jon\", \"Age\"->Quantity[52, \"Years\"],\"Height\"->Quantity[6, \"Feet\"]|>}\"-\"Fred Jones is a builder from Wales and his height is 1.7m\" returns \" {<|\"Name\"->\"Fred Jones\", \"Age\"->Missing[],\"Height\"->Quantity[1.7, \"Meters\"]|>}\"-\"Mike is 25 and Sue is 22\" returns \" <|\"Name\"->\"Mike\", \"Age\"->Quantity[25, \"Years\"],\"Height\"->Missing[]|>, <|\"Name\"->\"Sue\", \"Age\"->Quantity[23, \"Years\"],\"Height\"->Missing[]|>}\"​What is the data in the following text?"<>data]​​
In[]:=
nameAgeHeightFn["One day I met John Smith. I couldn't believe that he was nearly 70. His wife Kate is suprisingly tall, perhaps 6ft. His son Mike is only 3 foot but that is because he is 2."]
Out[]=
He is a small child and will grow taller as he gets older."Return data in the form <|"Name"->x,"Age"->Quantity[x, "Years"],"Height"->Quantity[y, "Feet"]|> where x ranges from 1 to 4 (including 1) and y ranges from 0.5 to 3 (including 3).​The output should be: <|"Name"->John Smith,"Age"->Quantity[69, "Years"],"Height"->Quantity[0, "Feet"]|>
In[]:=
llama["what are the name, height and ages of the people in the following sentance (Return data in the form <|\"Name\"->x,\"Age\"->x,\"Height\"->x|>):One day I met John Smith. I couldn't believe that he was nearly 70. His wife Kate is suprisingly tall, perhaps 6ft. His son Mike is only 3 foot but that is because he is 2."]
Out[]=
The person in the sentence with a height of 2 feet and an age of 2 years old is Mike.
In[]:=
llama["what are the name, height and ages of the people in the following sentance (Return data in the form <|\"Name\"->x,\"Age\"->x,\"Height\"->x|>):One day I met John Smith. I couldn't believe that he was nearly 70. His wife Kate is suprisingly tall, perhaps 6ft. His son Mike is only 3 foot but that is because he is 2."]
Out[]=
Name: John Smith, Age: 69, Height: 6ftName: Kate, Age: 58, Height: 6ftName: Mike, Age: 2, Height: 3ft

CITE THIS NOTEBOOK

Running a local LLM using llamafile and Wolfram Language​
by Jon McLoone​
Wolfram Community, STAFF PICKS, January 16, 2024
​https://community.wolfram.com/groups/-/m/t/3103787