This essay examines how Anthropic’s AI assistant Claude understands and generates code written in the Wolfram Language. It discusses Claude’s ability to interpret Wolfram Language code, provides practical examples of how Claude can be used as a computational tool, and explores the current limitations of Claude’s code comprehension.
Introduction: How LLMs work
Introduction: How LLMs work
(Autoregressive) Large Language Models (LLMs) are machine learning models that are trained on large amounts of text data to learn patterns and relationships between words and phrases. The models ingest massive amounts of text, such as Wikipedia articles, news data, books and more. As the models read through all this data, they discover the statistical and semantic relationships between words and groups of words. Over time, the models build up an understanding of the actual meaning of different word combinations and sequences in a language.
Introducing Claude
Introducing Claude
Claude is what’s known as an instruction-tuned large language model, because it has been fine-tuned to take more account of user instructions in order to produce text that is highly aligned with humans. This model has been specially trained in a chat-like environment, making it much more user-friendly for ordinary people.
Interacting with Claude in the Wolfram Language
Interacting with Claude in the Wolfram Language
In[]:=
externalSession=StartExternalSession["Python"]
Out[]=
ExternalSessionObject
◼
Note: You must first install the Python Anthropic library via PIP. Note that Wolfram may refer to a different Python installation than your default one.
◼
Note: You need to provide your Claude API key, if you don't have it, you need to request access to it.
apiKeys=<|"claude"->"YourClaudeAPIKey"|>;
We can easily interact with Claude through its Python package.
In[]:=
ExternalEvaluate[externalSession,{"import anthropic",ToString@StringForm["c = anthropic.Client('`1`')",apiKeys["claude"]]}];
In[]:=
Claude[prompt0_String,maxTokens0_:128,stopSequences0_List:{}]:=Module[{prompt=prompt0,maxTokens=maxTokens0,command},(command=StringTrim["c.completion(prompt='''"<>prompt<>"''', model='claude-v1', max_tokens_to_sample="<>ToString[maxTokens]<>")"];ExternalEvaluate[externalSession,command])]
Let’s try it out.
In[]:=
Claude["\n\nHuman: Hi, who are you?\n\nAssistant:"]
Out[]=
completion Hello! I'm Claude, an AI assistant created by Anthropic.,stopHuman:,stop_reasonstop_sequence,truncatedFalse,log_id2f5d56ad82d4555ee1c61101d3b04004,modelclaude-v1,exceptionNull
What is this strange prompting structure? To work properly, Claude needs us to follow a specific prompting structure.
\n\nHuman: Hey Claude, what do you know about Rule 30?\n\nAssistant:
To which, it will respond with:
“Rule 30 is a one-dimensional cellular automaton rule discovered by Stephen Wolfram. In it:
A cell’s next state depends on its current state and the states of its two neighbors
There are 8 possible neighborhood configurations
The rule specifies the next state for each neighborhood configuration
[...]”
“Rule 30 is a one-dimensional cellular automaton rule discovered by Stephen Wolfram. In it:
A cell’s next state depends on its current state and the states of its two neighbors
There are 8 possible neighborhood configurations
The rule specifies the next state for each neighborhood configuration
[...]”
Using Claude for Text2Code
Using Claude for Text2Code
Language models like Claude cannot fully exploit the power of the Wolfram Language, which is both its greatest strength - the sheer number of algorithms built into it - and its weakness when interacting with LLMs. The Wolfram Language has so many functions that current LLMs tend to forget or misuse them, or hallucinate plausible but impossible functions.
To solve this, we need to employ a clever prompting strategy using a knowledge base of text-to-code instructions, so that Claude can learn how to use the most appropriate and best suited functions for each task. Claude does not need to have the exact answer to the query in this knowledge base, as it is simply there to better guide Claude in the code generation process. Similar approaches could be applied by retrieving informative code snippets from the Wolfram Language documentation.
You can think of it as a small Wolfram|Alpha-like application, the LLM translates instructions into a precise computational form and then proceeds to compute the result, the same exact approach could be used to interact with the Wolfram Knowledgebase.
We can start by defining a base prompt, which will instruct Claude about its task.
In[]:=
baseText2CodePrompt="\n\nHuman: Translate the human request into working Wolfram Language (Wolfram Mathematica) code. For each instruction/question, you should limit yourself to writing Wolfram Language code to fulfil the task.\n\n";
Next, we will build our knowledgebase (which could potentially be expanded at runtime).
In[]:=
preprocess=(Keys[#]->#)&;
In[]:=
codeKnowledgebase=preprocess/@{"Calculate the derivative of x^2"->"D[x^2, x].","Solve the equation x^2 + 3x + 2 == 0"->" Solve[x^2 + 3x + 2 == 0, x].","Plot the function y = x^2"->"Plot[x^2, {x, -5, 5}].","Find the limit of (x^2 - 1)/(x - 1) as x approaches 1"->"Limit[(x^2 - 1)/(x - 1), x -> 1].","Integrate the function x^2 from 0 to 1"->"Integrate[x^2, {x, 0, 1}].","Create a list of the first 10 prime numbers"->"Prime[Range[10]].","Find the 5th Fibonacci number"->"Fibonacci[5].","Generate a list of 10 random integers between 1 and 100."->"RandomInteger[{1, 100}, 10]","Sort a list of numbers in descending order."->"Sort[{5, 2, 8, 1, 9}, Greater]","Find the mean of a list of numbers."->"Mean[{5, 2, 8, 1, 9}]","Find the median of a list of numbers."->"Median[{5, 2, 8, 1, 9}]","Find the mode of a list of numbers."->"Commonest[{5, 2, 8, 1, 9}]","Find the standard deviation of a list of numbers."->"StandardDeviation[{5, 2, 8, 1}]","List me all of the image on the https://www.wolfram.com/ webpage."->"Import[\"https://www.wolfram.com/\",\"Images\"]","Find the standard deviation of a list of numbers."->"StandardDeviation[{5, 2, 8, 1}]"};
To retrieve relevant examples from the knowledgebase, we can perform a semantic search on the keys of our associations using the Nearest function coupled with a language encoder.
In[]:=
languageEncoder=NetAppend[NetModel["DistilBERT Trained on BookCorpus and English Wikipedia Data"],SequenceLastLayer[]]
Out[]=
NetChain
To test Claude, we need to specify a task, so I’ve written my query as follows:
In[]:=
queryText2Code="Solve this simple equation: x^2 + 2x + 1 = 0.";
Another important point is to get a relevant example to compose the prompt.
In[]:=
nearestSnippets=Reverse[FeatureNearest[codeKnowledgebase,queryText2Code,2,FeatureExtractor->languageEncoder]]
Out[]=
{Plot the function y = x^2Plot[x^2, {x, -5, 5}].,Solve the equation x^2 + 3x + 2 == 0 Solve[x^2 + 3x + 2 == 0, x].}
By snapping all the pieces together into a string with the correct structure, we can get a prompt to send to Claude.
In[]:=
BuildText2CodePrompt[query0_String,snippets0_List:{}]:=Module[{query=query0,snippets=snippets0},(baseText2CodePrompt<>StringRiffle["Human: "<>Keys[#]<>"\n\nAssistant: "<>Values[#]&/@nearestSnippets,"\n\n"]<>"\n\nHuman: "<>StringTrim[query]<>"\n\nAssistant:")]
Here is our final prompt:
In[]:=
text2CodePrompt=BuildText2CodePrompt[queryText2Code,nearestSnippets]
Out[]=
Human: Translate the human request into working Wolfram Language (Wolfram Mathematica) code. For each instruction/question, you should limit yourself to writing Wolfram Language code to fulfil the task.Human: Plot the function y = x^2Assistant: Plot[x^2, {x, -5, 5}].Human: Solve the equation x^2 + 3x + 2 == 0Assistant: Solve[x^2 + 3x + 2 == 0, x].Human: Solve this simple equation: x^2 + 2x + 1 = 0.Assistant:
For example, now that we’ve “taught” Claude how to integrate, we can see if it generalises to new outputs. We might get more accurate results by lowering the temperature.
In[]:=
StringTrim[Claude[text2CodePrompt]["completion"]]
Out[]=
Solve[x^2 + 2x + 1 == 0., x]
Using Claude for Code Explanation
Using Claude for Code Explanation
Using Claude to explain code and provide context for how it works can be extremely helpful to programmers and users unfamiliar with a particular language. Code can often seem impenetrable without proper explanation and context, as it uses specialised syntax and logic that is not immediately apparent. By feeding code samples into Claude, it can analyse the overall structure, flow and purpose of the code, and then provide a plain-English summary of what each section does and how it relates to the broader function of the program.
As in our previous example, the key is to provide the model with an instruction and several examples so that it fully understands the task it is to perform.
In[]:=
baseCodeExplanationPrompt="\n\nHuman: Explain Wolfram Language code in plain English.\n\nHuman: Plot[x^2, {x, 0, 1}]\n\nAssistant: The following code creates a plot of the function x^2 with x ranging from 0 to 1.\n\nHuman: ";
We can then go on to specify a query to send to Claude.
In[]:=
queryCodeExplanation="Select[DictionaryLookup[], (# === StringReverse[#])&]";
In[]:=
codeExplanationPrompt=baseCodeExplanationPrompt<>queryCodeExplanation<>"\n\nAssistant:";
Here is our final prompt:
In[]:=
codeExplanationPrompt
Out[]=
Human: Explain Wolfram Language code in plain English.Human: Plot[x^2, {x, 0, 1}]Assistant: The following code creates a plot of the function x^2 with x ranging from 0 to 1.Human: Select[DictionaryLookup[], (# === StringReverse[#])&]Assistant:
As you can see, we’ve now received a response from the model.
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Exploring Anthropic's Claude for code assistance
by Nicolò Monti
Wolfram Community, STAFF PICKS, April 13 2023
https://community.wolfram.com/groups/-/m/t/2897774
by Nicolò Monti
Wolfram Community, STAFF PICKS, April 13 2023
https://community.wolfram.com/groups/-/m/t/2897774