On its DevDay OpenAI announced new powerful features around GPT including large updates to the capabilities of the API. In this post I will focus on general GPT completions, DALL-E, GPT-4-vision, and Text-To-Speech. As the abilities of GPT are changing at a great pace, the flexibility that HTTPRequest offers is too great to ignore.
I will assume that the OpenAI key is already in the system credentials, which happens automatically if you use GPT within Mathematica.
In[]:=
myAPIKey=SystemCredential["OPENAI_API_KEY"];
There are more secure ways to implement the key retrieval and you should be safeguarding your key at all times.
For full disclosure, I have used a GPT driven approach to build the functions in this post. I showed GPT an API call via HTTPRequest that I had implemented for a different API. I then pointed it to examples on OpenAI’s website and asked it to adjust the HTTPrequest to interact with the respective model.
I make small adjustments and then sent the working functions back to GPT for improvement. It mainly suggested error handling and some additional functionality.
I am aware that these functions are simple and untested. Use at your own risk. However, they allow to immediately interact from within the Wolfram Language with GPT before more robust functions will be provided by Wolfram Research.
I make small adjustments and then sent the working functions back to GPT for improvement. It mainly suggested error handling and some additional functionality.
I am aware that these functions are simple and untested. Use at your own risk. However, they allow to immediately interact from within the Wolfram Language with GPT before more robust functions will be provided by Wolfram Research.
Definition of chat GPT call
Definition of chat GPT call
We start with the general function which will return a standard call go gpt-3.5-turbo-16k meta information. The function is rather simple. It is possible to change the “model” parameter to other completion type models e.g. gpt-4.
In[]:=
callChatGPT[myPrompt_,myAPIKey_]:=Module[{myRequest,myResponse},myRequest=HTTPRequest["https://api.openai.com/v1/chat/completions",<|Method->"POST","Body"->ExportString[{"model"->"gpt-3.5-turbo-16k","messages"->{<|"role"->"user","content"->myPrompt|>},"temperature"->0.7},"JSON"],"ContentType"->"application/json","Headers"->{"Authorization"->"Bearer "<>myAPIKey}|>];myResponse=URLExecute[myRequest]]
This function gives the raw output of the API. We then implement a second function to clean the output so that we can extract the answer only:
In[]:=
answerCleanup[out_]:=Association[Association[First[Association[out]["choices"]]]["message"]]["content"]
Here is an example of a call:
In[]:=
answerCleanup[callChatGPT["Why was Pluto downgraded from a planet to a dwarf planet?",SystemCredential["OPENAI_API_KEY"]]]
Out[]=
Pluto was downgraded from a planet to a dwarf planet by the International Astronomical Union (IAU) in 2006 due to a newly established definition of a planet. The IAU defined a planet as a celestial body that orbits the Sun, is spherical in shape, and has cleared its orbit of other debris. Pluto met the first two criteria, but it did not fulfill the third requirement of clearing its orbit. It shares its orbit with other objects in the Kuiper Belt, a region beyond Neptune that contains numerous small icy bodies. As a result, the IAU reclassified Pluto as a dwarf planet, a category that includes other similar objects in the solar system.This decision sparked some controversy and debate among scientists and the general public. Many people had grown up considering Pluto as the ninth planet, and the reclassification challenged that perception. Nonetheless, the IAU's decision was based on scientific criteria and the need to establish consistent definitions for celestial bodies.
Calling the Text-To-Speech API (TTS)
Calling the Text-To-Speech API (TTS)
To use the text-to-speech API we start with the following HTTPRequest
In[]:=
callTTS[myPrompt_,myAPIKey_]:=Module[{myRequest,myResponse},myRequest=HTTPRequest["https://api.openai.com/v1/audio/speech",<|Method->"POST","Body"->ExportString[{"model"->"tts-1","input"->myPrompt,"voice"->"alloy"},"JSON"],"ContentType"->"application/json","Headers"->{"Authorization"->"Bearer "<>myAPIKey}|>];myResponse=URLExecute[myRequest]]
This is an example call:
In[]:=
callTTS["Today is a wonderful day to build something people love!",myAPIKey]
Out[]=
I have not used the tts-1-HD version which provides even better sound quality.
Creating images via the DALL-E API
Creating images via the DALL-E API
The DALL-E-3 model helps to make rather realistic images. This function fetches the image
In[]:=
callDALLE[myPrompt_,myAPIKey_]:=Module[{myRequest,myResponse},myRequest=HTTPRequest["https://api.openai.com/v1/images/generations",<|Method->"POST","Body"->ExportString[{"model"->"dall-e-3","prompt"->myPrompt,"size"->"1024x1024","n"->1},"JSON"],"ContentType"->"application/json","Headers"->{"Authorization"->"Bearer "<>myAPIKey}|>];myResponse=Import[URLExecute[myRequest][[2,2,1,2,2]]]]
The following call produces (a different) image for each call:
callDALLE["A cute baby sea otter",myAPIKey]
GPT-vision
GPT-vision
Finally, we interact with GPT vision, which allows us to look at an image and have GPT interpret what it sees. We create the following HTTPRequest
In[]:=
callGPTVision[myTextPrompt_,myImageUrl_,myAPIKey_]:=Module[{myRequest,myResponse},myRequest=HTTPRequest["https://api.openai.com/v1/chat/completions",<|Method->"POST","Body"->ExportString[{"model"->"gpt-4-vision-preview","messages"->{<|"role"->"user","content"->{<|"type"->"text","text"->myTextPrompt|>,<|"type"->"image_url","image_url"->{"url"->myImageUrl}|>}|>},"max_tokens"->300},"JSON"],"ContentType"->"application/json","Headers"->{"Authorization"->"Bearer "<>myAPIKey}|>];myResponse=URLExecute[myRequest][[6,2,1,1,2,2,2]]]
The power of this is that zou can add and modify the parameters, e.g. max_tokens. This is an example call:
In[]:=
response=callGPTVision["What’s in this image?","https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",myAPIKey]
Out[]=
The image shows a wooden boardwalk extending through a lush green wetland or meadow. The boardwalk allows for a walking path over the dense vegetation, likely providing a means to explore the area without disturbing the natural habitat. The greenery is vibrant and appears to be well-illuminated by the sunlight, suggesting it may be a pleasant day. The sky is partly cloudy with a soft blue hue, which adds to the serene appearance of the scene. In the distance, the landscape is dotted with what appears to be bushes or small trees. This type of environment could be a habitat for various wildlife and serves as a peaceful, natural locale for visitors to enjoy the outdoors.
More Examples
More Examples
We will now look at examples some of which will consist of refined functions.
Thee Chat-GPT-4-turbo variant.
Thee Chat-GPT-4-turbo variant.
First we modify the first text completion function to work with the new gpt-4-1106-preview model. It is cheaper priced than the gpt-4 model and the keynote suggests that it is more powerful. This is the modified callChatGPT function.
In[]:=
callChatGPT[myPrompt_,myAPIKey_]:=Module[{myRequest,myResponse},myRequest=HTTPRequest["https://api.openai.com/v1/chat/completions",<|Method->"POST","Body"->ExportString[{"model"->"gpt-4-1106-preview","messages"->{<|"role"->"user","content"->myPrompt|>},"temperature"->0.7},"JSON"],"ContentType"->"application/json","Headers"->{"Authorization"->"Bearer "<>myAPIKey}|>];myResponse=URLExecute[myRequest]]
And this is an example call:
In[]:=
answerCleanup[callChatGPT["Why was Pluto downgraded from a planet to a dwarf planet?",SystemCredential["OPENAI_API_KEY"]]]
A larger TTS example
A larger TTS example
Here we produce and audio file for Alice in Wonderland. There is no modification to the callTTS function from above. It is interesting to see how realistic the voice reacts an even “mimics” different characters.
Refinements
Refinements
DALL-E refinement
DALL-E refinement
Next we look at (untested) refinements of the functions. After asking GPT to improve the functions this is an updated callDALLE function, which now handles errors:
The call is still the same:
Next refinement of the ChatGPT function
Next refinement of the ChatGPT function
ChatGPT suggests the following changes to the callChatGPT function and its cleaning function:
This is an example use of the callChatGPT function which only extracts the answer from the complete API reply, which includes the meta information.
Refinement the GPT-4-Vision function (as suggested by GPT)
Refinement the GPT-4-Vision function (as suggested by GPT)
The following suggestion has been suggested by GPT (and some small corrections by myself) to handle errors in the API reply:
The main problem I have with the function above is that it requires the image to be uploaded to some server and to be certain that it is in the right format. What if we have an image file on the local machine and we want to check out what is on it? GPT suggested to implement an upload function (which I needed to modify):
Once the function called a local photo is uploaded to the Wolfram Cloud sent to GPT-4-vision. After the response is received the image is deleted from the cloud.
Here is a typical function call:
Conclusion
Conclusion
In spite of these functions being very rudimentary, I enjoyed the experience of interacting with ChatGPT and “working together” on the implementation of calls to interact with the latest models by OpenAI.
I have benefited form suggestions of GPT, but needed occasionally to fix some parts of the code as talking was just not enough.
The functions above are not tested and will have bugs, but they will help Wolfram Language users to quickly interact with OpenAI’s new functionality. The can easily be adjusted to work in different ways, polling different models or for example adding the option of uploading more than one image to GPT-4-vision calls. The advantage is that when new features are added to GPT it might be relatively easy to modify these functions - perhaps with the help of GPT itself. Therefore, I am not sure whether it is worthwhile adding these functions to a paclet. I consider them too fluid and unstable at this point.
I have benefited form suggestions of GPT, but needed occasionally to fix some parts of the code as talking was just not enough.
The functions above are not tested and will have bugs, but they will help Wolfram Language users to quickly interact with OpenAI’s new functionality. The can easily be adjusted to work in different ways, polling different models or for example adding the option of uploading more than one image to GPT-4-vision calls. The advantage is that when new features are added to GPT it might be relatively easy to modify these functions - perhaps with the help of GPT itself. Therefore, I am not sure whether it is worthwhile adding these functions to a paclet. I consider them too fluid and unstable at this point.
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Direct API access to new features of GPT-4 (including vision, DALL-E, and TTS)
by Marco Thiel
Wolfram Community, STAFF PICKS, November 8, 2023
https://community.wolfram.com/groups/-/m/t/3062403
by Marco Thiel
Wolfram Community, STAFF PICKS, November 8, 2023
https://community.wolfram.com/groups/-/m/t/3062403