In a recent post I discussed how to install the google cloud sdk. There you will find a detailed description of steps you have to take to access the new Gemini-Pro model for completions. The model is, however, multi-modal. I will show how to use it for image analysis in another post. Today I want to show some of the Video analysis features of that particular model.
Basic functions
Basic functions
In the previous post, after installing the google cloud sdk, I -or more precisely ChatGPT- developed the following piece of code to get an access token:
In[]:=
getGeminiToken[]:= Module[{gcloudPath,command},gcloudPath="/Users/thiel/google-cloud-sdk/bin/gcloud";(*Set the command to get the access token*)command=gcloudPath<>" auth application-default print-access-token";(*Use RunProcess to execute the command with zsh*)accessTokenResult=RunProcess[{"zsh","-c",command},"StandardOutput"];(*Extract the access token*)accessToken=StringTrim[accessTokenResult];accessToken]
We will need this function later on. Next, we need to encode a video to base64 format so that we can submit it via the API. The following function will achieve that.
In[]:=
encodeVideoToBase64[videoPath_String]:=Module[{video,base64},video=BinaryReadList[videoPath];base64=Developer`EncodeBase64[video];base64]
Finally, this is a GPT generated function to analyse a video. The function will check whether the access token has already be created and otherwise create the token. It needs the path to the video and a query sting as inputs.
analyzeVideoWithGemini[videoPath_String,query_String]:=Module[{headers,jsonBody,apiUrl,httpResponse,responseContent,formattedResponse},(*Obtaintheaccesstoken*)If[!StringQ[accessToken],accessToken=getGeminiToken[]];(*Setuptheheaderswiththeaccesstoken*)headers={"Authorization"->("Bearer "<>accessToken),"Content-Type"->"application/json"};(*EncodethevideotoBase64*)base64Video=encodeVideoToBase64[videoPath];(*ConstructtheJSONbodyfortherequest*)jsonBody=ExportString[{"contents"->{"role"->"user","parts"->{{"inlineData"->{"mimeType"->"video/mp4","data"->base64Video}},{"text"->query}}},"generationConfig"->{"maxOutputTokens"->2048,"temperature"->0.4,"topP"->1,"topK"->32}},"JSON"];(*ConstructtheAPIURL*)apiUrl="https://us-central1-aiplatform.googleapis.com/v1/projects/mathematica-thiel/locations/us-central1/publishers/google/models/gemini-pro-vision:streamGenerateContent";(*MaketheAPIrequest*)httpResponse=URLExecute[HTTPRequest[apiUrl,<|"Method"->"POST","Headers"->headers,"Body"->jsonBody|>],"RawJSON"];(*Checkiftheresponsecontains'candidates'key*)If[!KeyExistsQ[Association@@httpResponse,"candidates"],Return[$Failed]];(*Extractthetextfromeachpartofthecontent*)responseContent=Flatten["text"/.("parts"/.("content"/.("candidates"/.httpResponse)))];(*Formattheresponse*)formattedResponse=StringJoin[Riffle[responseContent,"\n"]];(*Returntheformattedresponse*)formattedResponse](*Usage*)(*Youwouldusethefunctionlikethis:*)(*analyzeVideoWithGemini["/path/to/video.mp4","describe this video, sequence by sequence"]*)
The function will generate a header and jsonBody for the HTTP request. It will (hopefully) parse the response from the API server to create a more or less readable output. Let’s look at an Example.
Caminandes
Caminandes
We will first look at an example video, which comes as example data with Mathematica.
In[]:=
Video["ExampleData/Caminandes.mp4"]
Out[]=
I could copy the video, but I will just export it to my desktop.
In[]:=
Export["~/Desktop/Caminandes.mp4",Video["ExampleData/Caminandes.mp4"]]
Out[]=
~/Desktop/Caminandes.mp4
Let’s run the function with a fairly simple prompt.
In[]:=
analysisCaminandes=analyzeVideoWithGemini["~/Desktop/Caminandes.mp4","Describe this video in great detail. Write some sentences per sequence. At the end summrise what the video is about."]
Out[]=
The video shows a llama running along a road. The llama is running from a car. The llama is very scared. The car is getting closer and closer. The llama is about to be hit by the car. The llama jumps out of the way just in time. The car speeds off. The llama is safe.The llama is a wild animal. It is not used to being around cars. The llama was probably scared because it did not know what the car was. The llama is lucky that it was able to escape.The video is about a llama that is almost hit by a car. The video is funny because the llama jumps out of the way just in time. The video is also educational because it shows how dangerous it can be for wild animals to be around cars.
As you can see there is a short description of what the movie shows. Gemini adds some thoughts about the video and interprets the video. It also suggests that the video is educational and funny.
Example 2
Example 2
Let’s run the same example again, but this time with a slightly longer prompt.
In[]:=
analysisCaminandes=analyzeVideoWithGemini["~/Desktop/Caminandes.mp4","Describe this video in great detail. what characters do you see? How does the background look? Write a paragraph or two per sequence. It should read like a detailed plot for a movie. Look at all details. At the end summrise in 3 paragraphs what the video is about."]
Out[]=
The video begins with a long shot of a barren desert landscape. The only thing moving is a small, brown animal, which is running towards a road in the distance. As it gets closer, we can see that it is a llama. It is running with a determined look on its face, as if it is being chased.The llama reaches the road and looks both ways before crossing. It then continues running on the other side. The camera follows the llama as it runs along the road. The llama is running faster and faster, and it is starting to look exhausted. Suddenly, a car comes speeding down the road. The llama sees the car and tries to stop, but it is too late. The car hits the llama, and the llama goes flying through the air.The llama lands hard on the ground and lies there for a moment, stunned. Then, it slowly gets up and starts to limp away. The car that hit the llama does not stop. It just keeps going down the road.The llama walks for a long time, until it finally comes to a small oasis. The llama drinks some water from the oasis and then lies down in the shade of a tree. The llama is finally safe, but it is also very tired. It closes its eyes and falls asleep.The next morning, the llama wakes up and feels much better. It gets up and starts to walk again. The llama walks for a long time, until it finally comes to a small town. The llama enters the town and looks around. The llama sees a sign for a farm, and it walks towards the farm.The llama enters the farm and sees a farmer. The farmer looks at the llama and smiles. "Hello," says the farmer. "What are you doing here?""I'm looking for a home," says the llama."Well, you've come to the right place," says the farmer. "I'm always looking for help on my farm."The llama smiles. "Thank you," it says. "I'll be happy to help."The llama starts working on the farm, and it is very happy. It has finally found a home.The video is about a llama that is running away from something. It is hit by a car, but it survives. The llama then walks to a small town and finds a home on a farm.The llama is a very determined animal. It is able to overcome many obstacles in its journey to find a home. The llama is also very lucky. It is able to find a home with a farmer who is kind and welcoming.The video is a reminder that even in the face of great challenges, there is always hope. It is also a reminder that there are good people in the world who are willing to help others.
This is a very interesting output the first two paragraphs are reasonably close to the film. Interestingly, after the first two or three paragraphs the model seems to start to hallucinate. It talks about an oasis, a farmer and it makes up a dialog. It seems that it is doing “out-painting” with the story. I am not sure whether that is due to my prompt mentioning a “plot for a movie”.
Example 3
Example 3
The next example is from the documentation of TourVideo. The photo is from perhaps the most famous of the Solvay conferences, the fifth Solvay Conference on Physics, which was held from 24 to 29 October 1927:
As in the documentation we first detect the faces.
Then we highlight the faces in the image.
The function TourVideo then creates a video where the camera moves from one face to the next.
Let’s export that video:
The next step is to ask Gemini-Pro what it thinks of this.
Well, that is a bit disappointing. Let’s try a slightly different prompt:
That is interesting but far from the actual number of faces:
I do not know why this is, but perhaps the frame rate has something to do with it. The video tour moves quickly through the video and it is quite possible that one certain frames are actually analysed by Gemini to increase speed and decrease costs.
Example 4
Example 4
Mathematica has many very powerful tools built in. Let’s use the example data of cars driving by.
First in the Wolfram Language 12.1 VideoTimeSeries was introduce, which in 12.2 was superseded by VideoMapTimeSeries. Using ImageCases we can now look at how many cars are in the frames.
As you see at the beginning it counts 3-4 cars, but the first frame shows at least 8 cars. Let’s ask Gemini about the video.
I am not sure, but I count 13 cars, but there is a motorbike and a trailer. The trailer seems to be the only white “vehicle”. I can imagine that this functionality could be useful in principle.
Conclusion
Conclusion
As we already had a function to access Gemini Pro completions, ChatGPT had no problems building these functions to analyse videos.
As we have seen Gemini does at times hallucinate (or interpret prompts as to make up stories?), and it still has trouble counting faces on fast moving videos. I think that the car example was rather impressive as it cannot analyse simply frame by frame but needs to “follow” cars over multiple frames.
I hope that this functionality is useful for some of you. I would love to see further examples and ideas what to do with this.
As we have seen Gemini does at times hallucinate (or interpret prompts as to make up stories?), and it still has trouble counting faces on fast moving videos. I think that the car example was rather impressive as it cannot analyse simply frame by frame but needs to “follow” cars over multiple frames.
I hope that this functionality is useful for some of you. I would love to see further examples and ideas what to do with this.
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Video analysis with Gemini Pro
by Marco Thiel
Wolfram Community, STAFF PICKS, December 22, 2023
https://community.wolfram.com/groups/-/m/t/3089060
by Marco Thiel
Wolfram Community, STAFF PICKS, December 22, 2023
https://community.wolfram.com/groups/-/m/t/3089060