In this follow-up to my original post, Recipe management with SemanticSearchIndex and retrieval-augmented generation (RAG), I use semantic indexing and feature space plots to explore clusters of ingredients and cooking methods within a large collection of recipes. I also build LLM tools and use dynamic prompt generation to engage with my recipes using natural language:
◼
Which of my recipes are high in antioxidants?
◼
Show three of my recipes that bake in less than 45 minutes.
◼
What are the most versatile vegetables in my recipe collection?
Retrieval-augmented generation (RAG)
Retrieval-augmented generation (RAG)
The first step in RAG is to retrieve relevant information from an external data source, such as a database or document. The large language model (LLM) will use this external knowledge to augment its own internal knowledge.
Import the recipes
Import the recipes
Import a PDF file of 1,500 publicly available recipes as a list of strings. Each string is the plain text content of a page in the file. The document length is almost 1,700 pages:
In[]:=
recipePagesAF=Import["https://www.marines.mil/Portals/1/Publications/MCO%20P10110.42B.pdf","PagePlaintext"];Length[recipePagesAF]
Out[]=
1691
Extract the pages that contain recipes, and count the number of recipes:
In[]:=
recipes=Select[recipePagesAF,StringMatchQ[#,___~~"Method"~~___]&];Length[recipes]
Out[]=
1539
Get recipe instruction sentences
Get recipe instruction sentences
Write a function to retrieve the instruction sentences from the recipes:
In[]:=
getInstructionSentences[recipe_]:=Module[{recipeTitle,str1,str2,str3,l1},recipeTitle=StringTrim[StringSplit[recipe,"\n"][[2]]];If[StringMatchQ[recipeTitle,(_?UpperCaseQ|" "|"("|")")...]&&Not[StringMatchQ[recipeTitle,___~~"INDEX"~~___]],str1=StringReplace[recipe,___~~"Method"~~instructions_:>instructions];str2=StringTrim[StringReplace[str1,"\n "~~DigitCharacter:>""]];str3=StringReplace[str2,"F. "~~n_:>If[StringMatchQ[n,LetterCharacter]&&UpperCaseQ[n],"F. "<>n,"F "<>n]];l1=DeleteCases[StringReplace[StringTrim[StringSplit[str3,"."|";"]],"\n":>" "],""];recipeTitle->l1,{}]]
Check the result by requesting the 214th recipe in the dataset:
In[]:=
getInstructionSentences[recipes[[214]]]
Out[]=
LONG GRAIN AND WILD RICE{Combine rice mix, water and salad oil,bring to a boil,Stir occasionally,Cover tightly,simmer 20 to 25 minutes,DO NOT STIR,Remove from heat,transfer to shallow serving pans,CCP: Hold for service at 140 F or higher}
Add label
Add label
Add the recipe name to each sentence as a label:
In[]:=
makeLabeledSentences[name_->instructions_]:={#->#<>" (from: "<>name<>")"}&/@instructionsmakeLabeledSentences[{}]:={}allLabeledSentences=Flatten[makeLabeledSentences[getInstructionSentences[#]]&/@recipes];
Count the labeled sentences:
In[]:=
allLabeledSentences=Flatten[makeLabeledSentences[getInstructionSentences[#]]&/@recipes];Length[allLabeledSentences]
Out[]=
19203
Check a randomly selected example:
In[]:=
RandomChoice[allLabeledSentences]
Out[]=
add to ingredients in mixer bowladd to ingredients in mixer bowl (from: JALAPENO CORN BREAD)
Semantic indexing
Semantic indexing
During semantic indexing, the text data retrieved from the recipes is converted into embeddings. Embeddings are numerical vector representations of the text. They capture the semantic meaning and relationships between words and are stored in a vector database.
When I submit a query, it also is transformed into an embedding, using the same model that was used for the recipes. The semantic search index calculates the similarity between my query’s embedding and those in the vector database, to identify recipes that are semantically related to my query. After retrieving the relevant recipes based on their semantic similarity, the text of these recipes is provided to the LLM, for it to generate a coherent and informed response for me.
When I submit a query, it also is transformed into an embedding, using the same model that was used for the recipes. The semantic search index calculates the similarity between my query’s embedding and those in the vector database, to identify recipes that are semantically related to my query. After retrieving the relevant recipes based on their semantic similarity, the text of these recipes is provided to the LLM, for it to generate a coherent and informed response for me.
Create semantic search index of instruction sentences
Create semantic search index of instruction sentences
With the pre-processing steps complete, create a semantic search index of the instruction sentences:
In[]:=
ssi=CreateSemanticSearchIndex[allLabeledSentences]
Out[]=
SemanticSearchIndex
Use the index to search for instructions semantically similar to “sauté onions and minced garlic in olive oil”:
In[]:=
StringSplit[SemanticSearch[ssi,"sauté onions and minced garlic in olive oil",MaxItems->6],"(from:"][[All,1]]//Column
Out[]=
Saute garlic and onions in shortening, salad oil, or olive oil until tender |
Saute garlic and onions in salad oil or olive oil until tender |
Saute onions in shortening, salad or olive oil until tender |
Saute onions and peppers in salad oil or olive oil until tender |
Saute onions and garlic powder in butter or margarine until tender |
Combine sauteed onions and garlic with tomatoes, tomato paste, water, bay leaves, oregano, basil, salt, sugar and thyme |
Get complete recipes
Get complete recipes
Write a function to retrieve the complete instructions for each recipe, rather than the individual sentences:
In[]:=
getCompleteRecipe[recipe_]:=Module[{recipeTitle,str1,str2,str3},recipeTitle=StringTrim[StringSplit[recipe,"\n"][[2]]];If[StringMatchQ[recipeTitle,(_?UpperCaseQ|" "|"("|")")...]&&Not[StringMatchQ[recipeTitle,___~~"INDEX"~~___]],str1=StringReplace[recipe,___~~"Method"~~instructions_:>instructions];str2=StringTrim[StringReplace[str1,"\n "~~DigitCharacter:>""]];str3=StringReplace[str2,"F. "~~n_:>If[StringMatchQ[n,LetterCharacter]&&UpperCaseQ[n],"F. "<>n,"F "<>n]];recipeTitle->str3,{}]]
Check an example from the 308th recipe in the dataset:
In[]:=
getCompleteRecipe[recipes[[308]]]
Out[]=
CARAMEL FROSTINGMelt butter or margarine. Add brown sugar; mix thoroughly while mixing at low speed. Cook over low heat 2 minutes; stir constantly.Combine milk and water. Add to butter mixture. Bring mixture to a boil; stir constantly. Remove from heat.Pour into mixer bowl; cool 10 minutes.Add powdered sugar gradually while mixing at low speed. Mix 2 minutes at medium speed until smooth.Spread immediately on cooled cakes.
Flatten the structure of the complete recipe instructions:
In[]:=
recipeList=Flatten[getCompleteRecipe/@recipes];
As before, write a function to add the recipe name as a label:
In[]:=
makeLabeledRecipe[name_->recipe_]:=recipe->namemakeLabeledRecipe[{}]:={}
Count the number of labeled recipes, then check an example. The function successfully pairs the name with the recipe:
In[]:=
allLabeledRecipes=Flatten[makeLabeledRecipe/@recipeList];Length[allLabeledRecipes]
Out[]=
1355
In[]:=
allLabeledRecipes[[230]]
Out[]=
Place thawed eggs in mixer bowl. Using wire whip, beat just enough to thoroughly blend.Lightly spray griddle with non-stick cooking spray. Pour 1/3 cup egg mixture for individual omelets on 325 F griddle.Cook until bottom is golden brown. DO NOT STIR. If necessary, gently lift cooked portion with a spatula to permit uncooked mixture to flow underneath.Dice ham. Sprinkle about 1 tablespoon cheese and 1 tablespoon ham over eggs when partially set. Continue cooking until eggs are set and well done. CCP: Internal temperature must reach 145 F or higher for 15 seconds, 155 F for fresh shell eggs.Fold omelet in half or into thirds making a long oval shaped omelet. CCP: Hold for service at 140 F or higher.HAM AND CHEESE OMELET
Create semantic search index of complete recipes
Create semantic search index of complete recipes
Create a semantic search index of the complete recipes, then get the embeddings and recipe names out of the new index:
In[]:=
ssiR=CreateSemanticSearchIndex[recipeList]
Out[]=
SemanticSearchIndex
In[]:=
embeddingsAndRecipeNames=Transpose[Normal[Values[ssiR[{"Embeddings","Items","Label"}]]]];
Check an example:
In[]:=
Short[#,15]&/@embeddingsAndRecipeNames[[108]]
Out[]=
{{-0.0545514,-0.172574,0.252702,0.456959,0.0198199,-0.145371,0.573816,-0.0884567,0.450537,-0.265388,0.0892962,-0.18615,0.0452482,-0.121872,-0.215391,0.294314,0.351576,0.328092,-0.22933,-0.285209,0.355514,-0.325738,0.529487,0.252395,0.0296306,-0.382642,0.0390401,-0.131995,-0.126321,-0.346504,0.1635,0.370888,-0.0146556,-0.143263,-0.10354,-0.533598,0.776071,-0.318163,-0.0129879,-0.305083,0.0520585,0.161228,-0.121271,0.25483,-0.313478,-0.0663724,0.512577,-0.161221,288,0.269202,0.19336,0.397854,0.289116,-0.0800695,0.252973,0.303876,0.610799,0.193023,0.0701957,-0.0730745,0.140893,0.299371,0.501921,0.375226,0.0567098,-0.276038,-0.265958,-0.378837,-0.241374,-0.0656777,0.275363,-0.233792,-0.0502023,0.04043,-0.0770415,0.642554,0.0711969,-0.0434791,-0.152965,-0.196337,0.0197462,-0.263013,0.191909,-0.175301,0.434488,0.132632,0.7821,-0.0486851,-0.28542,0.387303,-0.314571,0.0293699,-0.212707,0.0672505,-0.0442281,0.184654,-0.254047},OATMEAL RAISIN MUFFINS,In mixer bowl, sift together flour, sugar, milk, baking powder, salt, and cinnamon. Add rolled oats and raisins. Mix at low speed for 1 minute or until blended.Add warm water, eggs, applesauce, salad oil, and vanilla; mix at low speed until dry ingredients are moistened about 15 seconds; scrape down sides and bottom of mixer bowl; continue to mix at low speed another 15 seconds. DO NOT OVER MIX. Batter will be lumpy.Lightly spray muffin cups with non-stick cooking spray. Fill each muffin cup 2/3 full.Using a convection oven, bake at 350 F 23 to 26 minutes with open vent, fan turned off the first 10 minutes, and then low fan. Remove muffins from oven and let cool.}
Feature space plots
Feature space plots
Feature space plots are visual representations of high-dimensional data projected into two or three dimensions. Use feature space plots to visualize clusters of semantic similarities. In recipes, these plots can reveal groupings based on similar cooking methods or ingredient profiles.
Visualize recipe clusters
Visualize recipe clusters
With the embeddings and recipe names, create 2D and 3D feature space plots of all recipes in the dataset. First, write a function that streamlines the Tooltip function to display only the label:
Create a 2D feature space plot, and use the tooltip to view the recipe name for each plot point:
The plot shows the outer, well-defined clusters include potato dishes, baked chicken, soups, salad dressings, cakes and cookies.
With the same embeddings, create a 3D plot of the recipes:
Rotating the 3D plot provides new views of the recipe dataset, such as the proximity of fruit-based recipes like pies, turnovers, drinks and gelatins.
Select and embed only the chicken recipes, then evaluate the clusters:
Plot the prevalence and proximity of the chicken, beef, fish and turkey recipes:
The beef recipes are mostly clustered together, demonstrating their semantic similarity, but the chicken recipes are more varied in their ingredients and preparation.
More clusters to explore
More clusters to explore
◼
Visualize clusters of cooking methods in your recipes, such as bake, broil, roast, fry and steam
◼
Explore clusters of salads, soups, burgers and sandwiches
◼
Compare clusters of recipes that use olive oil versus canola oil
Get ingredients
Get ingredients
Write a function to get all recipe ingredients, then check an example:
Create semantic search index of ingredients
Create semantic search index of ingredients
Create a semantic search index of the recipe ingredients:
Create LLM tools to query the recipe database
Create LLM tools to query the recipe database
Ingredients, nutrients and health benefits
Ingredients, nutrients and health benefits
Using the semantic search index of ingredients, create an LLM tool to query for recipes in the database with particular ingredients, nutrients or health benefits:
Cooking actions and equipment
Cooking actions and equipment
Create an LLM tool to query for recipes in the database that use certain cooking actions, methods or equipment:
Dynamic prompt generation
Dynamic prompt generation
Another way to augment the LLM’s response is through dynamic prompt generation. LLMPromptGenerator may incorporate additional context or modify the original query for a more effective interaction with the LLM:
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Part 2: Recipe management with SemanticSearchIndex and RAG
by Gay Wilson
Wolfram Community, STAFF PICKS, March 4, 2025
https://community.wolfram.com/groups/-/m/t/3404408
by Gay Wilson
Wolfram Community, STAFF PICKS, March 4, 2025
https://community.wolfram.com/groups/-/m/t/3404408