Abstract: This project aims to advance interactive tutoring by leveraging the capabilities of large language models (LLMs), thereby enhancing the learning experience for students. The proposed research integrates prompt engineering techniques, utilizing LLMs like GPT-3.5 and GPT-4, with the versatility of Wolfram coding programming to create comprehensive subject-sheets on any requested topic. Despite the progress made with LLMs such as GPT-4, challenges remain in generating complex codes, including interactive animations, as well as ensuring the inclusion of accurate and up-to-date scientific references. To address these issues, this project developed a system that generates a “subject-sheet” comprising an abstract, keywords, an introduction, interactive animations, and current scientific journal and book references relevant to each user-requested subject. To achieve this, we employed LLMs to generate textual content and utilized available demonstrations from the Wolfram website for the creation of animations. The most relevant demonstrations were selected using LLMs. References were obtained through a service connection between Wolfram and the CrossRef website. All collected data were curated and consolidated into a single generated notebook based on a predefined template.
1. Project Challenges
1
. Project ChallengesIn recent months, significant progress has been made in the utilization of large language models (LLMs), thanks in large part to solutions provided by OpenAI, such as the OpenAI API and ChatGPT services. Despite their remarkable ability to generate human-like content, several challenges remain unresolved. The generation of complex program code and the production of updated and authentic scientific references are two key issues that persist. LLMs often produce false information and fabrications, which is particularly problematic when it comes to programming code and scientific references. To address these challenges, Wolfram and OpenAI have collaborated to combine the strengths of Wolfram’s data acquisition capabilities with the power of LLMs. Wolfram boasts an extensive collection of over 12,800 interactive animations covering various scientific and engineering fields, along with access to more than 20 websites and services, including Wikipedia, CrossRef, and LinkedIn. The wide-ranging accessibility and the user-friendly nature of the Wolfram Language make it the optimal choice for this project.
1.1 Project components
1
.1 Project componentsOur project consists of three primary components:
◼
Optimal Prompt Selection: This involves identifying the most suitable prompt to retrieve precise answers for a specific user query. The output will be carefully curated to extract the title, abstract, keywords, and introduction relevant to the subject of the question. This step presents its own set of challenges and difficulties, as finding the ideal prompt may appear straightforward but proves to be challenging to overcome.
◼
Reference Import: Using the extracted keywords, the next step is to import the ten most significant references, comprising five journal articles and five books. The selection criterion for these references is based on the number of times they have been cited in the literature. This stage also presents its unique challenges, as identifying the most relevant and widely cited references is not a trivial task.
◼
Interactive Demonstrations Selection: The final step involves selecting the four most relevant interactive demonstrations from a vast collection of over 12,800 interactive animations. Choosing the most suitable demonstrations poses its own difficulties, as determining relevance and significance requires careful evaluation. While these components may seem deceptively simple, they present inherent challenges that require meticulous attention to overcome.
1.2 LLM prompting challenges
1.
2
LLM prompting challengesThe utilization of LLMs, specifically the OpenAI API, presents several challenging aspects, which include:
◼
Optimal Prompt Selection: One of the main challenges is choosing the most suitable prompt to obtain the best solution, while considering token constraints. In the case of the free access OpenAI API GPT 3.5 version, the token limit is approximately 4000 tokens. Additionally, the generated data format often lacks coherence, making it crucial to efficiently select the most appropriate wording.
◼
Post-processing Coding: A post-processing step is always necessary to extract the desired information from the generated output. In our case, this involves retrieving a relevant title related to the user’s request, along with an abstract, keywords, and an introduction to the subject. Ensuring the extraction of these elements in a coherent and accurate manner poses a significant challenge.
Successfully addressing these challenges is crucial for leveraging the full potential of LLMs and obtaining the desired results .
1.3 Journal and Books references
1.3 Journal and Books references
In the scientific community, most if not all indexing services, such as IEEExplore, ScienceDirect, Google Scholar, are either limiting access to their database, or providing a limited access to a small part of it. For instance, Google provides API for all its services excluding Google Scholar.
Fortunately, thanks to Wolfram ServiceConnect, it is quite straight forward to query all indexed documents on the web by a Digital Object Identifier (DOI). The main remaining task is curating the data, in order to extract the most recent documentation, selecting the most significant ones, and then exporting them in a suitable form suitable for the user.
Fortunately, thanks to Wolfram ServiceConnect, it is quite straight forward to query all indexed documents on the web by a Digital Object Identifier (DOI). The main remaining task is curating the data, in order to extract the most recent documentation, selecting the most significant ones, and then exporting them in a suitable form suitable for the user.
1.4 Interactive Demonstration Data
1.4 Interactive Demonstration Data
Wolfram Demonstrations files can be accessed using Entities, providing a seemingly straightforward way to import the necessary information. However, during our implementation, we encountered a significant hurdle. More than 4000 demonstrations were found to be missing vital information or poorly formatted, rendering them unusable. This discovery posed a real challenge as we embarked on the task of locating and curating this collected data.
The second task involves generating local embeddings for the demonstration files, including their titles, captions, and descriptions. These embeddings are essential for efficiently searching and identifying the most relevant demonstrations based on the user’s specific request. By leveraging these embeddings, we can enhance the process of selecting the most significant demonstrations from the vast collection of Wolfram Demonstrations.
The second task involves generating local embeddings for the demonstration files, including their titles, captions, and descriptions. These embeddings are essential for efficiently searching and identifying the most relevant demonstrations based on the user’s specific request. By leveraging these embeddings, we can enhance the process of selecting the most significant demonstrations from the vast collection of Wolfram Demonstrations.
2. Project Deliverables
2
. Project Deliverables2.1 Connections setup - CrossRef and OpenAI
2.1 Connections setup - CrossRef and OpenAI
Access to OpenAI services can be achieved programmatically through the OpenAI API. To establish a connection with the API on Wolfram tools, it is necessary to first create a new account on the OpenAI website and generate an API Key [1]. The OpenAI API Key, which is obtained free of charge, grants users $5 that can be utilized to cover the expenses incurred by each request or interaction with the OpenAI services [2].
The cost is calculated per 1K Token, with an average of 1K Tokens corresponding to approximately 750 English words. To obtain more detailed information about Tokenization, please refer to the OpenAI Tokenize website [3].
CrossRef [4], the provider of the largest database indexed by Digital Object Identifier (DOI), offers free access to its entire indexed database, which can be downloaded or accessed via the CrossRef API [5]. Setting up an API Key is not required; however, it is advisable to limit the number of reference queries per minute to a reasonable level to avoid service interruption due to misuse.
The cost is calculated per 1K Token, with an average of 1K Tokens corresponding to approximately 750 English words. To obtain more detailed information about Tokenization, please refer to the OpenAI Tokenize website [3].
CrossRef [4], the provider of the largest database indexed by Digital Object Identifier (DOI), offers free access to its entire indexed database, which can be downloaded or accessed via the CrossRef API [5]. Setting up an API Key is not required; however, it is advisable to limit the number of reference queries per minute to a reasonable level to avoid service interruption due to misuse.
Open a windows form in order to add an OpenAPI Key:
openai=ServiceConnect["OpenAI","New"];
Setup a connection with CrossRef:
crossref=ServiceConnect["CrossRef"];
2.2 LLM Prompting - Abstract, Keywords and Introduction
2.2 LLM Prompting - Abstract, Keywords and Introduction
In this project, LLM plays a crucial role in two stages: firstly, it serves to offer comprehensive responses or explanations to user queries, and secondly, it is utilized for embedding Wolfram demonstration files.
To address the first stage, we have developed a new Wolfram Persona [6] called SubjectSheetPersona. This Persona enables the generation of essential details, such as the Title, Abstract, Keywords, and an Introduction, in the form of a JSON file [7].
To address the first stage, we have developed a new Wolfram Persona [6] called SubjectSheetPersona. This Persona enables the generation of essential details, such as the Title, Abstract, Keywords, and an Introduction, in the form of a JSON file [7].
Definition of a new persona SubjectSheetPersona:
In[]:=
SubjectSheetPersona=LLMFunction[StringTemplate["You are a helpful expert in science and engineering subjects and a professional in english writing. Your job is to answer the question `1`by generating text in a JSON format, that includes the following four elements:* An interesting title for the subject related to the question, defined as title.* A one paragraph Abstract that answers and explains the concept presented in the question, defined as abstract.* A list of five keywords related to the question subject, defined as keywords.* An advanced detailed explanation of the subject on the question while including mathematic equations in the same section when needed, defined as explanation.Generated text must have the following format:{\"Title\"-> title, \"Abstract\"-> abstract, \"Keywords\"-> keywords, \"Explanation\"-> explanation}"]]
Out[]=
LLMFunction
Subsequently, the created persona is capable of responding to and furnishing the necessary information based on a user’s request. In the following example, the user’s query is “Explain the Nyquist Sampling Theorem.” This query will be combined with the prompt mentioned earlier and transmitted through the OpenAI API to generate the ensuing response. It is not necessary to use polite language in formulating the prompt, as it may result in unfavorable or inadequate outputs.
Submitting of the prompt using new developed Persona:
In[]:=
LLMAnswer=SubjectSheetPersona["Explain the Nyquist Sampling Theorem"]
Below output looks like a JSON file, however, it includes mistakes in the formatting, so we can’t use the Wolfram Import function [6] to extract the data part by part.
Out[]=
{"Title": "Understanding the Nyquist Sampling Theorem", "Abstract": "The Nyquist Sampling Theorem is a fundamental concept in signal processing that establishes the minimum sampling rate required to accurately reconstruct a continuous-time signal from its discrete samples. This theorem states that in order to avoid aliasing, the sampling rate must be at least twice the highest frequency component present in the signal. By adhering to this principle, we can ensure that the original signal can be faithfully reconstructed without any loss of information.", "Keywords": ["Nyquist Sampling Theorem", "signal processing", "sampling rate", "aliasing", "reconstruction"], "Explanation": "The Nyquist Sampling Theorem, also known as the Nyquist-Shannon Sampling Theorem, provides a guideline for determining the minimum sampling rate necessary to capture all the information contained in a continuous-time signal. It is named after Harry Nyquist and Claude Shannon, who independently formulated this theorem in the early 20th century. To understand the Nyquist Sampling Theorem, let's consider a continuous-time signal, such as an audio waveform. This signal can be represented mathematically as a function of time, where the amplitude of the signal varies continuously. In order to process or transmit this signal digitally, we need to convert it into a discrete-time signal by sampling it at regular intervals.The Nyquist Sampling Theorem states that in order to avoid aliasing, which is the distortion or loss of information that can occur when a signal is undersampled, the sampling rate must be at least twice the highest frequency component present in the signal. This is often referred to as the Nyquist rate or the Nyquist frequency. Mathematically, if we denote the highest frequency component in the signal as 'f_max', then the Nyquist rate, 'f_s', can be calculated as:f_s = 2 * f_maxBy sampling the signal at a rate equal to or higher than the Nyquist rate, we can ensure that all the frequency components of the original signal are captured accurately. If the sampling rate is lower than the Nyquist rate, aliasing can occur, leading to the folding of higher frequency components into lower frequency ranges, resulting in distortion and loss of information.In practice, it is common to use a sampling rate that is significantly higher than the Nyquist rate to provide a safety margin and accommodate any unexpected frequency components. This is known as oversampling.In summary, the Nyquist Sampling Theorem is a crucial principle in signal processing that guides the selection of an appropriate sampling rate to accurately capture and reconstruct a continuous-time signal. By adhering to the Nyquist rate, we can avoid aliasing and ensure faithful representation of the original signal in the digital domain."}
In this particular scenario, we have devised a tailored post-processing script to extract individual elements separately. Although it is possible to send multiple OpenAI API requests for each element, this approach would result in fourfold expenses compared to the procedure we have implemented. This is due to the requirement of sending the SubjectSheetPersona prompt for each request.
Post-processing script of LLM output in order to generate individual parameters subject, abstract, keyword, and introduction:
In[]:=
subject=StringDelete[StringSplit[LLMAnswer,{"\"Abstract\":","\"Keywords\":","\"Explanation\":"}][[1]],{"\"",":","{",",","Title","\n"}];abstract=StringDelete[StringSplit[LLMAnswer,{"\"Abstract\":","\"Keywords\":","\"Explanation\":"}][[2]],{"\"",","}];keywords=StringDelete[StringSplit[LLMAnswer,{"\"Abstract\":","\"Keywords\":","\"Explanation\":"}][[3]],{"\"","[","]"}];introduction=StringDelete[StringSplit[LLMAnswer,{"\"Abstract\":","\"Keywords\":","\"Explanation\":"}][[4]],{"\"","}"}];
The example below demonstrates how we can extract the Abstract from the data generated by LLM.
In[]:=
StringSplit[LLMAnswer,{"\"Abstract\":","Keywords","Explanation"}][[2]]
" The Nyquist Sampling Theorem is a fundamental concept in signal processing that establishes the minimum sampling rate required to accurately reconstruct a continuous-time signal from its discrete samples. This theorem states that in order to avoid aliasing, the sampling rate must be at least twice the highest frequency component present in the signal. By adhering to this principle, we can ensure that the original signal can be faithfully reconstructed without any loss of information. "
2.3 Embedding of the Wolfram Demonstration Files Data
2.3 Embedding of the Wolfram Demonstration Files Data
The second application of LLM involves embedding data from demonstration files, which primarily includes the Title, caption, and keywords associated with each valid demonstration file. The following codes and comments outline the various steps involved in generating the embedded demonstration data.
OpenAPI connexion if not already done:
In[]:=
Needs["ChristopherWolfram`OpenAILink`"]
Definition of the OpenAPI User Key:
Applying the User OpenAI Key:
Downloading of GPT-2 NetModel for Tokenization:
Selection the NetEncoder for the Tokenization process:
Importing Titles of all demonstrations files using Wolfram EntityValue function:
Importing Keywords of all demonstrations files using Wolfram EntityValue function:
Concatenation of all three data {Caption, Title, keywords} in order to make one embedding request per demonstration file:
Submission of the OpenAI embeddings request of all gathered demonstration data using OpenAIEmbedding function:
Saving the received Embedding data in a local file, under the filename “EmbeddingDemonstrationVectorsFinal.mx”.
The embedding process is performed once and can be updated whenever the website incorporates new demonstration files.
2.4 Extraction of the Most Significant Demonstration Animations
2.4 Extraction of the Most Significant Demonstration Animations
The resulting embedding file can be shared and utilized by any user to aid in the selection of the most relevant demonstration files based on their specific request. The following section outlines the detailed steps required to achieve this objective.
Importing the embedding file into Wolfram environment [9]:
Defining locally the Nearest function, using NetModel GPT-2, consideration all embeddings of demonstrations files:
Selecting the four most relevant demonstration files, i.e., the nearest ones to the embedding of the user request:
2.5 Insertion of Interactive Demonstrations
2.5 Insertion of Interactive Demonstrations
In this phase, we have gathered the Entities from all the relevant demonstration files. The subsequent step involves post-processing this information to arrange the four demonstrations in a 2x2 grid format, which will then be inserted into the Subject-sheet notebook.
Definition of function InsertDemonstration, which Insert a demonstration Entity into a Notebook, including links to the title, author, and the manipulate code:
Inserting the four demonstrations interactive animations on a 2x2 Grid:
2.6 Extraction of most cited references - Journals and Books
2.6 Extraction of most cited references - Journals and Books
The extraction of the most significant references relies on the keywords generated by LLM, rather than the user-submitted query. This approach is adopted to prevent any potential errors on the part of the user.
Importing the 1000 most recent journals in relation with submitted keywords:
Importing the 1000 most recent books in relation with submitted keywords:
Defining a new post-processing function named authorToString in order to generate an authors in the following format: Given Name, Name:
Generating one reference line for each Journal, following the following format: author, Title, Publisher, Pages, Date. In addition only the 5 most cited Journal references are kept:
Generating one reference line for each Book, following the following format: author or Editor, Title, Publisher, Date. In addition, only the 5 most cited Book references are kept:
2.7 Generation of the Notebook
2.7 Generation of the Notebook
Once all the required data has been gathered and the “SubjectSheetNotebookTemplate.nb” notebook template has been developed, the process can be easily executed using the following code.
Definition of the author
Definition of he Notebook Template full path:
Generation of a new document with the same format as a template but includes all collected or generated data:
Saving the generated notebook and saving it under the filename “SubjectSheet.nb”.
The summer school encompassed a significant undertaking that warrants discussion in this concluding section. Specifically, an essential task involved embedding all Manipulate Demonstration codes, which went beyond merely incorporating the titles and captions. This aspect was particularly crucial since the Entity service did not provide this functionality. Consequently, a substantial amount of data, exceeding 22GB, was retrieved, meticulously curated, and transformed into a JSON file with a size of 1.4GB. The purpose of this extensive process was to enable the utilization of the Manipulate codes for fine-tuning an LLM model or, at the very least, employing them in the context of few-shot prompts. By undertaking this endeavor, we have paved the way for enhanced functionality and expanded possibilities in future research and applications.
◼
Assisted Tutoring
◼
LLM Embedding
◼
Wolfram Demonstration Interactive Animations
◼
Scientific Journals and Books citations
◼
CrossRef
I would like to express my sincere gratitude to the numerous individuals who have made a significant impact on my journey during the Wolfram Summer School. I sincerely hope that I do not inadvertently overlook anyone as I extend my thanks. Firstly, I am immensely grateful to Pr. Craig Carter for his invaluable support, without which I would not have had the opportunity to attend this summer school. From the outset, he presented this challenging project and provided continuous remote guidance, despite the six-hour time difference. I have the utmost respect for him and his dedication to enhancing educational teaching techniques. I extend an equal measure of respect to Pr. Paul Abbott, whom I had the pleasure of meeting at the Wolfram Summer School in 2017 and again this year. He is a remarkable individual with exceptional expertise in editing Wolfram notebooks. I truly admire him, particularly for the effort he devotes to his students in crafting meticulously designed Wolfram notebooks.
I would like to offer my sincere thanks to Daniel Robinson for his unwavering support right from the beginning. He established our group chat and closely monitored my progress at every stage, providing assistance with coding on numerous occasions. His invaluable ideas and constructive feedback greatly enhanced my project proposal and final report. Daniel is unquestionably the go-to person for scientific report and presentation writing and editing. I extend my appreciation to Robert Nachbar for his informative lectures, which served as a comprehensive introduction to Wolfram Language. Despite their extended durations, often exceeding four hours without breaks, they were immensely valuable. I also want to express my gratitude to Mark Greenberg for his continuous support and availability. He proved extremely helpful, both online and in person, and demonstrated a commitment to resolving any issues, even late at night. I am thankful for his efficient management of all sessions and prompt sharing of lecture materials ahead of each lecture.
I must also extend my gratitude to Christian Pasquel, whose expertise in connecting to the CrossRef website proved invaluable. His ability to provide instantaneous solutions to queries is truly impressive. I am grateful for the unwavering support of the management team, including Mads Bahrami, Yi Yin, and John McNally, who have been instrumental since the very beginning, ensuring this journey becomes an unforgettable one. I extend my heartfelt thanks to Stephanie Bowyer, without whom I would not have gained access to Wolfram demonstrations data, and for her ongoing support.
Saving the best for last, I would like to acknowledge two remarkable individuals who have become close friends to me, Daniel Sanchez and Swastik Banerjee. I firmly believe that they should be recognized as co-authors of this work, as well as many other projects. Daniel’s expertise in Wolfram coding is truly exceptional. He not only provides solutions to queries but also offers valuable comments to aid in understanding and learning from the examples. He has generously addressed all my questions, even at 1:00 am. I am sincerely grateful for everything he has done for me, despite never having met him in person or seen a picture of him. Lastly, my dear Swastik, a truly wonderful person. On one occasion, we missed dinner because he insisted on finding a solution to my problem before attending. Many nights, we worked until 3:00 am on various challenging tasks related to curating data from a dataset of over 12,800 entries. I will genuinely miss Swastik and Daniel, as they have been instrumental in helping me when I was truly lost in coding. Thank you both for your exceptional support. I will never forget you.
I would like to offer my sincere thanks to Daniel Robinson for his unwavering support right from the beginning. He established our group chat and closely monitored my progress at every stage, providing assistance with coding on numerous occasions. His invaluable ideas and constructive feedback greatly enhanced my project proposal and final report. Daniel is unquestionably the go-to person for scientific report and presentation writing and editing. I extend my appreciation to Robert Nachbar for his informative lectures, which served as a comprehensive introduction to Wolfram Language. Despite their extended durations, often exceeding four hours without breaks, they were immensely valuable. I also want to express my gratitude to Mark Greenberg for his continuous support and availability. He proved extremely helpful, both online and in person, and demonstrated a commitment to resolving any issues, even late at night. I am thankful for his efficient management of all sessions and prompt sharing of lecture materials ahead of each lecture.
I must also extend my gratitude to Christian Pasquel, whose expertise in connecting to the CrossRef website proved invaluable. His ability to provide instantaneous solutions to queries is truly impressive. I am grateful for the unwavering support of the management team, including Mads Bahrami, Yi Yin, and John McNally, who have been instrumental since the very beginning, ensuring this journey becomes an unforgettable one. I extend my heartfelt thanks to Stephanie Bowyer, without whom I would not have gained access to Wolfram demonstrations data, and for her ongoing support.
Saving the best for last, I would like to acknowledge two remarkable individuals who have become close friends to me, Daniel Sanchez and Swastik Banerjee. I firmly believe that they should be recognized as co-authors of this work, as well as many other projects. Daniel’s expertise in Wolfram coding is truly exceptional. He not only provides solutions to queries but also offers valuable comments to aid in understanding and learning from the examples. He has generously addressed all my questions, even at 1:00 am. I am sincerely grateful for everything he has done for me, despite never having met him in person or seen a picture of him. Lastly, my dear Swastik, a truly wonderful person. On one occasion, we missed dinner because he insisted on finding a solution to my problem before attending. Many nights, we worked until 3:00 am on various challenging tasks related to curating data from a dataset of over 12,800 entries. I will genuinely miss Swastik and Daniel, as they have been instrumental in helping me when I was truly lost in coding. Thank you both for your exceptional support. I will never forget you.
1
.OpenAI API Keys: https://platform.openai.com/account/api-keys
2
.OpenAI API Pricing: https://openai.com/pricing
3
.OpenAIT Tokenizer: https://platform.openai.com/tokenizer
4
.CrossRef Website: https://www.crossref.org/
5
.CrossRef Dataset Torrent: https://academictorrents.com/browse.php?search=Crossref
6
.Wolfram Prompt Repository: https://resources.wolframcloud.com/PromptRepository/
7
.Wolfram JSON Function: https://reference.wolfram.com/language/ref/format/JSON.html
8
.Wolfram Demonstration Project, https://demonstrations.wolfram.com/
9
.InsertDemonstration is a function [beta version], developed by Paul Abbott and will be included in the Wolfram Function Repository: https://resources.wolframcloud.com/FunctionRepository/
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Fusing prompt engineering and Wolfram coding for enhanced tutoring
By Ghassane Aniba
Wolfram Community, STAFF PICKS, July 13, 2023
https://community.wolfram.com/groups/-/m/t/2960330
By Ghassane Aniba
Wolfram Community, STAFF PICKS, July 13, 2023
https://community.wolfram.com/groups/-/m/t/2960330