Project Description
Project Description
The ArXiv repository is a vast and growing collection of academic papers that is now capturing a growing audience in industry, in particular for LLMs and AI. It’s also inspired several side projects, from Kaggle and similar platforms, aimed at making it easier to access.
It is a daunting corpus of documents to sift through wholesale. The URL identifiers, for one, use an indexing scheme (e.g., https://arxiv.org/abs/2407.06105 ) that has changed with growth and new categorization. The kinds of papers that compile or restate other research can have whole paragraphs dedicated to meticulous citation. And of course there are simply a lot of things. The chart above shows just the rate of growth in submissions from 1991, the year ArXiv was formed, through 2021.
ArXiv users could benefit even from integrating summaries of these documents with a Large Language Model (LLM). The natural language interaction offered by OpenAI, Claude, or similar services, make it plausible to review ArXiv content, without first learning how best to search it, and have the result cast in a conversational manner.
This project is an experiment in that direction. It uses a method of supplementing an LLM with additional data called Retrieval Augmented Generation (RAG). A number of paper abstracts are collected and included in a prompt sent to the LLM. The user agent asks the LLM to incorporate this data into its response.
The end goal is to incorporate this scheme into a Chat Notebook, such that a user agent can prompt the LLM for information that, unaided, it might otherwise respond to in tangential or even inaccurate ways. (It is after all the goal of a Generative AI system to keep a conversation going, even at the risk of regressing to standard search or calling up suspect data from the corpus.)
It is a daunting corpus of documents to sift through wholesale. The URL identifiers, for one, use an indexing scheme (e.g., https://arxiv.org/abs/2407.06105 ) that has changed with growth and new categorization. The kinds of papers that compile or restate other research can have whole paragraphs dedicated to meticulous citation. And of course there are simply a lot of things. The chart above shows just the rate of growth in submissions from 1991, the year ArXiv was formed, through 2021.
ArXiv users could benefit even from integrating summaries of these documents with a Large Language Model (LLM). The natural language interaction offered by OpenAI, Claude, or similar services, make it plausible to review ArXiv content, without first learning how best to search it, and have the result cast in a conversational manner.
This project is an experiment in that direction. It uses a method of supplementing an LLM with additional data called Retrieval Augmented Generation (RAG). A number of paper abstracts are collected and included in a prompt sent to the LLM. The user agent asks the LLM to incorporate this data into its response.
The end goal is to incorporate this scheme into a Chat Notebook, such that a user agent can prompt the LLM for information that, unaided, it might otherwise respond to in tangential or even inaccurate ways. (It is after all the goal of a Generative AI system to keep a conversation going, even at the risk of regressing to standard search or calling up suspect data from the corpus.)
RAG Development
RAG Development
Testing Setup
Testing Setup
A proof-of-technology demonstration in Wolfram Language is a straightforward affair, starting with connecting to the service and trying it out. There are a number of built-in services you can browse at will. (Note we can connect to both ArXiv and OpenAI this way):
In[]:=
$Services
Out[]=
{AI21,AlephAlpha,Anthropic,ArXiv,AWS,BingSearch,CharityEngine,ChemSpider,Cohere,CrossRef,DeepSeek,Dropbox,ElevenLabs,Facebook,Factual,FederalReserveEconomicData,Fitbit,Flickr,GoogleAnalytics,GoogleCalendar,GoogleContacts,GoogleCustomSearch,GoogleGemini,GooglePlus,GoogleSpeech,GoogleTranslate,Groq,HuggingFace,IBMQ,Instagram,IPFS,LinkedIn,MailChimp,MicrosoftTranslator,MistralAI,Mixpanel,MusicBrainz,OpenAI,OpenLibrary,OpenPHACTS,PaLM,PubChem,PubMed,Pushbullet,Reddit,RunKeeper,SeatGeek,SurveyMonkey,TogetherAI,Twilio,Twitter,Wikipedia,Yelp}
In[]:=
arxiv=ServiceConnect["ArXiv"]
Out[]=
ServiceObject
In this example we retrieve a single record from the Quantum Physics category to examine the format:
In[]:=
category=arxiv["CategorySearch",{"Query"->"quant-ph",MaxItems1}]
Out[]=
Connecting to the OpenAI service requires an API key, which can be added to the platform environment for ongoing use. There is no cost for persisting a connection indefinitely:
ServiceExecute is also available and useful for quick testing:
In[]:=
ServiceExecute[openAI,"Completion",{"Prompt"->"What I know about ArXiv is..."}]
Out[]=
{"text""\\nArXiv is an open-access digital repository for scientific papers, primarily in","index"0,"logprobs"Null,"finish_reason""length"}
Getting Data
Getting Data
The guardrails for a short-term project include drawing enough ArXiv data to demonstrate RAG to advantage, but without violating the terms of service. ArXiv encourages retrieving “descriptive metadata” for projects that aim to broaden access to the service, but there are still reasonable rate limits and terms to observe.
Queries through the ArXiv service connection were challenging. Retrieving records was inconsistent, possibly due to competing requests. There wasn’t enough time here to investigate the causes for error or perhaps denial of service. We ran the query below several times, sometimes successful, sometimes not:
Queries through the ArXiv service connection were challenging. Retrieving records was inconsistent, possibly due to competing requests. There wasn’t enough time here to investigate the causes for error or perhaps denial of service. We ran the query below several times, sometimes successful, sometimes not:
In[]:=
res=arxiv["Search",{"Query""quant-ph",MaxItems1000,"StartIndex"1}]//Normal
The response renders to a Dataset view by default, but as shown above in the single record test, it is cumbersome to inspect. We can view a record more easily with a Normal transformation. Each record is an association, further nested to accommodate multiple authors and link references.
In[]:=
Keys[res[[1]]]
Out[]=
{ID,URL,Updated,Published,Title,Summary,Author,Link,PrimaryCategory,Category}
In[]:=
res[[600]]
Out[]=
ID0009028v1,URLhttp://arxiv.org/abs/quant-ph/0009028v1,Updated,Published,TitleNew schemes for manipulating quantum states using a Kerr cell,SummaryIn this proceeding we describe various proposals of application of an high coefficient Kerr cell to quantum states manipulation, ranging from fast modulation of quantum interference, GHZ states generation, Schroedinger cats creation, translucent eavesdropping, etc.,Author{NameM. Genovese,NameC. Novero},Commentproc. of QCM&C,Link{Hrefhttp://arxiv.org/abs/quant-ph/0009028v1,Relalternate,Typetext/html,Titlepdf,Hrefhttp://arxiv.org/pdf/quant-ph/0009028v1,Relrelated,Typeapplication/pdf},PrimaryCategory{Quantum Physics},Category{Quantum Physics,Physics - Optics}
In[]:=
res[[600]][[6]]
Out[]=
In this proceeding we describe various proposals of application of an high coefficient Kerr cell to quantum states manipulation, ranging from fast modulation of quantum interference, GHZ states generation, Schroedinger cats creation, translucent eavesdropping, etc.
We also saw cases where the Summary text might not transform in a meaningful way:
The text as it appears on the website:
A consideration for future development is how best to treat special notations in a natural language processing (NLP) context.
Initial Process and Results
Initial Process and Results
From here the path forward would appear plain. Convert each summary into an embedding, populate the vector database, test against an LLM with and without the RAG data and observe the result. The approach ultimately broke down, for lack of time, a steep learning curve writing functional code in a notebook format.
We started out making use of an experimental feature in an upcoming release called SemanticSearchIndex. This object is backed by a VectorDatabase builtin. It uses SentenceBERT as its feature extractor (i.e., text parser) and a EuclideanDistance function. This means the embeddings and database population are handled in-house -- quite convenient.
The first model used 5000 ArXiv records from the Quantum Physics category. It was paired with an LLMTool that could match up to 15 items in the database against the query string:
We started out making use of an experimental feature in an upcoming release called SemanticSearchIndex. This object is backed by a VectorDatabase builtin. It uses SentenceBERT as its feature extractor (i.e., text parser) and a EuclideanDistance function. This means the embeddings and database population are handled in-house -- quite convenient.
The first model used 5000 ArXiv records from the Quantum Physics category. It was paired with an LLMTool that could match up to 15 items in the database against the query string:
An LLMEvaluator uses one or more LLMTool objects to build a ChatObject, which in turn supports ChatEvaluate:
As expected, the LLM response had the lazy cut-and-paste feel that tells us it is out of its depth. Next we tried to invoke our tool in the prompt:
We didn’t see the dialog box to indicate the tool was in play. What was surprising was the citation of specific ArXiv papers, including authors and links. In prior testing, OpenAI said it didn’t have access to the ArXiv site and recommended we browse it ourselves. It remains unclear by what influence this change took place.
Things Fall Apart, The Center Cannot Hold
Things Fall Apart, The Center Cannot Hold
The project work that followed was a rich proving ground for several hard lessons. A few of them are described.
At about 9000 records, populating the SemanticSearchIndex threw errors possibly related to unfriendly elements in the text, perhaps blank spaces or metacharacters in an empty field. We also questioned whether the StartIndex property worked properly, so we tested for overlap.
At about 9000 records, populating the SemanticSearchIndex threw errors possibly related to unfriendly elements in the text, perhaps blank spaces or metacharacters in an empty field. We also questioned whether the StartIndex property worked properly, so we tested for overlap.
Previous runs of confirmed the indexing was working.
We decided to try getting records in XML format that would be easier to clean and inspect, and included a batch process with a timed pause.
We decided to try getting records in XML format that would be easier to clean and inspect, and included a batch process with a timed pause.
We also decided to use OpenAI’s embedding service:
Conclusions
Conclusions
The RAG model is a well-established pattern with active investigations into more sophisticated and powerful approaches. This project could have easily been implemented in Python and integrated into the Wolfram platform, but there wasn’t much learn from that exercise. We took our chances with experimental tools, and a learning curve to the platform that was steeper than expected.
I have many thanks to give, beginning with the encouragement and support of my colleagues. This project started with Stephen Wolfram, who saw it as a “crunchy” undertaking but sees the value of this tool for this community. I would not have got half so far without Christopher Wolfram and John McNally for their support, ideas, and code I could not have written without much more time.
I have many thanks to give, beginning with the encouragement and support of my colleagues. This project started with Stephen Wolfram, who saw it as a “crunchy” undertaking but sees the value of this tool for this community. I would not have got half so far without Christopher Wolfram and John McNally for their support, ideas, and code I could not have written without much more time.
Cite This Notebook
Cite This Notebook