Roee-san,
Thank you for the valuable information.
I also had a post about RAG about before. I used the Japanese Constitution (all 103 articles) as my task and attempted to extract three relevant clauses for a query using SemanticSearch.
However, it was never selected among the top three results.
I report the result of applying your method to my task.
The query and chunks are as follows .
In[]:=
query="What is the term of office for members of both houses of Congress as defined by the Constitution of Japan? Also, what article does it say?";​​chunksEN=
;​​Length@chunksEN
Out[]=
103
The following three chunks are required for the query :
In[]:=
chunksEN[[{45,46,102}]]//TableForm
Out[]//TableForm=
Article 45. The term of office of members of the House of Representatives shall be four years. However, the term shall be terminated before the full term is up in case the House of Representatives is dissolved.
Article 46. The term of office of members of the House of Councillors shall be six years, and election for half the members shall take place every three years.
Article 102. The term of office for half the members of the House of Councillors serving in the first term under this Constitution shall be three years. Members falling under this category shall be determined in accordance with law.
As a result of SemanticSearch, these three appear in the Top 6.
In[]:=
index=CreateSemanticSearchIndex[chunksEN];​​(candidatesSS=SemanticSearch[index,query,"Item"])//TableForm
Out[]//TableForm=
Article 45. The term of office of members of the House of Representatives shall be four years. However, the term shall be terminated before the full term is up in case the House of Representatives is dissolved.
Article 102. The term of office for half the members of the House of Councillors serving in the first term under this Constitution shall be three years. Members falling under this category shall be determined in accordance with law.
Article 43. Both Houses shall consist of elected members, representative of all the people.
Article 103. The Ministers of State, members of the House of Representatives and judges in office on the effective date of this Constitution, and all other public officials who occupy positions corresponding to such positions as are recognized by this Constitution shall not forfeit their positions automatically on account of the enforcement of this Constitution unless otherwise specified by law. When, however, successors are elected or appointed under the provisions of this Constitution, they shall forfeit their positions as a matter of course.
Article 97. The fundamental human rights by this Constitution guaranteed to the people of Japan are fruits of the age-old struggle of man to be free; they have survived the many exacting tests for durability and are conferred upon this and future generations in trust, to be held for all time inviolate.
Article 46. The term of office of members of the House of Councillors shall be six years, and election for half the members shall take place every three years.
Article 44. The qualifications of members of both Houses and their electors shall be fixed by law. However, there shall be no discrimination because of race, creed, sex, social status, family origin, education, property or income.
Article 55. Each House shall judge disputes related to qualifications of its members. However, in order to deny a seat to any member, it is necessary to pass a resolution by a majority of two-thirds or more of the members present.
Article 9. Aspiring sincerely to an international peace based on justice and order, the Japanese people forever renounce war as a sovereign right of the nation and the threat or use of force as means of settling international disputes.
Article 98. This Constitution shall be the supreme law of the nation and no law, ordinance, imperial rescript or other act of government, or part thereof, contrary to the provisions hereof, shall have legal force or validity.
As a result of bm25Search, these three appear in the Top 7.
In[]:=
(candidatesBM=bm25Search[query,chunksEN,10])//TableForm
Out[]//TableForm=
Article 59. A bill becomes a law on passage by both Houses, except as otherwise provided by the Constitution.
10.3007
Article 102. The term of office for half the members of the House of Councillors serving in the first term under this Constitution shall be three years. Members falling under this category shall be determined in accordance with law.
10.0079
Article 50. Except in cases provided by law, members of both Houses shall be exempt from apprehension while the Diet is in session, and any members apprehended before the opening of the session shall be freed during the term of the session upon demand of the House.
8.90009
Article 46. The term of office of members of the House of Councillors shall be six years, and election for half the members shall take place every three years.
8.33578
Article 43. Both Houses shall consist of elected members, representative of all the people.
8.31933
Article 51. Members of both Houses shall not be held liable outside the House for speeches, debates or votes cast inside the House.
8.11456
Article 45. The term of office of members of the House of Representatives shall be four years. However, the term shall be terminated before the full term is up in case the House of Representatives is dissolved.
8.08487
Article 47. Electoral districts, method of voting and other matters pertaining to the method of election of members of both Houses shall be fixed by law.
7.96797
Article 49. Members of both Houses shall receive appropriate annual payment from the national treasury in accordance with law.
7.54676
Article 64. The Diet shall set up an impeachment court from among the members of both Houses for the purpose of trying those judges against whom removal proceedings have been instituted.
7.34948
As a result of hybridSearch, these three appear in the Top 4.
In[]:=
(candidates=hybridSearch[query,index,chunksEN])//TableForm
Out[]//TableForm=
Article 102. The term of office for half the members of the House of Councillors serving in the first term under this Constitution shall be three years. Members falling under this category shall be determined in accordance with law.
Article 45. The term of office of members of the House of Representatives shall be four years. However, the term shall be terminated before the full term is up in case the House of Representatives is dissolved.
Article 43. Both Houses shall consist of elected members, representative of all the people.
Article 46. The term of office of members of the House of Councillors shall be six years, and election for half the members shall take place every three years.
Article 59. A bill becomes a law on passage by both Houses, except as otherwise provided by the Constitution.
As a result of rerank, these three appear in the Top 5.
In[]:=
(reranks=rerank[query,candidates,5])//TableForm
Out[]//TableForm=
Article 45. The term of office of members of the House of Representatives shall be four years. However, the term shall be terminated before the full term is up in case the House of Representatives is dissolved.
Article 102. The term of office for half the members of the House of Councillors serving in the first term under this Constitution shall be three years. Members falling under this category shall be determined in accordance with law.
Article 43. Both Houses shall consist of elected members, representative of all the people.
Article 59. A bill becomes a law on passage by both Houses, except as otherwise provided by the Constitution.
Article 46. The term of office of members of the House of Councillors shall be six years, and election for half the members shall take place every three years.
Unfortunately, hybridSearch and rerank were unable to extract the required three chunks within the top 3.
My previous post have addressed this issue by using embedding models Cohere:embed-multilingual-v3.0 or Hugging Face:multilingual-e5-large-embedding.
Furthermore, I’ve used a new approach(which I will call an oracle filter): “ask the LLM whether each chunk is useful for the query.” In particular, the bm25Search is unlikely to function well for Japanese due to the word segmentation, making this approach highly likely to be effective.
Try it for the result of rerank.
template=StringTemplate["If the information contains even a some part of what is needed to create an answer for the query, output YES. Otherwise, output NO.\n\n\=== QUERY ===\n`query`\n\n=== INFORMATION ===\n`info`"];
LLMSynthesize[template[<|"query"->query,"info"->#|>]]&/@reranks
Out[]=
{YES,YES,NO,NO,YES}
It was able to extract only the three necessary chunks.
The results above are summarized in the table below.
Out[]=
SemanticSearch
bm25Search
hybridSearch
rerank
oracle filter
Article 45
1
7
2
1
1
Article 46
6
4
4
5
3
Article 102
2
2
1
2
2