Indexing “This Week’s Finds in Mathematical Physics”

Create list of “This Week’s Finds in Mathematical Physics”, TWF, articles and import them from the web into a list and then a string.
In[]:=
twfArticleCount=300;​​Off[$CharacterEncoding::utf8];​​twfList=Import/@Table["https://math.ucr.edu/home/baez/week"<>ToString[k]<>".html",{k,twfArticleCount}];​​On[$CharacterEncoding::utf8];​​twfString=StringJoin[twfList]//TextWords//Sort//DeleteDuplicates;
In[]:=
Short[twfList]
Out[]//Short=
{10336,13675,14599,15668,16298,21603,11921,8049,17513,11642,6821,9137,12773,274,35277,46156,28294,23865,29290,29071,44654,21649,21595,23043,25485,35874,34188}
TWF contains quoted text.
In[]:=
twfQuote=StringTake[StringCases[StringJoin[twfList],RegularExpression["\\\".+?\""]],{2,-2}]//Sort//DeleteDuplicates;​​twfQuote//Length​​twfQuote//Short
Out[]=
6725
Out[]//Short=
{!,@,&,+,=,?, ,???,....,") = ,0,0-cells.,6701,ω-magma,ω times 2,ω times ω,†-Frobenius algebra,∞,(∞,1)-category,(∞,k)-categories,∞ minus ∞,(∞,n)-category,£,⊗,→}
In[]:=
twfValid=CharacterRange["0","9"]|CharacterRange["A","Z"]|CharacterRange["a","z"]|CharacterRange["α","ω"]|CharacterRange["Α","Ω"]|{"-","_","'"};
In[]:=
twfInvalid="#!\"%()+,./:;=?@[\\]^`{|}©³½ħ∗∩∪⋅$°°C£�℘↓↑√∇∂⊗×∏⊕∫∑±≥≤≠≅≈←→∈⊆⊇∧"//Characters;
In[]:=
twfDict=StringSplit[StringReplace[ToString[twfString],twfInvalid->""]]//Sort//DeleteDuplicates;​​twfDict//Short
Out[]//Short=
{0,00,000,0000,00000-0,000000,0000000,00000000000000,00000388,0001,0001083,35917,ω-groupoidal,ω-groupoids,ω-GROUPOIDS,ω-magma,ω-magmas,Ω-magmas,ω-sized,ωt,ω-th,ων,∞}
In[]:=
twfDeleteNumbers=DeleteCases[Table[If[StringMatchQ[twfDict[[k]],NumberString,IgnoreCase->True],Null,twfDict[[k]]],{k,Length[twfDict]}],Null];​​twfDeleteNumbers//Short​​twfDict=DeleteCases[Table[If[StringMatchQ[twfDeleteNumbers[[k]],RegularExpression[".*\\d+[-].*"],IgnoreCase->True],Null,twfDeleteNumbers[[k]]],{k,Length[twfDeleteNumbers]}],Null];​​twfDict//Short
Out[]//Short=
{00000-0,00-1,0-1,01-0,012n-1,01-junctions,01n-1,0-2,02-1,03002B,0-387-90336-4,33719,ω-groupoids,ω-GROUPOIDS,ω-magma,ω-magmas,Ω-magmas,ω-sized,ωt,ω-th,ων,∞}
Out[]//Short=
{012n-1,01n-1,03002B,0611930v1,'07,0A36,0AF8,0B,0B-A,0d,0i,0's,32080,ω-groupoidal,ω-groupoids,ω-GROUPOIDS,ω-magma,ω-magmas,Ω-magmas,ω-sized,ωt,ω-th,ων,∞}
In[]:=
twfGreek=Take[twfDict,-109]
Out[]=
{α-sulfur,αth,β,βE,βH,βk,β-normal,β-sulfur,γ,Γ,γ-ambiguity,γ-ray,Γ's,γth,δ,Δ,δA,Δ-action,δe,δI,δJ,Δt,ΔX,Δ-Y,ε,ζ,Ζ,η,η-expanded,η-long,θ,Θ,κ,λ,Λ,λ-calculus,λ_i,λ-notation,ΛV,μ,ν,νdω,ν_L,νω,π,Π,π_0,π_1,π_2,π_3,π_4,π_7,πD,π_i,π_k,π_n,πon,ΠSX,πα,ρ,σ,Σ,σ-compact,σ-model,σ-models,σ-polytopes,Σ-structures,τ,τ's,φ,Φ,Φ-1,φ_a,Φl,χ,ψ,Ψ,ψ_a,ψ-bar,ψ_M,ψ_P,ψ_S,Ψ's,ω,Ω,ω2,ω3,ω4,ω5,ω-ATEGORIES,ω-ATEGORY,ΩB,ω-categories,ω-category,ω-dimensional,ω-FUNTORS,ω-groupoid,ω-GROUPOID,ω-groupoidal,ω-groupoids,ω-GROUPOIDS,ω-magma,ω-magmas,Ω-magmas,ω-sized,ωt,ω-th,ων,∞}
In[]:=
twfChars=Characters[twfString]//Flatten//Sort//DeleteDuplicates
Out[]=
!,@,#,%,&,(,),_,-,+,=,~,`,<,>,.,,,;,?,',/,:, ,³,´,½,ċ,ħ,∗,∩,∪,⋅,0,1,2,3,4,5,6,7,8,9,a,á,à,ä,ã,A,Á,Å,b,B,c,C,d,ð,D,e,é,è,ê,E,É,f,F,g,G,h,H,i,í,I,j,J,k,K,l,L,m,M,n,N,o,ó,ö,ø,O,Ö,Ø,p,P,q,Q,r,R,s,S,Š,t,T,u,ü,U,Ü,v,V,w,W,x,X,y,Y,z,Z,α,β,γ,Γ,δ,Δ,ε,ζ,Ζ,η,θ,Θ,κ,λ,Λ,μ,ν,π,Π,ρ,σ,Σ,τ,φ,Φ,χ,ψ,Ψ,ω,Ω,$,©,–,°,,∞,£,�,℘,’,↓,↑,√,∇,∂,⊗,×,∏,⊕,∫,∑,±,≥,≤,≠,≅,≈,←,→,∈,⊆,⊇,∧
In[]:=
twfChars//Length
Out[]=
177
In[]:=
Table[CharacterName[twfChars[[k]]],{k,Length[twfChars]}]//Short
Out[]//Short=
{RawExclamation,RawAt,RawNumberSign,RawPercent,RawAmpersand,RawLeftParenthesis,RawRightParenthesis,163,TildeTilde,LeftArrow,RightArrow,Element,SubsetEqual,SupersetEqual,And}
In[]:=
twfData=Import["https://www.science.co.il/internet/html/Greek-characters.php","Data"];​​twfGreekChars=Table[{twfData[[2,1,3,k,2]],twfData[[2,1,3,k,3]],twfData[[2,1,3,k,4]],CharacterName[twfData[[2,1,3,k,3]]]},{k,24}]~Join~Table[{twfData[[2,2,3,k,2]],twfData[[2,2,3,k,3]],twfData[[2,2,3,k,4]],CharacterName[twfData[[2,2,3,k,3]]]},{k,25}];​​{{"Name","Symbol","Code","Mathematica Name"}}~Join~twfGreekChars//TableForm​​
Out[]//TableForm=
Name
Symbol
Code
Mathematica Name
Alpha
Α
913
CapitalAlpha
Beta
Β
914
CapitalBeta
Gamma
Γ
915
CapitalGamma
Delta
Δ
916
CapitalDelta
Epsilon
Ε
917
CapitalEpsilon
Zeta
Ζ
918
CapitalZeta
Eta
Η
919
CapitalEta
Theta
Θ
920
CapitalTheta
Iota
Ι
921
CapitalIota
Kappa
Κ
922
CapitalKappa
Lambda
Λ
923
CapitalLambda
Mu
Μ
924
CapitalMu
Nu
Ν
925
CapitalNu
Xi
Ξ
926
CapitalXi
Omicron
Ο
927
CapitalOmicron
Pi
Π
928
CapitalPi
Rho
Ρ
929
CapitalRho
Sigma
Σ
931
CapitalSigma
Tau
Τ
932
CapitalTau
Upsilon
Υ
933
CapitalUpsilon
Phi
Φ
934
CapitalPhi
Chi
Χ
935
CapitalChi
Psi
Ψ
936
CapitalPsi
Omega
Ω
937
CapitalOmega
alpha
α
945
Alpha
beta
β
946
Beta
gamma
γ
947
Gamma
delta
δ
948
Delta
epsilon
ϵ
949
Epsilon
zeta
ζ
950
Zeta
eta
η
951
Eta
theta
θ
952
Theta
iota
ι
953
Iota
kappa
κ
954
Kappa
lambda
λ
955
Lambda
mu
μ
956
Mu
nu
ν
957
Nu
xi
ξ
958
Xi
omicron
ο
959
Omicron
pi
π
960
Pi
rho
ρ
961
Rho
sigmaf
ς
962
FinalSigma
sigma
σ
963
Sigma
tau
τ
964
Tau
upsilon
υ
965
Upsilon
phi
ϕ
966
Phi
chi
χ
967
Chi
psi
ψ
968
Psi
omega
ω
969
Omega
In[]:=
toHTML[s_String]:=If[Between[ToCharacterCode[s][[1]],{945,969}],ToLowerCase["&"<>StringReplace[CharacterName[s]<>";","Capital"->""]],"&"<>StringReplace[CharacterName[s]<>";","Capital"->""]]
In[]:=
Select[twfDict,DictionaryWordQ]//Length
Out[]=
22693
In[]:=
Complement[twfDict,Select[twfDict,DictionaryWordQ]]//Length
Out[]=
9410
In[]:=
Complement[twfDict,Select[twfDict,DictionaryWordQ]]//Short
Out[]//Short=
{012n-1,01n-1,03002B,0611930v1,'07,0A36,0AF8,0B,0B-A,0d,0i,0's,9387,ω-groupoidal,ω-groupoids,ω-GROUPOIDS,ω-magma,ω-magmas,Ω-magmas,ω-sized,ωt,ω-th,ων,∞}
In[]:=
twfWikiMath=Join[WikipediaData[Table["List_of_mathematics_articles_("<>FromCharacterCode[k]<>")",{k,65,90}]],1];
This defines what makes it into the index.

TWF Version 2.0

Perform syntactic analysis of dictionary.