Hi Jonathan,
I also wanted to play around to see how fast I could get it .
Only got a 60 % reduction in time so nowhere near as impressive as Chris did .
My approach is basically implementing the Paired correlation, the time index is in seconds but probably could also be changed to date .
If the time lists do not match I find the matching indices using a very basic intersection method for ordered lists .
And also implemented a very basic Correlation function that beats the default Correlation function for small lists .
Hope it helps .
I also wanted to play around to see how fast I could get it .
Only got a 60 % reduction in time so nowhere near as impressive as Chris did .
My approach is basically implementing the Paired correlation, the time index is in seconds but probably could also be changed to date .
If the time lists do not match I find the matching indices using a very basic intersection method for ordered lists .
And also implemented a very basic Correlation function that beats the default Correlation function for small lists .
Hope it helps .
In[]:=
PairwiseCorrelation[stockdata_]:=Module[{assocStocks,pairs,correl},assocStocks=Apply[Rule,stockdata,{2}]//Map[Association];pairs=Subsets[Range@Length@assocStocks,{2}];correl=Map[Correlation@@Values@KeyIntersection[assocStocks[[#]]]&,pairs];{correl,pairs}]
(*onlyselectindicesiftimeisnotmatching*)PairedCorrelation[data1_,data2_]:=If[data1[[All,1]]===data2[[All,1]],CorrelationC[data1[[All,2]],data2[[All,2]]],CorrelationC[data1[[First@#,2]],data2[[Last@#,2]]]&@SelectIndex[data1[[All,1]],data2[[All,1]]]](*assumingsortedlists*)SelectIndex=Compile[{{t1,_Real,1},{t2,_Real,1}},Block[{l1,l2,i1,i2,b,out},(*lengthsofdata*)l1=Length[t1];l2=Length[t2];(*testifsame*)If[t1===t2,(*ifsameoutputallindices*){Range[l1],Range[l2]},(*preparewhile*)out=ConstantArray[0,{Min[l1,l2],2}];i1=i2=1;b=0;(*loopoverindicesuntilsamevalueisfound-incrementthelowestvalue-ifsimilarstoreindices*)While[i1<=l1&&i2<=l2,Which[t1[[i1]]<t2[[i2]],i1++,t2[[i2]]<t1[[i1]],i2++,True,b++;out[[b]]={i1,i2};i1++;i2++;]];(*outputmatchingindixes*)Transpose[out[[;;b]]]]]];(*fastcompilescorrelationfunction*)CorrelationC=Compile[{{c1,_Real,1},{c2,_Real,1}},Block[{mn1=c1-Mean[c1],mn2=c2-Mean[c2]},(mn1.mn2/(Sqrt[Total[mn1^2]]*Sqrt[Total[mn2^2]]))]];
(*downloadallthedata*)tickers=Take[FinancialData["^GSPC","Members"]];tsStocks=FinancialData[tickers,"Return",DatePlus[Today,{-2753,"BusinessDay"}]];(*timeinsecondsandvalueinnumberwithoutunits*)data=N@QuantityMagnitude[#["Path"]]&/@tsStocks;
In[]:=
(*testcorrelationfunction,fastforsmallsets*){d1,d2}=Transpose@RandomVariate[BinormalDistribution[1/3],10^3];Correlation[d1,d2]//RepeatedTimingCorrelationC[d1,d2]//RepeatedTiming(*testcorrelationfunction,equalforintermediatesets*){d1,d2}=Transpose@RandomVariate[BinormalDistribution[1/3],10^5];Correlation[d1,d2]//RepeatedTimingCorrelationC[d1,d2]//RepeatedTiming(*testcorrelationfunction,slowerforlargesets*){d1,d2}=Transpose@RandomVariate[BinormalDistribution[1/3],10^6];Correlation[d1,d2]//RepeatedTimingCorrelationC[d1,d2]//RepeatedTiming
Out[]=
{0.0000454254,0.376036}
Out[]=
{2.70699×,0.376036}
-6
10
Out[]=
{0.000184674,0.335178}
Out[]=
{0.000181322,0.335178}
Out[]=
{0.0074216,0.334247}
Out[]=
{0.0119511,0.334247}
In[]:=
(*nonmatchingdates,findingpairstakestime*)PairedCorrelation[data[[1]],data[[2]]]//RepeatedTiming(*matchingdates,onlycalculatecorrelation*)PairedCorrelation[data[[1]],data[[31]]]//RepeatedTiming
Out[]=
{0.00212525,0.49233}
Out[]=
{0.000372427,0.55602}
cor1=PairwiseCorrelation[data][[1]];//AbsoluteTimingcor2=PairedCorrelation[data[[First@#]],data[[Last@#]]]&/@Subsets[Range@Length@data,{2}];//AbsoluteTiming
Out[]=
{506.574,Null}
Out[]=
{200.461,Null}
In[]:=
Through[{Mean,Median,Min,Max}[cor1]]Through[{Mean,Median,Min,Max}[cor2]]
Out[]=
{0.371174,0.370584,-0.266977,0.992836}
Out[]=
{0.371174,0.370584,-0.266977,0.992836}