Wolfram Cloud Document

Stock data series pairwise correlation

So here is my fastest pairwise correlation version so far.

Instead of computing all (505*504/2 = 127260) possible intersections between pairs of the stock members series, first I search for the members which share exactly the same dates (actually it was faster here to search for the missing dates). In the stockdata series example i use for the test (below), there is indeed only about 20 such families (two main with about 100 and 300 members each), so there is only the need to compute about 200 ( 20*19/2) intersections instead of 127260. This however costs some time, but it is still 2-4x faster than the "brute force" approach (127260 intersections) that I have also tested (not shown here).

I also made use of basic compilation (old compiler) in order to greatly speed up the correlation computation itself. It was also fast to generate the pairs of values to correlate (however in version 12.3 the "Outer" function seems as fast for the same task). I didn't succeed in improving the timings using parallelisation (with the "Listable" attribute) and 2 cores, there seems to be some overhead. For technical reasons I could not compile to C, but that would interesting to test.

Functions

corrxy=Compile[{{xy,_Real,2}},Module[{n,x,y,sx,sy},x=xy[[1]];y=xy[[2]];n=Length@x;sx=Total[x];sy=Total[y];(n*Total@(x*y)-sx*sy)/Sqrt[(n*Total@(x^2)-sx^2)*(n*Total@(y^2)-sy^2)]],RuntimeAttributes{Listable}]; corrinter=With[{corrxy=corrxy}, Compile[{{t1,_Real,2},{t2,_Real,2}},Module[{n1=Length@t1,n2=Length@t2} ,Table[corrxy@{t1[[i]],t2[[j]]},{j,1,n2},{i,1,n1}]] ]]; corrintra=With[{corrxy=corrxy},Compile[{{t1,_Real,2}},Module[{n1=Length@t1,listcor=Internal`Bag[]} ,Do[Internal`StuffBag[listcor,corrxy[{t1[[i]],t1[[j]]}]],{i,1,n1-1},{j,i+1,n1}]; Internal`BagPart[listcor,All]] ]]; (*-----------------Mainfunction*) (*Ididnotlocalizedthevariables,foraneasiercodetesting*)pwc[stockdata_]:=Module[{},listAlldates=stockdata[[All,All,1]]//Join@@#&//DeleteDuplicates//Sort;missingDates=Complement[listAlldates,#[[All,1]]]&/@stockdata;sameDatesFamilies=MapIndexed[{Sequence@@#2,Hash@#}&,missingDates]//GroupBy[#,Last,Map[First]]&//Values;(*-----CheckingforHashcollisionSameQ[missingDates[[#]]]&/@sameDatesFamilies//And@@#&*)familiesDates=missingDates[[sameDatesFamilies[[All,1]]]];subsetsFamilies=Subsets[Range@Length@familiesDates,{2}];(*------Regularstockdatamatrix*)stockPA={Thread[{#,-9999.}]&/@missingDates,stockdata}//Transpose//Map[Sort[Join@@#][[All,2]]&,#]&//Developer`ToPackedArray;(*------*)rangeDates=Range@Length@listAlldates;rulesDatesToIndices=listAlldates//Thread[Rule[#,Range@Length@#]]&//Dispatch;(*------Interfamiliescorrelations*)rescorinter=Table[commonind=Union@@familiesDates[[subsetsFamilies[[isub]]]]//Complement[rangeDates,Replace[#,rulesDatesToIndices,{1}]]&;{t1,t2}=Map[stockPA[[#,commonind]]&,sameDatesFamilies[[subsetsFamilies[[isub]]]]];corrinter[t2,t1],{isub,Length@subsetsFamilies}];(*------Intrafamiliescorrelations*)rescorintra=Table[common=familiesDates[[ifam]]//Replace[#,rulesDatesToIndices,{1}]&//Complement[rangeDates,#]&;stockPA[[sameDatesFamilies[[ifam]],common]]//corrintra,{ifam,Length@familiesDates}];(*------Finalcorrelationslist*)rescor=Flatten@Join[rescorinter,rescorintra]];

Tests

Wolfram free basic cloud

In[]:=

$Version

Out[]=

12.3.0 for Linux x86 (64-bit) (May 10, 2021)

In[]:=

stockdata=CloudGet[CloudObject[

https://www.wolframcloud.com/obj/98b7b9af-1c7d-49d1-b8a1-76c267fcfb1d

]];

In[]:=

Length@stockdataLength/@stockdata

In[]:=

(res=pwc@stockdata);//RepeatedTiming

Out[]=

{1.0924,Null}

In[]:=

Length[res]Through[{Mean,Median,Min,Max}[res]]

Out[]=

127260

Out[]=

{0.428747,0.43533,-0.167036,0.996379}