Interpolation functions of large arrays in combination with parallel evaluation cannot be evaluated using normal code.
Based on my educated guess has to do with issues of memory management of unpacked arrays and parallel kernels.
To benefit from parallel evaluation using large datasets one has to overcome the burden of parallel distribution of large arrays which is very slow for arrays that are not in packed array form. Additionally using normal distribution of functions wont work since somehow the int-function is between 13x and 50000x slower on parallel kernels (depending on size of data). This slowdown in on top of the slow parallel distribution of large arrays.
Using using parallel evaluate of the direct definition of int function using packed arrays one is able to perform efficient parallel evaluation with the expected speed up of around 3x using 4 kernels.
This has been tested on mac (12.5, i5) and windows (10, i7) in Mathematica 13.1
Based on my educated guess has to do with issues of memory management of unpacked arrays and parallel kernels.
To benefit from parallel evaluation using large datasets one has to overcome the burden of parallel distribution of large arrays which is very slow for arrays that are not in packed array form. Additionally using normal distribution of functions wont work since somehow the int-function is between 13x and 50000x slower on parallel kernels (depending on size of data). This slowdown in on top of the slow parallel distribution of large arrays.
Using using parallel evaluate of the direct definition of int function using packed arrays one is able to perform efficient parallel evaluation with the expected speed up of around 3x using 4 kernels.
This has been tested on mac (12.5, i5) and windows (10, i7) in Mathematica 13.1
Needs["Developer`"]ParallelEvaluate[Needs["Developer`"]];MBCount=UnitConvert[Quantity[N@ByteCount[#],"Byte"],"MB"]&;ClearAll[MakeIntFunc,MakeIntFuncList];MakeIntFunc[dat_,vox_,int_?IntegerQ]:=Block[{range,d=dat[[1,1,1]]},range=Thread[{vox,voxDimensions[dat][[1;;3]]}]-(0.5vox);d=0.If[ListQ[d],Flatten@d,d];With[{ex=d},InterpolatingFunction[range,(*nocluewhatthisvectormeansjustcopyitfromoutputofListInterpolation*){5,If[ArrayDepth[dat]===3,6,2],0,Dimensions[dat][[;;3]],{int,int,int}+1,0,0,0,0,ex&,{},{},False},Range[range[[#,1]],range[[#,2]],vox[[#]]]&/@{1,2,3},dat,{Automatic,Automatic,Automatic}]]]MakeIntFuncList[dat_,vox_,int_?IntegerQ]:=Block[{range,d=dat[[1,1,1]]},range=Thread[{vox,voxDimensions[dat][[1;;3]]}]-(0.5vox);d=0.If[ListQ[d],Flatten@d,d];With[{ex=d},ListInterpolation[dat,range,InterpolationOrder->int,"ExtrapolationHandler"->{ex&,"WarningMessage"->False}]]]DistributeDefinitions[MakeIntFunc,MakeIntFuncList];vox={6,3,3};rr6=ToPackedArray@RandomReal[{0,1},{50,160,160,1,6}];n=10;(*normalevaluation*)ClearAll[f];ParallelEvaluate[ClearAll[f]];m1=AbsoluteTiming[t1=AbsoluteTiming[f=MakeIntFunc[rr6,vox,1];][[1]];t2=AbsoluteTiming[res1=Table[Do[f[i,j,k],{n}];f[i,j,k],{i,2,10,.1},{j,5,10,.1},{k,5,10,.1}];][[1]];{t1,t2}];(*parallelevaluationusingdistributebuildinintfunc*)ClearAll[f];ParallelEvaluate[ClearAll[f]];m3=AbsoluteTiming[t1=AbsoluteTiming[ParallelEvaluate[f=MakeIntFuncList[rr6,vox,1];];][[1]];t2=AbsoluteTiming[res3=ParallelTable[Do[f[i,j,k],{n}];f[i,j,k],{i,2,10,.1},{j,5,10,.1},{k,5,10,.1}];][[1]];{t1,t2}];(*!!!!!!!!!!winner!!!!!!!!!*)(*parallelevaluationusingdistribute,customintfunc*)ClearAll[f];ParallelEvaluate[ClearAll[f]];m4=AbsoluteTiming[t1=AbsoluteTiming[ParallelEvaluate[f=MakeIntFunc[rr6,vox,1];];][[1]];t2=AbsoluteTiming[res4=ParallelTable[Do[f[i,j,k],{n}];f[i,j,k],{i,2,10,.1},{j,5,10,.1},{k,5,10,.1}];][[1]];{t1,t2}];Column@{m1,m2,m4}res1==res3==res4
(*windows*)
Out[]=
{33.0514,{0.463127,32.5883}} |
{21.1686,{4.02558,17.143}} |
{16.3276,{0.702725,15.6248}} |
Out[]=
True
(*mac*)
Out[]=
{30.21,{0.357606,29.8524}} |
{12.0607,{2.00843,10.0523}} |
{9.8253,{0.443977,9.38131}} |
Out[]=
True
(*parallelevaluationusingdistribute,Thisdoesnotworkforlargearraysdidnotfinishafterovernightevaluation*)ClearAll[f];ParallelEvaluate[ClearAll[f]];m2=AbsoluteTiming[t1=AbsoluteTiming[f=MakeIntFunc[rr6,vox,1];][[1]];t2=AbsoluteTiming[res2=ParallelTable[Do[f[i,j,k],{n}];f[i,j,k],{i,2,10,.1},{j,5,10,.1},{k,5,10,.1}];][[1]];{t1,t2}];