You are using a browser not supported by the Wolfram Cloud
Supported browsers include recent versions of Chrome, Edge, Firefox and Safari.
I understand and wish to continue anyway »
WOLFRAM NOTEBOOK
General Initialization
General Initialization
In[]:=
StableDiffusionSynthesize=ResourceFunction["https://www.wolframcloud.com/obj/nikm/DeployedResources/Function/StableDiffusionSynthesize/"];
In[]:=
$NetModelPath="/home/sw/Dropbox/GeneralBox/Research/LLMScience/FromNik";
In[]:=
sdOptions=Normal@<|TargetDevice->"GPU",AssociationMap[FileNameJoin[{$NetModelPath,"Parameters/Stable-Diffusion-V1",#<>".wlnet"}]&,{"Unet","Encoder","Decoder"}]|>;
Knockout targets
Knockout targets
In[]:=
unet=Import[Lookup[sdOptions,"Unet"]];
In[]:=
unet
Out[]=
In[]:=
Information[unet,"SummaryGraphic"]
Out[]=
In[]:=
Information[unet,"FullSummaryGraphic"]
Out[]=
$Aborted
In[]:=
Information[unet,"LayersGraph"]
Out[]=
In[]:=
Graph[%,AspectRatio->1/3]
Out[]=
Encoder, decode: downsampling, upsampling image
Encoder, decode: downsampling, upsampling image
Knockout
Knockout
In[]:=
StableDiffusionSynthesize["A cat in a party hat"->BlockRandom[SeedRandom[33];RandomReal[NormalDistribution[],{4,64,64}]],sdOptions]
Out[]=
Nik:StableDiffusionKnockout
In[]:=
unet=Import[Lookup[sdOptions,"Unet"]];
In[]:=
layers=Information[unet,"Layers"];
In[]:=
Length[%]
Out[]=
992
In[]:=
Take[layers,2]
Out[]=
In[]:=
ReverseSort[Counts[Head/@Values[layers]]]
Out[]=
LinearLayer184,TransposeLayer154,ThreadingLayer125,NormalizationLayer109,ReshapeLayer99,ConvolutionLayer98,FlattenLayer77,ElementwiseLayer65,PartLayer32,DotLayer32,CatenateLayer13,ResizeLayer3,NetArrayLayer1
Only some of these layers contain trainable weights...
In[]:=
tlayers=Select[layers,MatchQ[_ConvolutionLayer|_LinearLayer|_NormalizationLayer]];
In[]:=
Length[tlayers]
Out[]=
391
In[]:=
Clear[unetRandom]
In[]:=
unetRandom[laynames_]:=NetReplacePart[unet,#->NetInitialize[tlayers[#],All,RandomSeeding->6573]&/@laynames]
In[]:=
Take[tlayers,5]
Out[]=
In[]:=
unetRandom[{#}]&/@Take[Keys[tlayers],3]
Out[]=
In[]:=
ByteCount/@%
Out[]=
{3443249376,3443249376,3443249376}
In[]:=
StableDiffusionSynthesize["A cat in a party hat"->BlockRandom[SeedRandom[33];RandomReal[NormalDistribution[],{4,64,64}]],"Unet"->unetRandom[{#}],sdOptions]&/@Take[Keys[tlayers],2]
Out[]=
[[[ In this picture we want to indicate with arrows where data comes in and goes out ... ]]]
Zeroing
Zeroing
Randomized:
Showing knockouts in their place
Showing knockouts in their place
This fluctuated because the time embedding got changed
Collapsing into blocks
Collapsing into blocks
Making a random image
Making a random image
Colors of individual pixels.... [what is the typical ImageHistogram of the world??]
LearnDistribution | “Diffusion”
LearnDistribution | “Diffusion”
Diffusion here means adding pixel noise [or latent space noise]
Training:
[[ Trains at each step of the diffusion ]]