General Initialization

In[]:=
StableDiffusionSynthesize=ResourceFunction["https://www.wolframcloud.com/obj/nikm/DeployedResources/Function/StableDiffusionSynthesize/"];
In[]:=
$NetModelPath="/home/sw/Dropbox/GeneralBox/Research/LLMScience/FromNik";
In[]:=
sdOptions=Normal@<|​​TargetDevice->"GPU",​​AssociationMap[FileNameJoin[{$NetModelPath,"Parameters/Stable-Diffusion-V1",#<>".wlnet"}]&,{"Unet","Encoder","Decoder"}​​]​​|>;

Knockout targets

In[]:=
unet=Import[Lookup[sdOptions,"Unet"]];
In[]:=
unet
Out[]=
NetGraph

InputPorts
​
Input
:
array
(size: 4×64×64)
TimeStep
:
vector
(size: 1)
of integers
TextEmbedding
:
matrix
(size:
n
×768)
OutputPort
Output
:
array
(size: 4×64×64)
up:
NetGraph
Input Ports
​
Input1
:
matrix
(size:
n
×768)
Input2
:
array
(size: 320×64×64)
Input3
:
vector
(size: 1280)
Input4
:
array
(size: 1280×8×8)
Input5
:
array
(size: 320×32×32)
Input6
:
array
(size: 320×64×64)
Input7
:
array
(size: 320×64×64)
Input8
:
array
(size: 640×16×16)
Input9
:
array
(size: 640×32×32)
Input10
:
array
(size: 640×32×32)
Input11
:
array
(size: 1280×8×8)
Input12
:
array
(size: 1280×16×16)
Input13
:
array
(size: 1280×16×16)
Input14
:
array
(size: 1280×8×8)
Input15
:
array
(size: 1280×8×8)
Output Port
Output
:
array
(size: 320×64×64)
This object cannot be used as input.

In[]:=
Information[unet,"SummaryGraphic"]
Out[]=
In[]:=
Information[unet,"FullSummaryGraphic"]
Out[]=
$Aborted
In[]:=
Information[unet,"LayersGraph"]
Out[]=
In[]:=
Graph[%,AspectRatio->1/3]
Out[]=

Encoder, decode: downsampling, upsampling image

Knockout

In[]:=
StableDiffusionSynthesize["A cat in a party hat"->BlockRandom[SeedRandom[33];RandomReal[NormalDistribution[],{4,64,64}]],sdOptions]
Out[]=
Nik:StableDiffusionKnockout
In[]:=
unet=Import[Lookup[sdOptions,"Unet"]];
In[]:=
layers=Information[unet,"Layers"];
In[]:=
Length[%]
Out[]=
992
In[]:=
Take[layers,2]
Only some of these layers contain trainable weights...
[[[ In this picture we want to indicate with arrows where data comes in and goes out ... ]]]

Zeroing

Randomized:

Showing knockouts in their place

This fluctuated because the time embedding got changed

Collapsing into blocks

Making a random image

Colors of individual pixels.... [what is the typical ImageHistogram of the world??]

LearnDistribution | “Diffusion”

Diffusion here means adding pixel noise [or latent space noise]
Training:
[[ Trains at each step of the diffusion ]]