Neural Radiance Fields (Mildenhall et al., ECCV 2020) revolutionized computer vision and computer graphics by representing complex 3D scenes not as explicit polygonal meshes or voxel grids, but as continuous coordinate-based neural networks .
A common misconception is that training a NeRF requires complex custom CUDA kernels, external C++ build chains, or PyTorch. In this notebook, we demonstrate how to build, train, and render a complete NeRF entirely within native Wolfram Language using NetGraph, MXNet primitives, and the WLNeRFLite paclet.
f:(x,d)->(c,σ)
A common misconception is that training a NeRF requires complex custom CUDA kernels, external C++ build chains, or PyTorch. In this notebook, we demonstrate how to build, train, and render a complete NeRF entirely within native Wolfram Language using NetGraph, MXNet primitives, and the WLNeRFLite paclet.
Key technical highlights:
1
.Direct camera ray generation from standard NeRF / OpenGL 4x4 camera-to-world (c2w) pose matrices (transforms.json).
2
.GPU-embedded volumetric quadrature: computing continuous optical transmittance and alpha compositing entirely in-graph using lower-triangular matrix multiplication (trilStrict), eliminating CPU-GPU transfer bottlenecks.
3
.Automated dataset ingestion of the classic synthetic 'Lego' bulldozer dataset via native Wolfram Language (DownloadNeRFDataset & LoadNeRFDataset).
4
.Interactive 3D camera rig visualization with camera frustums and optical axes.
5
.High-throughput ray pooling with stratified foreground sampling (>25,000 rays/sec).
6
.Quantitative validation against held-out views (PSNR / MSE) and 360-degree novel-view turntable synthesis.
7
.Continuous 3D density field extraction via Image3D and ListContourPlot3D.
1. Environment Setup & Package Loading
1. Environment Setup & Package Loading
Make sure the WLNeRFLite paclet is in the same directory of this notebook, then evaluate the following to load it into the Wolfram kernel session and detect the compute target device (CUDA GPU or multi-core CPU).
In[]:=
SetDirectory[NotebookDirectory[]];
In[]:=
PacletDirectoryLoad[Directory[]];Needs["WLNeRFLite`"];
(*$TargetDevice="GPU";*)
2. Automated Dataset Ingestion: The Classic Synthetic Lego Bulldozer
2. Automated Dataset Ingestion: The Classic Synthetic Lego Bulldozer
The synthetic Lego bulldozer dataset from Mildenhall et al. contains 100 inward-facing hemispherical views for training and 100 validation views at 800x800 resolution with ground-truth camera matrices in transforms_train.json and transforms_val.json.
If the dataset is not yet present in data/lego, we can automatically fetch it directly in Wolfram Language using DownloadNeRFDataset:
In[]:=
dataDir=FileNameJoin[{Directory[],"data","lego"}];
Download the dataset if it’s not here already. This utility function goes to a repository with various NeRF datasets and downloads them.
In[]:=
DownloadNeRFDataset[dataDir,"Scene"->"lego","Splits"->{"train","val"}]
Out[]=
Directory/Users/fxpppr/GitHub/WLNeRFLite/data/lego,Scenelego,Splits{train,val},JSONFilestrain/Users/fxpppr/GitHub/WLNeRFLite/data/lego/transforms_train.json,val/Users/fxpppr/GitHub/WLNeRFLite/data/lego/transforms_val.json,TotalFrames200,DownloadedFiles0,CachedFiles200
Now load the training split into memory using LoadNeRFDataset. We downsample the training views to 200x200 for rapid interactive training and set WhiteBackground -> True to blend transparent rays against pure white.
In[]:=
views=LoadNeRFDataset[dataDir,"Split"->"train","TargetDimensions"->{200,200},"WhiteBackground"->True];
In[]:=
Length[views]
Out[]=
100
The example dataset contains various useful bits and pieces for each view. Most important, the camera intrinsics and extrinsics, and respective images.
In[]:=
Keys[views[[1]]]
Out[]=
{CameraMatrix,FOV,Dimensions,RGB,Alpha,ForegroundPixels,BackgroundColor,File,FrameIndex}
So, for example, this is the first training view image:
In[]:=
{views[[1]]["Dimensions"],views[[1]]["FOV"]}
Out[]=
{{200,200},39.5978}
In[]:=
Image[views[[1]]["RGB"]]
Out[]=
3. Interactive 3D Camera Rig Visualization
3. Interactive 3D Camera Rig Visualization
WLNeRFLite automatically maps standard NeRF OpenGL poses (Z-backwards) to world camera frames. VisualizeNeRFCameraRig renders the 3D origin, camera orbital positions, and pyramidal viewing frustums pointing towards the Lego bulldozer:
In[]:=
VisualizeNeRFCameraRig[views[[1;;;;3]],"FrustumScale"->0.35,"PointSize"->0.015]
Out[]=
4. Differentiable Volumetric NetGraph Architecture
4. Differentiable Volumetric NetGraph Architecture
Standard volume rendering models radiant light transmission along ray between bounds . For stratified samples:
r(t)o+t*d
[,]
t
near
t
far
N
τ
i
σ
i
δ
i
T
i
∑
j<i
τ
j
α
i
τ
i
C(r)**+*
N
∑
i1
T
i
α
i
c
i
T
N+1
c
bg
where τ is optical thickness, is accumulated transmittance, α is the sample opacity, and the color.
T
C(r)
In WLNeRFLite, this entire computation—harmonic positional encoding, coordinate MLP, lower-triangular optical accumulation (trilStrict), and alpha compositing—is encapsulated in an end-to-end differentiable NetGraph without external C++/CUDA custom kernels.
Initialize a coordinate MLP with 32 stratified samples per ray, 8 harmonic frequency bands, 96 hidden units, and 3 layers:
In[]:=
net=NetInitialize@CreateNeRFLiteNet["NumSamples"->32,"HarmonicBands"->8,"HiddenDim"->96,"NumLayers"->3,"BackgroundColor"->{1.,1.,1.},"ColorActivation"->LogisticSigmoid]
Out[]=
NetGraph
5. High-Throughput Ray-Pool Training
5. High-Throughput Ray-Pool Training
Rather than evaluating entire images, WLNeRFLite pools candidate rays across all 100 views, sampling 1,024 random rays per batch with a 70% foreground bias (SampleForegroundRatio -> 0.7). This ensures the network concentrates on the intricate details of the Lego bulldozer rather than empty background.
Train for 3,000 rounds with real-time interactive progress reporting:
In[]:=
trainedModel=TrainNeRFLiteModel[net,views,"BatchSize"->1024,"MaxTrainingRounds"->3000,"LearningRate"->0.001,"Loss"->"L2","RayBounds"->{2.0,6.0},"SampleForegroundRatio"->0.7,"TargetDevice"->Automatic,"TrainingProgressReporting"->"Panel"]
Out[]=
NetGraph
Save the trained model checkpoint to disk:
In[]:=
outputDir=FileNameJoin[{Directory[],"output"}];If[!DirectoryQ[outputDir],CreateDirectory[outputDir]];Export[FileNameJoin[{outputDir,"lego_nerflite.wlnet"}],trainedModel]
6. Novel View Synthesis & Validation Evaluation
6. Novel View Synthesis & Validation Evaluation
Evaluate reconstruction fidelity against held-out validation poses from transforms_val.json (poses not seen during training):
First, we load the held-out validation views.
Here’s a sample ‘ground truth’ image:
Let’s look at a quantitative difference here:
Here’s a side by side comparison, along with an error map that is amplified by 3x.
7. 360-Degree Turntable Orbit Animation
7. 360-Degree Turntable Orbit Animation
RenderNeRFLiteTurntable generates a smooth circular camera trajectory orbiting the origin along the Z-axis (Z-up) and renders novel views from the trained field:
Optionally export the turntable animation to a self-contained GIF:
8. Continuous 3D Volumetric Field Sampling & Geometry Extraction
8. Continuous 3D Volumetric Field Sampling & Geometry Extraction
Sample 64x64x64 density grid in world space
Here’s the density image with the default color map:
It’s not too tough to add your own color map with ColorFunction:
The density volume can also be converted into a (somewhat messy looking) isosurface:
9. Summary & Forum Discussion Points
9. Summary & Forum Discussion Points
Summary of Key Concepts for Wolfram Community:
- Fully Native Architecture: Demonstrates that complex radiance fields do not require external CUDA kernels or PyTorch—Wolfram NetGraph cleanly handles end-to-end differentiable volumetric quadrature.
- Stratified Ray Pooling: Demonstrates efficient data generation with foreground sampling bias (>25,000 rays/sec).
- Standard NeRF Compatibility: Directly supports canonical transforms.json datasets (NeRF Synthetic, Instant-NGP, Nerfstudio) with native Wolfram Language dataset downloading via DownloadNeRFDataset.
- Fully Native Architecture: Demonstrates that complex radiance fields do not require external CUDA kernels or PyTorch—Wolfram NetGraph cleanly handles end-to-end differentiable volumetric quadrature.
- Stratified Ray Pooling: Demonstrates efficient data generation with foreground sampling bias (>25,000 rays/sec).
- Standard NeRF Compatibility: Directly supports canonical transforms.json datasets (NeRF Synthetic, Instant-NGP, Nerfstudio) with native Wolfram Language dataset downloading via DownloadNeRFDataset.
Original Paper: NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12346. Springer, Cham. https://doi.org/10.1007/978-3-030-58452-8_24
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Training a neural radiance field (NeRF) from scratch in Wolfram Language: the classic Lego bulldozer
by Flip Phillips
Wolfram Community, STAFF PICKS, November 16, 2026
https://community.wolfram.com/t/28600
by Flip Phillips
Wolfram Community, STAFF PICKS, November 16, 2026
https://community.wolfram.com/t/28600