ABSTRACT: Biological neural networks (BNNs) are still qualitatively better than current artificial (ANNs) e.g. in learning, robustness, flexibility - just increasing the number of neurons seems insufficient to catch up. Such high level superiority comes for low level differences - found through evolution, more powerful mathematical tools hidden in dynamics of single biological neurons - we would like to understand and recreate. From statistics perspective, the most information a single neuron could gather is a model of joint distribution of its neighbors - we will use HCR (hierarchical correlation reconstruction) (hierarchical correlation reconstruction) as such model in proposed artificial neurons: for joint density of normalized variables as just a linear combination in chosen orthonormal basis. Such HCRNN parametrization allows for inexpensive calculations, can reduce to popular KAN-like: using only trained single-parameter functions and summations, what seems biologically plausible. Like BNNs, it also allows multidirectional propagation of values and probability distributions. Also for multiple additional learning approaches, like through information bottleneck method - by directly optimizing content of hidden layer communicating only with neighboring layers - what, in contrast to standard backpropagation, seems biologically possible. CITATION (original article): Jarek Duda, Biology-inspired joint distribution neurons based on Hierarchical Correlation Reconstruction allowing for multidirectional neural networks, arXiv:2405.05097. https://doi.org/10.48550/arXiv.2405.05097
This notebook introduces to HCR-based artificial neurons modelling local joint distributions. Here is introduction to HCR (also short below), article, and slides used in this presentation.

BNNs and current ANNs

Here is summary of the popular ANNs: MLP (multi-layer perceptron) and recent KAN (Kolmogorov-Arnold network) from https://arxiv.org/pdf/2404.19756. Both assume some guessed parametrization type to be optimized mainly by backpropagation:
They are very different from biological, and e.g. spiking neural networks try to reduce this difference. However, all of them are very far e.g. in learning, flexibility, robustness capabilities of BNNs, summarized below (source):
Above high level differences need to come from low level - qualitative differences between neurons. While current ANNs are simple parametrizations trained by backpropagation, biological have some limitations and might hide more sophisticated mathematics - optimized through millions of years of evolution. Searching for mathematical differences: 1) while current ANNs are optimized for unidirectional propagation, biological have multidirectional: ”it is not uncommon for axonal propagation of action potentials to happen in both directions” (from https://journals.aps.org/pre/abstract/10.1103/PhysRevE.92.032707 ), or https://en.wikipedia.org/wiki/Neural_backpropagation , 2) while current ANNs operate on values, biological need to also work with probability distributions, e.g. estimating uncertainties for distance evaluation, 3) while current ANNs are trained by backpropagation, it is claimed not being biologically plausible for biological (e.g. https://www.nature.com/articles/s41593-019-0520-2 ) - they need some different learning mechanisms, e.g. using only communication with neighboring layers. A basic approach seems reduction of prediction error by the succeeding layer, its choice independent mathematical description is maximization of mutual information with the next layer. Additionally, as noticed by information bottleneck method, it is worth to simultaneously reduce mutual information with the previous layer - to remove noise, extract only information crucial for prediction. Thinking about recreation of the above properties in artificial neurons, all can be achieved if neuron contains model of joint distribution of its neighbors - complete statistical description of information neuron has access to, hence millions of years of evolution should find a way to exploit it. Such neuron could propagate probability distributions: substituting some of variables it can calculate conditional distributions for the remaining, in practice can be restricted e.g. to expected values.
Modeling general joint distributions is a nontrivial tasks - classical methods: kernel density estimation and copulas have poor performance and are costly - we will use hierarchical correlation reconstruction (HCR) instead to build such model: for variables normalized to ~uniform[0,1], model joint density as just a linear combination e.g. polynomial. Below is comparison with KDE of log-likelihood in cross validation for the shown example dataset: the best evaluation was obtained by m=4 degree polynomial, while KDE barely got above 0 baseline: obtained also be ρ = 1 trivial assumption. Polynomials work on moments describing global trends from dataset, while KDE focuses on local trends: predicts that further points will be close to the old ones - being much worse at generalization. Additionally, HCR is very simple computationally: models joint density of normalized variables as just a linear combination in orthonormal basis. It can be operated as KAN-like parametrization: with trained single parameter functions and summations, what seems biologically plausible: might be hidden in BNN dynamics. Also HCR-based neural networks (HCRNN) allow for multiple additional training approaches, e.g. by direct parameter estimation, mentioned information bottleneck, or through linearization to tensor decomposition problem.

HCR introduction and its artificial neuron

KAN-like example

Training intermediate layers

CITE THIS NOTEBOOK

HCRNN: biology-inspired joint distribution artificial neural networks reducing to KAN-like​
by Jarek Duda​
Wolfram Community, STAFF PICKS, August 6, 2024
​https://community.wolfram.com/groups/-/m/t/3241700