What if in-order, lossless, exactly-once message delivery wasn’t patched in at the edges, but was built into the fabric of the network itself? I’ve developed a cellular automaton rule that models packet and acknowledgment propagation as a deterministic, lossless flow, revealing what a truly ordered substrate might look like. It shows how reliable delivery and strict causal ordering could emerge from local interactions, not global guesswork. Real networks aren’t clean, so noise, contention, and congestion back-pressure must be modeled directly in the transport, not ignored and handled only after failure.
Introduction
Introduction
Imagine the billions of dollars a day flowing through our financial markets daily. They are built on communication networks that drop, delay, reorder and duplicate events. The services that we need to be rock solid in operation, like databases, bank systems, energy systems rely on the most price-performant scalable infrastructure available: Ethernet/IP. This is the equivalent of building a courier system where we send packets via messengers on bikes, and we tell them “pedal hard, pedal fast, and don’t stop pedalling. It doesn’t matter if you make it to your destination, get lost, or get black holed. All that matters is you pedal as fast as you can.”
We trust, as we do when constructing communication systems at human scales, that reliability is a requirement at the end-to-end level. If you want to get something done, you assign a team to the task, and they will get it done regardless of missed emails, sick days, vacations, or fire. The equivalent in networks is an orchestra of containers, databases, block storage, load balancers, hosted on a cloud provider that can only guarantee a “asymptote” of 99.9% availability and 99.9% application consistency. The industry has traded semantic clarity and atomicity for scalability, and the stretch marks are starting to show.
Ethernet/IP/TCP: The Foundation of Modern Networking
Ethernet/IP/TCP: The Foundation of Modern Networking
At this point, I should make it clear Ethernet is not type of plug, it' s the biggest of a few competing protocols, it's a standard interconnect between processors .
An interconnect in the context of computer networks refers to a communication system that connects various components within a computer or between computers. It enables data exchange and communication between processors, memory, storage devices, and other peripherals.
Ethernet is one of the most common interconnect protocols due to its widespread adoption and cost-effectiveness. It provides a standard way of connecting devices within a network, enabling data exchange over various scales, from local networks (LANs) to broader networks (WANs).
Other interconnect protocols, like InfiniBand, are used for specialized environments, offering features like low latency and high bandwidth, which are essential for high-performance computing (HPC) and applications requiring rapid data transfer. These protocols can be proprietary and are often employed in scenarios where performance and reliability are critical.
Ethernet is one of the most common interconnect protocols due to its widespread adoption and cost-effectiveness. It provides a standard way of connecting devices within a network, enabling data exchange over various scales, from local networks (LANs) to broader networks (WANs).
Other interconnect protocols, like InfiniBand, are used for specialized environments, offering features like low latency and high bandwidth, which are essential for high-performance computing (HPC) and applications requiring rapid data transfer. These protocols can be proprietary and are often employed in scenarios where performance and reliability are critical.
IP, the (I)nternet (P)rotocol layer, is a standard for naming compute nodes, and giving processors identities. It gives computers IP addresses on the network which allows other nodes to send traffic to it. IP makes no guarantees about delivery, it is simply a protocol for naming and routing to different names. IPv6, as compared to IPv4, adds some missing security and tools, but is fundamentally the same idea with the ability to have 128bit names compared to 32bit names.
The End-To-End Principle
The End-To-End Principle
At the foundation of modern networked systems lies Ethernet and IP, a stack designed for best-effort delivery. These protocols make no guarantees about whether a message arrives, when it arrives, how many times it arrives, or in what order. Packets may be dropped, delayed, duplicated, or reordered en route. This simplicity allows for scalability and speed, but comes at the cost of reliability.
To handle this, the end-to-end (E2E) principle was adopted: the network remains intentionally “dumb,” while endpoints (the application layer) are responsible for ensuring reliability. This works well when a single machine coordinates system state, allowing it to manage retransmissions and correctness locally. Historically, this was sufficient—many early systems centralized logic and state, making coordination straightforward.
However, in distributed systems, state is split across multiple nodes that must coordinate under uncertain network conditions. A packet silently dropped or delayed beyond a known window introduces a fundamental ambiguity: the receiving node cannot know if the event occurred, or if it’s merely late. This epistemic uncertainty is not a temporary glitch—it is a structural barrier. Higher-level protocols attempt to patch over this with retries, timeouts, and consensus mechanisms, but these cannot fully restore correctness. In the presence of network faults, illusions of atomicity and consistency break down, leading to race conditions and irrecoverable anomalies in application logic.
Ethernet/IP’s strength is its simplicity and ubiquity—but its very design undermines coordination when correctness across distributed systems is required.
To handle this, the end-to-end (E2E) principle was adopted: the network remains intentionally “dumb,” while endpoints (the application layer) are responsible for ensuring reliability. This works well when a single machine coordinates system state, allowing it to manage retransmissions and correctness locally. Historically, this was sufficient—many early systems centralized logic and state, making coordination straightforward.
However, in distributed systems, state is split across multiple nodes that must coordinate under uncertain network conditions. A packet silently dropped or delayed beyond a known window introduces a fundamental ambiguity: the receiving node cannot know if the event occurred, or if it’s merely late. This epistemic uncertainty is not a temporary glitch—it is a structural barrier. Higher-level protocols attempt to patch over this with retries, timeouts, and consensus mechanisms, but these cannot fully restore correctness. In the presence of network faults, illusions of atomicity and consistency break down, leading to race conditions and irrecoverable anomalies in application logic.
Ethernet/IP’s strength is its simplicity and ubiquity—but its very design undermines coordination when correctness across distributed systems is required.
Reliable Communication over an Unreliable Substrate
Reliable Communication over an Unreliable Substrate
TCP (Transmission Control Protocol) provides a reliable, ordered, and error-checked delivery service on top of the unreliable IP network layer. While IP handles packet routing between hosts, TCP creates the abstraction of a reliable communication channel between applications.
TCP transforms the chaotic, packet-based internet into what appears to be a simple bidirectional pipe between two endpoints. From an application’s perspective, you can write data to one end and it will arrive intact and in order at the other end even though underneath, packets may be lost, duplicated, reordered, or corrupted as they traverse the network.
TCP transforms the chaotic, packet-based internet into what appears to be a simple bidirectional pipe between two endpoints. From an application’s perspective, you can write data to one end and it will arrive intact and in order at the other end even though underneath, packets may be lost, duplicated, reordered, or corrupted as they traverse the network.
In[]:=
edgeTable=;UnreliableGraph[edgeTable_]:=Module;UnreliableGraph[edgeTable]
Out[]=
Consider a network with the SENDER and RECEIVER as nodes on a graph. Intermediate nodes are switching stations which can take packets from one link, and output it on another. An IP datagram contains the source and destination address, and each intermediate node must make a decision upon receipt which next hop to send it on. An IP datagram may be split into multiple individual packets by the ethernet layer if they exceed a maximum transmission size, so an acknowledgement can only be sent after the entire IP datagram arrives at the receiver. Most traffic on networks are small datagrams so this is usually a 1:1 relationship, but I have animated a large packet getting split into multiple transmissions.
Out[]=
If any of these packets are dropped, the sender will retransmit frames after a timeout. Some TCP implementations send a signal from the receiver to initiate a retransmission if the datagram arrives mangled -- the idea is to rely on the sender timeout as little as possible.
To show how latency is extremely volatile to packet loss, I’ve modeled each edge as a 1us propagation delay with a 0.99% of successful transmission. I want to ask a simple question, what is the expected time to delivery given 4 hops and 0.99% reliability on each link.
If any of these packets are dropped, the sender will retransmit frames after a timeout. Some TCP implementations send a signal from the receiver to initiate a retransmission if the datagram arrives mangled -- the idea is to rely on the sender timeout as little as possible.
To show how latency is extremely volatile to packet loss, I’ve modeled each edge as a 1us propagation delay with a 0.99% of successful transmission. I want to ask a simple question, what is the expected time to delivery given 4 hops and 0.99% reliability on each link.
In[]:=
{pData,dData}=;walkEdges[w_]:=UndirectedEdge@@@(Sort/@Partition[w,2,1]);PSuccess[w_]:=Times@@(pData/@walkEdges[w]);DTotal[w_]:=Total@(dData/@walkEdges[w]);walk={"SENDER","A","B","D","RECEIVER"};TemplateApply
Out[]=
A walk from SENDER to RECEIVER has a 96.06% chance of success with 4.0μs delay
Latency is a measure of the round trip delay from the sender’s perspective. Therefore a measure of expected time to delivery must account for the arrival of all individual packet transmissions at the receiver, and the arrival of the acknowledgement back to the receiver.
framePath=<|"dataPaths"->{{"SENDER","A","B","D","RECEIVER"},{"SENDER","A","C","D","RECEIVER"},{"SENDER","A","B","D","RECEIVER"}},"ackPath"->{{"RECEIVER","D","B","A","SENDER"}}|>
In “normal” operation where congestion is negligible, latency is predictable, and packet loss is rare, TCP is adequate. However, as the industry saying goes “TCP works... until it doesn’t.” When the network is congested, and too many flows are competing over the same links, the intermediate nodes will drop packets it cannot fit in it’s internal buffers. Packet reliability, and by extension latency distributions, are very dependent on the network weather for the day. Positive control loops where retransmissions cause congestion which cause more retransmissions can wipe out entire rows of the datacenter.
Everything about the way we reason in networks is probabilistic, and hidden somewhere in the 0.01% is the total destruction of the application.
Everything about the way we reason in networks is probabilistic, and hidden somewhere in the 0.01% is the total destruction of the application.
Networks for AI
Networks for AI
No conversation about the future of networks is complete without a discussion about AI
NVIDIA’s 2020 acquisition of Mellanox signaled their ambition to become a full-stack datacenter company. Mellanox, known for pioneering InfiniBand -- a lossless, low-latency, high-bandwidth alternative to Ethernet -- brought NVIDIA the networking backbone for AI scale infrastructure. As Jensen Huang echoed Sun Microsystems’ mantra, “The network is the computer,” NVIDIA began collapsing the boundaries between compute and communication. Their Spectrum-X platform, combining BlueField DPUs and Spectrum-4 Ethernet switches, delivers near-Infiniband performance over Ethernet, enabling direct, high-throughput GPU-to-GPU connectivity. Lossless delivery is possible, and it works, but it remains confined to proprietary stacks and specialized environments like HPC and storage networks, limiting its reach beyond elite datacenter deployments.
AI models, for both training and inference, are spread across multiple GPU chips, and the pod must collaborate to complete each training step or forward pass. There are a few network primitives that are compiled in NVIDIA’s communication libraries that represent the characteristic traffic patterns of AI.
NVIDIA’s 2020 acquisition of Mellanox signaled their ambition to become a full-stack datacenter company. Mellanox, known for pioneering InfiniBand -- a lossless, low-latency, high-bandwidth alternative to Ethernet -- brought NVIDIA the networking backbone for AI scale infrastructure. As Jensen Huang echoed Sun Microsystems’ mantra, “The network is the computer,” NVIDIA began collapsing the boundaries between compute and communication. Their Spectrum-X platform, combining BlueField DPUs and Spectrum-4 Ethernet switches, delivers near-Infiniband performance over Ethernet, enabling direct, high-throughput GPU-to-GPU connectivity. Lossless delivery is possible, and it works, but it remains confined to proprietary stacks and specialized environments like HPC and storage networks, limiting its reach beyond elite datacenter deployments.
AI models, for both training and inference, are spread across multiple GPU chips, and the pod must collaborate to complete each training step or forward pass. There are a few network primitives that are compiled in NVIDIA’s communication libraries that represent the characteristic traffic patterns of AI.
NCCL (NVIDIA Collective Communications Library) provides a set of primitives designed to optimize communication between GPUs, especially in distributed training of deep learning models. Here are some of the key NCCL primitives:
1. AllReduce: Combines data across all GPUs and distributes the result back to each GPU. This operation is crucial for synchronizing gradients during distributed training.
2. Broadcast: Sends data from one GPU to all other GPUs. This is often used to distribute model weights from a master GPU to others at the start of training.
3. Reduce: Combines data from all GPUs into one GPU, typically used for aggregating statistics or results.
4. AllGather: Collects data from each GPU and aggregates it into a single buffer on each GPU. This operation is useful when each GPU needs a full copy of results generated by other GPUs.
5. ReduceScatter: Combines data from all GPUs and scatters the results back to each GPU. This is efficient for scenarios where a reduced result is only needed on a subset of the GPUs.
These primitives are optimized for high-performance, low-latency, and high-bandwidth communication, essential for efficient distributed deep learning workloads.
1. AllReduce: Combines data across all GPUs and distributes the result back to each GPU. This operation is crucial for synchronizing gradients during distributed training.
2. Broadcast: Sends data from one GPU to all other GPUs. This is often used to distribute model weights from a master GPU to others at the start of training.
3. Reduce: Combines data from all GPUs into one GPU, typically used for aggregating statistics or results.
4. AllGather: Collects data from each GPU and aggregates it into a single buffer on each GPU. This operation is useful when each GPU needs a full copy of results generated by other GPUs.
5. ReduceScatter: Combines data from all GPUs and scatters the results back to each GPU. This is efficient for scenarios where a reduced result is only needed on a subset of the GPUs.
These primitives are optimized for high-performance, low-latency, and high-bandwidth communication, essential for efficient distributed deep learning workloads.
All the NCCL primitives are latency sensitive because they are synchronous, collective operations, The entire system must wait for all participants to complete before continuing. This means tail latencies caused by congestion and packet loss in TCP have a drastic performance impact, wasting time, and ultimately energy.
The current landscape introduces significant friction for non-NVIDIA players. InfiniBand and lossless Ethernet deployments are costly and complex, and standard TCP based Ethernet remains prone to tail latency, making it a poor fit for AI-scale collectives. This entrenches NVIDIA’s position as the only vendor offering tight integration between high-performance GPUs and high-throughput interconnects. For startups, hyperscalers, and open-source innovators, this represents a formidable barrier to entry. The industry is caught in a local minimum, where performance comes at the cost of openness and flexibility.
The current landscape introduces significant friction for non-NVIDIA players. InfiniBand and lossless Ethernet deployments are costly and complex, and standard TCP based Ethernet remains prone to tail latency, making it a poor fit for AI-scale collectives. This entrenches NVIDIA’s position as the only vendor offering tight integration between high-performance GPUs and high-throughput interconnects. For startups, hyperscalers, and open-source innovators, this represents a formidable barrier to entry. The industry is caught in a local minimum, where performance comes at the cost of openness and flexibility.
Exactly-Once, In-Order, Hop-By-Hop, Fault-Tolerant
Exactly-Once, In-Order, Hop-By-Hop, Fault-Tolerant
These words, put together as a requirement, should send a shiver down the spine of any backend engineer, followed by a discussion about a reduction in scope. But what if they were treated not as application-layer aspirations, but as first-class properties of the network substrate?
A new direction is emerging. Atomic Ethernet explores an alternative vision for AI scale interconnects: one that builds on the ubiquity of Ethernet, but reimagines its delivery semantics for loss sensitive, high throughput, collective workloads. Instead of bolting reliability onto lossy links through complex transport layers, Atomic Ethernet treats delivery and acknowledgment as first-class primitives within the link layer itself, enabling low-latency, in-order communication across standard infrastructure.
By embedding reliability and flow coordination into the packet fabric, Atomic Ethernet aims to reduce synchronization overhead, eliminate tail stalls, and allow networks to carry AI workloads natively without relying on closed stacks or expensive transport overlays. This approach could lower the barrier to entry for building competitive AI clusters, helping decouple performance from proprietary infrastructure and restoring openness to the datacenter interconnect layer.
A new direction is emerging. Atomic Ethernet explores an alternative vision for AI scale interconnects: one that builds on the ubiquity of Ethernet, but reimagines its delivery semantics for loss sensitive, high throughput, collective workloads. Instead of bolting reliability onto lossy links through complex transport layers, Atomic Ethernet treats delivery and acknowledgment as first-class primitives within the link layer itself, enabling low-latency, in-order communication across standard infrastructure.
By embedding reliability and flow coordination into the packet fabric, Atomic Ethernet aims to reduce synchronization overhead, eliminate tail stalls, and allow networks to carry AI workloads natively without relying on closed stacks or expensive transport overlays. This approach could lower the barrier to entry for building competitive AI clusters, helping decouple performance from proprietary infrastructure and restoring openness to the datacenter interconnect layer.
1
.Exactly Once
Every transmission should result in exactly one reception, or a failure that is deterministically known by the sender. No duplicates, no limbo states. This is the minimum semantic contract required for reasoning about transactions, causal order, and physical-world operations (like controlling a robotic actuator or committing a bank transfer). In an exactly-once substrate, protocol retries are no longer guesses; they are state transitions through a recoverable graph.
Every transmission should result in exactly one reception, or a failure that is deterministically known by the sender. No duplicates, no limbo states. This is the minimum semantic contract required for reasoning about transactions, causal order, and physical-world operations (like controlling a robotic actuator or committing a bank transfer). In an exactly-once substrate, protocol retries are no longer guesses; they are state transitions through a recoverable graph.
2
.In-Order
Packet order should be preserved across the delivery path, not restored after the fact. Eliminating reordering buffers at the receiver avoids protocol induced latency, simplifies program logic, and removes tail latency from the application’s reasoning model. When every message arrives in the order it was sent, even under loss and delay, we regain the ability to correlate cause and effect at the network level, not just in postmortems.
Packet order should be preserved across the delivery path, not restored after the fact. Eliminating reordering buffers at the receiver avoids protocol induced latency, simplifies program logic, and removes tail latency from the application’s reasoning model. When every message arrives in the order it was sent, even under loss and delay, we regain the ability to correlate cause and effect at the network level, not just in postmortems.
3
.Hop-By-Hop
Few systems operate in single-hop worlds. A realistic network path includes a sequence of unreliable intermediaries: switches, NICs, links, and queues. Guarantees must be composable across hops, with local forward progress and recovery semantics. This enables fast failover, bounded retransmission windows, and localized health monitoring, rather than relying on brittle end to end retries and timeouts. A hop-by-hop model turns the network from a passive black box into an active chain of accountable actors.
Few systems operate in single-hop worlds. A realistic network path includes a sequence of unreliable intermediaries: switches, NICs, links, and queues. Guarantees must be composable across hops, with local forward progress and recovery semantics. This enables fast failover, bounded retransmission windows, and localized health monitoring, rather than relying on brittle end to end retries and timeouts. A hop-by-hop model turns the network from a passive black box into an active chain of accountable actors.
4
.Fault-Tolerant
A fault-tolerant delivery model treats every transmission as a recoverable transaction, with well-defined rollback or replay semantics. Temporary unavailability of a node, queue overflow, or transient congestion should not break global invariants. The system must guarantee that information is never silently dropped or duplicated, even in the presence of partial failures. No split-brain connections, no half-committed states. Fault tolerance means building with the expectation that the network will fail, but correctness will not.
A fault-tolerant delivery model treats every transmission as a recoverable transaction, with well-defined rollback or replay semantics. Temporary unavailability of a node, queue overflow, or transient congestion should not break global invariants. The system must guarantee that information is never silently dropped or duplicated, even in the presence of partial failures. No split-brain connections, no half-committed states. Fault tolerance means building with the expectation that the network will fail, but correctness will not.
An Atomic Ethernet
An Atomic Ethernet
To evaluate the feasibility of these requirements, I went looking for a cellular automata rule that guarantees in order delivery through congestion and errors, as well as the conservation of packets in transit.
Packets (Blue) move to the right hop by hop, and ACK (Yellow) bounce off the receiver for each packet, and follow the same propagation rules to the sender. This is meant to represent the successful receipt of a packet, with the generation of an acknowledgement for a round trip interaction.
When a packet arrives at the receiver during an error state (Light Red), a negative acknowledgement (Red) is generated, and propagated backwards. This is the first signal that something is wrong and must be retransmitted. Packets (Blue) that arrive after the negative acknowledgements are deleted, and the implicit NACK is carried with the red packet. At this point the sender has not no knowledge of the failure, so it continues sending packets until the NACK arrives.
When a packet arrives at the receiver during an error state (Light Red), a negative acknowledgement (Red) is generated, and propagated backwards. This is the first signal that something is wrong and must be retransmitted. Packets (Blue) that arrive after the negative acknowledgements are deleted, and the implicit NACK is carried with the red packet. At this point the sender has not no knowledge of the failure, so it continues sending packets until the NACK arrives.
So far, congestion has not been a problem. Once the network has spurious delays due to contention, senders must be throttled to maintain full utilization of the network. To implement this “back-pressure,” I took inspiration from Rule 226 and 184. It simulates a 1-way traffic flow where a cell will move to the next spot only if there’s space. When there isn’t space to go forward, it halts, always preserving the same number of packets in the automata.
These images show a noisy channel with random contention (gray cells) that halt the propagation of packets and acknowledgments. In the p=0.01 case, you can see the ripples of back-pressure to the sender and receiver, preventing more packets from entering the channel and losing packets.
Concluding Remarks
Concluding Remarks
Computing is no longer speed bound, it is coordination bound. As models scale across thousands of chips, the ability to move data predictably, in order, and with minimal overhead has become the new frontier. But today’s networks were not built with this in mind. Ethernet, the backbone of modern infrastructure, was designed for bursty, best-effort traffic, not deterministic, collective computation. TCP papers over this mismatch, but at a steep cost: wasted energy, long-tail latencies, and brittle assumptions about delivery.
Efforts like InfiniBand and Fibre Channel show that lossless, low-latency delivery is achievable but at the cost of openness, simplicity, and global reach. These systems work well, but only in carefully controlled environments. Meanwhile, the explosion in AI demand is pushing infrastructure to its breaking point. As collective operations dominate workloads, and every millisecond of tail latency multiplies across thousands of GPUs, it becomes clear: the semantics of the network must evolve.
We need more than speed, we need semantic guarantees. Ordered delivery. Timely acknowledgment. Flow control that is native to the link itself. A language of coordination that is not abstracted away in layers, but built directly into the substrate. This isn’t just about making Ethernet better. It’s about reclaiming the network as a programmable, deterministic medium for distributed thought.
Efforts like InfiniBand and Fibre Channel show that lossless, low-latency delivery is achievable but at the cost of openness, simplicity, and global reach. These systems work well, but only in carefully controlled environments. Meanwhile, the explosion in AI demand is pushing infrastructure to its breaking point. As collective operations dominate workloads, and every millisecond of tail latency multiplies across thousands of GPUs, it becomes clear: the semantics of the network must evolve.
We need more than speed, we need semantic guarantees. Ordered delivery. Timely acknowledgment. Flow control that is native to the link itself. A language of coordination that is not abstracted away in layers, but built directly into the substrate. This isn’t just about making Ethernet better. It’s about reclaiming the network as a programmable, deterministic medium for distributed thought.
Acknowledgments
Acknowledgments
I am deeply grateful to Dugan Hammock, whose willingness to bounce ideas and share his Mathematica expertise helped sharpen both the technical and conceptual foundations of this work.
To Daniel Carvalho, thank you for the wide-ranging conversations on cellular automata, systems engineering, and the art of consulting. Your insights challenged assumptions and opened new directions.
To Rafael Caballero, John McNally, Christian Pasquel, Phileas Dazeley-Gaist, and Bowen Ping, thank you for being awesome collaborators and for bouncing ideas around with energy, curiosity, and generosity.
Special thanks to Stephen Wolfram for taking the time to listen to the full arc of this project and for distilling it into something focused, meaningful, and achievable in a matter of weeks.
Finally, thank you to everyone at the Wolfram Summer School for creating a rich environment of dialogue, rigor, and creativity. Your intellectual generosity and shared curiosity made this work possible.
To Daniel Carvalho, thank you for the wide-ranging conversations on cellular automata, systems engineering, and the art of consulting. Your insights challenged assumptions and opened new directions.
To Rafael Caballero, John McNally, Christian Pasquel, Phileas Dazeley-Gaist, and Bowen Ping, thank you for being awesome collaborators and for bouncing ideas around with energy, curiosity, and generosity.
Special thanks to Stephen Wolfram for taking the time to listen to the full arc of this project and for distilling it into something focused, meaningful, and achievable in a matter of weeks.
Finally, thank you to everyone at the Wolfram Summer School for creating a rich environment of dialogue, rigor, and creativity. Your intellectual generosity and shared curiosity made this work possible.
Cite This Notebook
Cite This Notebook
Towards a Practical Scheme for In-Order Delivery
by Sahas Munamala
Wolfram Community, STAFF PICKS, July 9, 2025
https://community.wolfram.com/groups/-/m/t/3496439
by Sahas Munamala
Wolfram Community, STAFF PICKS, July 9, 2025
https://community.wolfram.com/groups/-/m/t/3496439