Key Takeaway
One of the biggest bottlenecks for AI training clusters is the link flapping triggered by the physical link layers. There are ways, however, to prevent link flaps from happening based on advanced engineering in the optical transceivers.
Introduction
Today’s AI training cluster has been scaling at an unprecedented rate. The most expensive link isn't the one that fails — it's the one that flaps. Indeed, in our current industry, it is increasingly clear that bandwidth is no longer the most critical bottleneck, link stability is.
A recent study can be found in the publication [1] by Paul Borrill, entitled “The Ghost in the Datacenter: Link Flapping, Topology Knowledge Failures, and the FITO Category Mistake”. Here, the term “ghost” refers to link corruption caused by the link flaps. Based on the statistical results from this paper (See the following table), at 2025 scale, AI training clusters may have 3M GPUs and contain more than 10M optics. At such a scale, there are link flaps every 48 seconds. Consequently, this results in not only disruptive networking failures, but also big business costs nobody has budgeted for.