At the 70th edition of the IT Press Tour in Palo Alto, Clockwork.io delivered a refreshingly direct keynote. Its engineers pulled a network cable from a machine running an AI training job, and the job continued. They then powered down an active server. Training paused for roughly a minute, moved to replacement hardware, and resumed at the same step.
The demo exposed one of AI's least glamorous and costly problems. Companies can spend heavily on GPUs, networking, power, cooling, and storage, yet lose a meaningful share of that investment if one component fails.
Clockwork describes its solution as a Software-Driven AI Fabric, delivered through its FleetIQ platform. The language sounds ambitious. But the underlying proposition is easy to understand. Hardware will fail, and. the surrounding software should prevent every fault from becoming a full-cluster interruption.
The hidden cost of idle GPUs
Large AI training jobs operate differently from traditional business applications. Thousands of GPUs exchange information and advance through training steps together. But one faulty connection, accelerator, server, or process can grind a workforce to a halt.
Clockwork cited Meta research showing a mean time to failure of 7.9 hours for jobs running across 1,024 GPUs. It also referenced ByteDance research finding that 42.5 percent of jobs ran at least 10 percent slower because of stragglers. The precise result will vary between clusters, but the wider lesson is clear. As the number of components grows, the probability of something going wrong rises.
When something breaks, engineers typically identify the fault, restart the job, reload the most recent checkpoint, warm the cluster, and repeat everything completed since that checkpoint.
Clockwork said manual fault detection can take between one and three hours, while restarting a large training environment can take up to another 30 minutes. During this time, healthy GPUs remain powered but unproductive. This is where the industry needs a better measure than utilization alone.
Turning detection into action
Clockwork's origin story can be traced back to Stanford research on synchronizing clocks across thousands of machines without specialized timing hardware. At the time it was financial organizations that had originally adopted the technology to establish the exact order of transactions. But the company later recognized that accurate time could help diagnose AI infrastructure.
If every machine agrees on time, software can measure one-way network-traffic delay and pinpoint where performance is deteriorating. Instead of telling an operator a training job is slow, it can identify the responsible link, port, node, or GPU.
FleetIQ combines several capabilities. FleetAudit tests infrastructure before production. FleetLens locates operating faults. LinkPass redirects traffic when a network connection fails. TorchPass moves work from a failed GPU or node to healthy replacement capacity. But the only way to prove how Clockwork connects detection, localization, and recovery in one layer was quite literally to pull the plug.
What the live demonstration proved
When the Clockwork engineer removed the network cable, tech journalists were waiting to get their next headline. But LinkPass redistributed traffic across the remaining healthy interfaces.
The node failure was an even tougher test. But after the server lost power, TorchPass quickly detected the missing worker and provisioned a hot spare. The job stopped at training step 5,040 and resumed at step 5,040 around a minute later. Most importantly, it recovered state from healthy replicas rather than loading an older checkpoint.
Clockwork says network failures account for approximately 15 to 20 percent of the faults it encounters. LinkedIn has deployed LinkPass and reports that it prevents tens of thousands of GPU-hours of downtime each month. Together AI brings LinkPass and TorchPass to customers through its GPU cluster service, while WhiteFiber uses FleetAudit to identify hidden fabric problems before customers run jobs.
Resilience still has a price.
Live migration still needs replacement capacity. Clockwork said cloud operators commonly reserve between five and eight percent of their fleet for failures, although those machines can run lower-priority work until required. An organization operating every GPU at full allocation may have nowhere to move a failed workload.
FleetIQ is generally licensed per GPU per year. Buyers must compare the software and spare-capacity costs against the value of GPU time lost to detection, recovery, and recomputation.
Checkpoint and restart is also a competitor, even if nobody thinks of it as a product. Many organizations already tolerate the delay because the process is familiar. Clockwork must prove that its software reduces enough waste to justify deployment and operational change.
Clockwork announced another $31 million in funding, bringing its total to $73 million. The bigger question is whether enterprises and cloud providers are ready to treat resilience as a capacity source.
Buying additional GPUs is an obvious response to demand. Recovering capacity already lost to failures may offer a better return. Completing more useful work without consuming additional space or power can improve training schedules and customer economics.
Tech leaders evaluating AI infrastructure need to start asking different questions.
How often do its large jobs fail?
How long does detection and recovery take?
How much work is repeated?
What percentage of paid GPU time produces new progress?
Is spare capacity available, and can the recovery system prove that it resumed from the correct state?
Clockwork Chief Business Officer Dan Zheng summarized the briefing with the line, "Efficiency is the new capacity."
The company still has to show consistent results across varied production environments, but its live demo made the problem hard to dismiss. In an industry obsessed with getting more computing power, it may be smarter to start by making better use of what you've already purchased.
I will be speaking with the Clockwork team on my podcast. If you have any questions you would like me to ask on your behalf, please let me know, and I will bring them into the conversation.
Special thanks to Alex Zaharov-Reutt, iTWire's Global AI and Technology Editor, for filming the presentation and allowing me to use the footage.