When you think of AI, and Nvidia’s GPUs often come to mind because they have been so much a part of the ongoing AI revolution.
But at the recent Hot Chips show in California, an alternative future appears to be taking shape as corporate concerns continue to grow about cost, energy use and AI slowdowns.
New AI chips detailed at the event by OpenAI, Intel, Meta and others promise cheaper and faster token generation at lower power consumption. Chipmakers are also moving AI away from GPUs and onto CPUs and PCs.
“What Hot Chips demonstrated…is that the market underneath Nvidia is becoming much more diverse,” said Stephen Sopko, an analyst at Hyperframe Research.
Though enterprise AI budgets continue to go up, the focus is increasingly on finding ways to make AI more productive and reducing waste in the budget. “But I would not tell CIOs that their AI budgets are therefore going down,” Sopko said.
Hot Chips pointed to a future where more AI work can be squeezed out of the same watt, rack or dollar with new hardware designs and power management. “The cost of performing a given unit of AI work should continue falling dramatically,” Sopko said.
OpenAI shared details of its homegrown AI chip called Jalapeño, which will help the company “serve more demand and lower the cost of delivering a successful result,” according to a blog entry.
OpenAI started its AI journey with Nvidia GPUs, and plans to spend $750 billion on data centers to power its tool and services. The Jalapeño chip will presumably spice up those data centers for inferencing.
Research firm SemiAnalysis was granted exclusive access to test the chip and came away impressed with what it found. “In general, first-generation chips are not competitive, but OpenAI bucks the trend by being industry-leading and beating every Nvidia, AMD, and Google chip we have been able to test on multiple top open source models,” SemiAnalysis said in a newsletter report.
AI compute will over time be more distributed than it has been in recent years, but cloud will continue to be an important option for highly compute-intensive requirements, said Jack Gold, principal analyst at J. Gold Associates.
Within the next year or two, thousands of agents will be running on edge platforms, AI PCs, localized servers or sovereign data centers, Gold said. “It’s going to be a larger number of vendors’ chips running different apps that are not all GPUs from one vendor,” he said.
GPUs can be costly from a power and performance perspective for agentic AI, which is “why you see the major GPU players like Nvidia and AMD emphasize new AI-friendly CPU architectures,” Gold said.
Nvidia has recognized the need for chips beyond its own GPUs for inferencing. At Hot Chips, the company shared details about its Vera CPU and Groq 3 LPX inferencing chip, for instance.
Meanwhile, Intel talked up an upcoming server CPU called Diamond Rapids, and a PC CPU called Wildcat Lake; the latter is designed to bring AI to edge devices and low-cost laptops.
Diamond Rapids has AI extensions so inferencing can be done on the CPU without redirecting the workload to other co-processors. The Wildcat Lake chip has neural processing units (NPUs) and borrows features from Intel’s Xe3 graphics chip, which is also used in the company’s AI inferencing GPUs. (It’s similar to Intel’s existing Panther Lake chip.)
