Guide to Understanding CPU Inference
This comprehensive guide provides a deep dive into processing AI workloads on CPUs and the use cases for which this may be the practical choice. Explore the industries that are already benefiting from CPU inference and learn about real-world examples.
Which AI Workloads Run Best on the CPU?
.svg?rev=2512246e0ed34c6abb7d7dc5ffadf468&revision=2512246e-0ed3-4c6a-bb7d-7dc5ffadf468&la=en&hash=74A47C6518DDCDFD050930B696A07E74F24A935C){kind=icon}
Traditional ML and Deep Learning
Applications that combine machine learning (ML) and deep learning, such as cinematic photography, media and audio processing, automotive workloads (including digital cockpit and ADAS L2+), real-time voice assistants, real-time analytics and recommendations (for social media and e-commerce), and natural language processing (including sentiment analysis).
.svg?rev=8bf0ecad2a5247c29ce59d1ea8af4a25&revision=8bf0ecad-2a52-47c2-9ce5-9d1ea8af4a25&la=en&hash=0193BEBCC0616984DC567720E3C9FC68FB475CFD){kind=icon}
Generative AI
Generative AI use cases, including those that make use of small language models (SLMs) and LLMs, for example, conversation summarization, virtual assistants, customer UX (including agents and chat bots), content translation and summarization, plus content generation.
Two Decades of AI Architecture Innovation
Arm has a focus on fast-paced architectural innovation that prepares our vast ecosystem for ever-changing compute requirements and the future of AI. Arm has consistently and proactively evolved the AI capabilities of our CPUs over two decades with features such as Neon, Helium, Scalable Vector Extension (SVE), Scalable Matrix Extension (SME), and others. The latest Armv9 architecture features are driving increased compute performance, alongside reduced power consumption for AI workloads.
Accelerating Workloads with the Leading AI Frameworks
Arm’s partnerships with the leading AI frameworks and operating systems helps ensure fast and easy deployment for scaling AI workloads across Arm CPUs. We support key partners with techniques for optimizing new models using quantization and open-source software for AI acceleration, such as Arm Kleidi, used by frameworks and independent software vendors. Targeting accelerations at the AI framework level helps drive AI accelerations on Arm CPUs most broadly, across billions of AI inference installs for workloads at the edge, on mobile, and in the cloud. Without any extra optimization effort, application developers can expect best performance for their AI workloads by default on Arm CPUs, thanks to our work across gaming, computer vision, and language models.
