The Santa Clara company continues to open up its technologies little by little, putting them directly into the hands of developers. We have already seen the toolkit OpenVINO, which can be used by any programmer to implement and optimize deep learning models. It is a software tool that helps you develop scalable AI solutions and efficient with just a few lines of code. Audacity developers used OpenVINO to introduce AI-based effects, and Qualcomm itself used the toolkit to compare the performance of Snapdragon X Elite SoCs with Core Ultra.
Now Intel has published on GitHub its NPU Acceleration Library, a software component that helps developers run the most compact generative models directly on the processor Meteor Lake. For example, thanks to Intel librarywhich enables the use of the NPU (Neural Processing Unit) on the latest generation chips, LLM (Large Language Model) “reduced a little more to the bone” as TinyLlama.
How NPU acceleration works and how to enable it on Intel Core Ultra Meteor Lake chips
Tony Mongkolsmai, Intel Software Architect, published on
This tweet contains the code used by the Intel software engineer to activate the open source library in turn capable of moving part of the workload on the NPU integrated. As you can see in this postil Task Manager Windows reports activity on the NPU, confirming that the use of the Intel library is fully effective.
At the moment, the Python code and the open library developed by Intel technicians is obviously only compatible with Meteor Lake chips. In fact, they are the only processors from the company led by Pat Gelsinger to integrate an NPU.
I chip Arrow Lake e Lunar Lake they should arrive on the market by the end of the year, significantly extending the range of compatible Intel processors. They should also insure up to three times higher performance compared to Meteor Lake in the field of AI, enabling execution of heavier LLMs on notebook and desktop systems.
Intel’s library for the NPU of its processors still lacks many features: here are what they are
According to Intel, the NPU Acceleration Library it is still far from complete. Suffice it to say that today it barely integrates the half the features initially planned. What are the most relevant aspects that are still missing?
First of all, there is no support for the so-called mixed precision inference. The NPU cannot currently simultaneously use lower precision data formats, such as float16along with higher precision data formats, such as float32to reduce computational requirements and improve performance.
Furthermore, it is missing BFloat16, a 16-bit data format ideal for AI workloads. It allows you to balance the precision required for AI calculations with storage and processing efficiency compared to traditional 32-bit formats.
Finally, Intel’s open source library is not compatible with the distribution of workload between multiple computational units, for example between NPU and GPU. The addition of this feature would imply a significant performance improvement.
NPU also useful from a security point of view according to Intel
According to Intel, the NPU is also useful for improving cybersecurity. The ability to run deep learning models opens up the real-time detection of threats, without having to wait for cloud-based processing.
Using the NPU to run AI models directly on the endpoint reduces latency, i.e. the time it takes to process the situation and respond to threats; furthermore, it helps reduce stress on available bandwidth; improves privacy by allowing users to maintain control over their data; allows you to develop creative approaches for recognizing and neutralizing threats.
Opening image credit: Intel. Image in text, source: “Protect Your Business with AI-based Security (Intel)“.