We knew that too Applehad long since entered the competition Large Language Model (LLM). The very important news, however, is that this time Apple has made available a generative model open source capable of working in locale on devices such as iPhone and iPad. Is called OpenELMacronym for Open-source Efficient Language Modelsand is the result of the research efforts carried out by Cupertino engineers in the field of artificial intelligence.
What is the OpenELM model and how does it work: work locally on iPhone and iPad
OpenELM is available in four variants: 270M (with 270 million parameters), 450M (with 450 million parameters), 1_1B (with 1.1 billion parameters), and 3B (with 3 billion parameters).
In generative models, i parameters refer to the values learned during the process training of the model, then used to generate output. Also in OpenELM, parameters represent i connection weights between neurons in different parts of the model.
Despite being smaller than models like OpenAI GPT-4Claude 3 by Anthropic and Meta Llama, OpenELM is designed to highlight low execution costs and is optimized to run on smartphones and laptops.
A characteristic key to OpenELM is the use of the technique called “layer-wise scaling“, which efficiently assigns parameters throughout the model by modifying the number of parameters in each layer of the Transformers.
The technique layer-wise scaling in detail
In Transformer-based models, such as OpenELM, the fundamental components use so-called latent parameters: these are the weights learned during model training, to calculate the representations of the tokens that make up the input.
The technique layer-wise scaling causes the dimensionality of the parameters latent is reduced in the first layers of the model, those closest to the text input, and increased as one gets closer to theoutput. This means that the first layers rely on fewer parameters than the subsequent layers, which can instead use more to capture more complex information.
Since models have a finite number of parameters, this is important allocate efficiently these parameters so that each layer of the model has an adequate number of parameters to perform its task. Dimensionality reduction in early layers saves parameters, which can be assigned to later layers where they are most needed to capture complex information.
OpenELM, a lightweight and accurate model that required a limited volume of training data
According to the Apple research team, when trained only on public datasets,OpenELM achieved higher accuracy than open models with a similar number of parameters. For example, OpenELM, with approximately 1.1 billion parameters, was 2.36% more accurate than OLMo, a model with approximately 1.2 billion parameters, despite requiring half the training data of OLMo.
The phase of Instruction Tuningwhich instructs the model to perform a specific task and trains it specifically for that task, improved OpenELM’s accuracy by approximately 1-2%.
Furthermore, according to Apple, theInstruction Tuning not only did it increase OpenELM’s ability to understand the language but it optimized performance when carrying out very specific activities. Additionally, OpenELM can apply methods fine-tuning efficient in terms of parameters such as Low-Rank Adaptation (LoRA) and Decomposed Low-Rank Adaptation.
Combining OpenELM with the MLX library (Machine Learning Accelerator) from Apple, users can run any variant of the model efficiently on iPhone and iPad.
Where to find OpenELM and how to use it
OpenELM, developed as an open source product, is hosted on the online AI platform Hugging Face. To maximize the possible usage scenarios of OpenELM, the Apple research team provides not only the model weights but also the training code, the logs generated in the first phase, multiple checkpoints, the public dataset used to train the model, the parameters details and the code to carry out the conversion in a bookshelf MLX.
Apple has not yet revealed how it intends to use OpenELM in the future but it is highly likely that it could be integrated with iOS and iPadOS in order to better manage all tasks related to text management, such as composing emails, on Apple devices.
Opening image credit: Copilot Designer.