#ai21

New Post has been published on https://thedigitalinsider.com/cerebras-inference-and-the-challenges-of-challenging-nvidias-dominance/

Cerebras Inference and the Challenges of Challenging NVIDIA’s Dominance

Why does NVIDIA remains virtually unchallenged in the AI chip market?

Created Using Ideogram

Next Week in The Sequence:

Edge 427: Our series about state space models(SSM) continues with a review of AI21’s Jamba, a model that combines transformers and SSMs. We discuss Jamba’s original research paper and the DeepEval framework.

Edge 428: We dive into PromptPoet, Character.ai’s framework for prompt optimization.

You can subscribe to The Sequence below:

TheSequence is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

📝 Editorial: Cerebras Inference and the Challenges of Challenging NVIDIA’s Dominance

AI hardware is experiencing an innovation renaissance, with well-funded startups emerging everywhere. Yet, NVIDIA remains virtually unchallenged, holding close to a 90% share of the AI chip market. Why is that?

We’ve all heard explanations about the advantages of NVIDIA’s software stack for acceleration compared to platforms like AMD’s, which seems like a lazy explanation for why NVIDIA is out-innovating its competitors. A simple theory that I’ve discussed with several scientists and engineers who pretrain large foundation models is that NVIDIA is the only platform receiving regular feedback about the performance of chips during pretraining runs with tens of thousands of GPUs. It turns out that at that scale, many challenges arise that are nearly impossible to simulate on a smaller scale. I will elaborate more on that theory in a future post, but the main point is that there is a very high barrier to entry when it comes to challenging NVIDIA chips for pretraining. The only viable candidate seems to be Google TPUs, which have definitely been tested at massive scale.

If pretraining is out of the equation, the obvious area to explore is inference. Here, we have a completely different playing field, where performance optimizations can be applied at a smaller scale, making it more conducive to startup disruptions.

One of the viable challengers to NVIDIA’s dominance in AI inference is Cerebras. Just last week, the well-funded startup unveiled Cerebras Inference, a solution capable of delivering Llama 3.1 8B at 450 tokens per second for Llama 3.1 70B. This is approximately 20x faster than NVIDIA GPUs and about 2.4x faster than Groq. The magic behind Cerebras’ performance is its AI chip design, which allows the entire model to be stored on-chip, eliminating the need for GPU communication.

Cerebras Inference looks impressive from top to bottom and clearly showcases the massive potential for innovation in AI inference. Competing with NVIDIA will require more than just faster chips, but Cerebras appears to be a legitimate challenger.

🔎 ML Research

The Mamba in the Llama

Researchers from Princeton University, Together AI, Cornell University and other academic institutions published a paper proposing a technique to distill and accelerate transformer-SSM models. The method distills transformers into RNN-equivalents with a quarter of the hidden layers —> Read more.

Diffusion Models as Real Time Game Engines

Google Research published a paper presenting GameNGen, a game engine powered by diffusion models and interactions with real environments over long trajectories. GameNGen can simulate a DOOM game in over 20 frames in a single TPU —> Read more.

LLMs that Learn from Mistakes

Researchers from Meta FAIR and Carnegie Mellon University published a paper outlining a technique to include error-correction data directly in the pretraining stage in order to improve reasoning capabilities. The resulting model outperform alternatives trained in error-free data —> Read more.

Table Augmented Generation

In a new paper, Researchers from UC Berkeley proposed table augmented generation, a method that addresses some of the limitations of text-to-SQL and RAG and answer questions in relational databases. The TAG model captures a very complete sets of interaction between an LLM and a database —> Read more.

DisTrO

Nous Research published a paper introducing DisTrO, an architecture that reduces inter-GPU communication by up to 5 orders of magnitude. DisTrO is an important method for low latency training of large neural networks —> Read more.

Brain Inspired Design

Microsoft Research published a summary of their recent research in three projects that simualte the brain learns. One project simulates the brain computes information, another enhances accuracy and efficiency and the third one shows improves proficiency in language processing and pattern recognition —> Read more.

🤖 AI Tech Releases

Qwen2-VL

Alibaba Research released a new version of Qwen2-VL their marquee vision language model —> Read more.

Cerebras Inference

Cerebras released an impressive inference solution that can generate 1800 tokens per second in Llama 3.1 models —> Read more.

NVIDIA NIM Blueprints

NVIDIA released NIM Blueprints, a series of templates to help enterprises get started with generative AI applications —> Read more.

Gemini Models

Google DeepMind released a new series of experimental models —> Read more.

Command R

Cohere released a new version of Command R with improvements in coding, math, reasoning and latency —> Read more.

🛠 Real World AI

Recommendations at Netflix

Netflix discusses some of the AI techniques to enhances long term satisfaction in their content recommendations —> Read more.

📡AI Radar

AI coding platform Magic raised an impressive $320 million round.

Another AI coding platform Codium raised $150 million in a Series C.

Amazon hired the founders of Robotis startup Covariant.

Midjourney announced their intentions of going into hardware.

OpenAI is closing its tender offer at $100 billion valuation.

OpenAI and Anthropic agreed the U.S AI Institute to review and evaluate their models.

AI customer service platform Bland AI raised $16 million in new funding.

Inflection AI is porting its Pi chatbot to enterprise workflows.

Antrophic made its Artifacts solution available in IOS and Android.

Asset manager Magnetar Capital is launching its first venture fund focused on generative AI.

Atlassian acquired AI meeting bot company Rewatch.

New Post has been published on https://thedigitalinsider.com/jamba-ai21-labs-new-hybrid-transformer-mamba-language-model/

Jamba: AI21 Labs’ New Hybrid Transformer-Mamba Language Model

Language models has witnessed rapid advancements, with Transformer-based architectures leading the charge in natural language processing. However, as models scale, the challenges of handling long contexts, memory efficiency, and throughput have become more pronounced.

AI21 Labs has introduced a new solution with Jamba, a state-of-the-art large language model (LLM) that combines the strengths of both Transformer and Mamba architectures in a hybrid framework. This article delves into the details of Jamba, exploring its architecture, performance, and potential applications.

Overview of Jamba

Jamba is a hybrid large language model developed by AI21 Labs, leveraging a combination of Transformer layers and Mamba layers, integrated with a Mixture-of-Experts (MoE) module. This architecture allows Jamba to balance memory usage, throughput, and performance, making it a powerful tool for a wide range of NLP tasks. The model is designed to fit within a single 80GB GPU, offering high throughput and a small memory footprint while maintaining state-of-the-art performance on various benchmarks.

The Architecture of Jamba

Jamba’s architecture is the cornerstone of its capabilities. It is built on a novel hybrid design that interleaves Transformer layers with Mamba layers, incorporating MoE modules to enhance the model’s capacity without significantly increasing computational demands.

1. Transformer Layers

The Transformer architecture has become the standard for modern LLMs due to its ability to handle parallel processing efficiently and capture long-range dependencies in text. However, its performance is often limited by high memory and compute requirements, particularly when processing long contexts. Jamba addresses these limitations by integrating Mamba layers, which we will explore next.

2. Mamba Layers

Mamba is a recent state-space model (SSM) designed to handle long-distance relationships in sequences more efficiently than traditional RNNs or even Transformers. Mamba layers are particularly effective at reducing the memory footprint associated with storing key-value (KV) caches in Transformers. By interleaving Mamba layers with Transformer layers, Jamba reduces the overall memory usage while maintaining high performance, especially in tasks requiring long context handling.

3. Mixture-of-Experts (MoE) Modules

The MoE module in Jamba introduces a flexible approach to scaling model capacity. MoE allows the model to increase the number of available parameters without proportionally increasing the active parameters during inference. In Jamba, MoE is applied to some of the MLP layers, with the router mechanism selecting the top experts to activate for each token. This selective activation enables Jamba to maintain high efficiency while handling complex tasks.

The below image demonstrates the functionality of an induction head in a hybrid Attention-Mamba model, a key feature of Jamba. In this example, the attention head is responsible for predicting labels such as “Positive” or “Negative” in response to sentiment analysis tasks. The highlighted words illustrate how the model’s attention is strongly focused on label tokens from the few-shot examples, particularly at the critical moment before predicting the final label. This attention mechanism plays a crucial role in the model’s ability to perform in-context learning, where the model must infer the appropriate label based on the given context and few-shot examples.

The performance improvements offered by integrating Mixture-of-Experts (MoE) with the Attention-Mamba hybrid architecture are highlighted in Table. By using MoE, Jamba increases its capacity without proportionally increasing computational costs. This is particularly evident in the significant boost in performance across various benchmarks such as HellaSwag, WinoGrande, and Natural Questions (NQ). The model with MoE not only achieves higher accuracy (e.g., 66.0% on WinoGrande compared to 62.5% without MoE) but also demonstrates improved log-probabilities across different domains (e.g., -0.534 on C4).

Key Architectural Features

Layer Composition: Jamba’s architecture consists of blocks that combine Mamba and Transformer layers in a specific ratio (e.g., 1:7, meaning one Transformer layer for every seven Mamba layers). This ratio is tuned for optimal performance and efficiency.

MoE Integration: The MoE layers are applied every few layers, with 16 experts available and the top-2 experts activated per token. This configuration allows Jamba to scale effectively while managing the trade-offs between memory usage and computational efficiency.

Normalization and Stability: To ensure stability during training, Jamba incorporates RMSNorm in the Mamba layers, which helps mitigate issues like large activation spikes that can occur at scale.

Jamba’s Performance and Benchmarking

Jamba has been rigorously tested against a wide range of benchmarks, demonstrating competitive performance across the board. The following sections highlight some of the key benchmarks where Jamba has excelled, showcasing its strengths in both general NLP tasks and long-context scenarios.

1. Common NLP Benchmarks

Jamba has been evaluated on several academic benchmarks, including:

HellaSwag (10-shot): A common sense reasoning task where Jamba achieved a performance score of 87.1%, surpassing many competing models.

WinoGrande (5-shot): Another reasoning task where Jamba scored 82.5%, again showcasing its ability to handle complex linguistic reasoning.

ARC-Challenge (25-shot): Jamba demonstrated strong performance with a score of 64.4%, reflecting its ability to manage challenging multiple-choice questions.

In aggregate benchmarks like MMLU (5-shot), Jamba achieved a score of 67.4%, indicating its robustness across diverse tasks.

2. Long-Context Evaluations

One of Jamba’s standout features is its ability to handle extremely long contexts. The model supports a context length of up to 256K tokens, the longest among publicly available models. This capability was tested using the Needle-in-a-Haystack benchmark, where Jamba showed exceptional retrieval accuracy across varying context lengths, including up to 256K tokens.

3. Throughput and Efficiency

Jamba’s hybrid architecture significantly improves throughput, particularly with long sequences.

In tests comparing throughput (tokens per second) across different models, Jamba consistently outperformed its peers, especially in scenarios involving large batch sizes and long contexts. For instance, with a context of 128K tokens, Jamba achieved 3x the throughput of Mixtral, a comparable model.

Using Jamba: Python

For developers and researchers eager to experiment with Jamba, AI21 Labs has provided the model on platforms like Hugging Face, making it accessible for a wide range of applications. The following code snippet demonstrates how to load and generate text using Jamba:

from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1") tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1") input_ids = tokenizer("In the recent Super Bowl LVIII,", return_tensors='pt').to(model.device)["input_ids"] outputs = model.generate(input_ids, max_new_tokens=216) print(tokenizer.batch_decode(outputs))

This simple script loads the Jamba model and tokenizer, generates text based on a given input prompt, and prints the generated output.

Fine-Tuning Jamba

Jamba is designed as a base model, meaning it can be fine-tuned for specific tasks or applications. Fine-tuning allows users to adapt the model to niche domains, improving performance on specialized tasks. The following example shows how to fine-tune Jamba using the PEFT library:

import torch from datasets import load_dataset from trl import SFTTrainer, SFTConfig from peft import LoraConfig from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1") model = AutoModelForCausalLM.from_pretrained( "ai21labs/Jamba-v0.1", device_map='auto', torch_dtype=torch.bfloat16) lora_config = LoraConfig(r=8, target_modules=[ "embed_tokens","x_proj", "in_proj", "out_proj", # mamba "gate_proj", "up_proj", "down_proj", # mlp "q_proj", "k_proj", "v_proj" # attention], task_type="CAUSAL_LM", bias="none") dataset = load_dataset("Abirate/english_quotes", split="train") training_args = SFTConfig(output_dir="./results", num_train_epochs=2, per_device_train_batch_size=4, logging_dir='./logs', logging_steps=10, learning_rate=1e-5, dataset_text_field="quote") trainer = SFTTrainer(model=model, tokenizer=tokenizer, args=training_args, peft_config=lora_config, train_dataset=dataset, ) trainer.train()

This code snippet fine-tunes Jamba on a dataset of English quotes, adjusting the model’s parameters to better fit the specific task of text generation in a specialized domain.

Deployment and Integration

AI21 Labs has made the Jamba family widely accessible through various platforms and deployment options:

Cloud Platforms:

Available on major cloud providers including Google Cloud Vertex AI, Microsoft Azure, and NVIDIA NIM.

Coming soon to Amazon Bedrock, Databricks Marketplace, and Snowflake Cortex.

AI Development Frameworks:

Integration with popular frameworks like LangChain and LlamaIndex (upcoming).

AI21 Studio:

Direct access through AI21’s own development platform.

Hugging Face:

Models available for download and experimentation.

On-Premises Deployment:

Options for private, on-site deployment for organizations with specific security or compliance needs.

Custom Solutions:

AI21 offers tailored model customization and fine-tuning services for enterprise clients.

Developer-Friendly Features

Jamba models come with several built-in capabilities that make them particularly appealing for developers:

Function Calling: Easily integrate external tools and APIs into your AI workflows.

Structured JSON Output: Generate clean, parseable data structures directly from natural language inputs.

Document Object Digestion: Efficiently process and understand complex document structures.

RAG Optimizations: Built-in features to enhance retrieval-augmented generation pipelines.

These features, combined with the model’s long context window and efficient processing, make Jamba a versatile tool for a wide range of development scenarios.

Ethical Considerations and Responsible AI

While the capabilities of Jamba are impressive, it’s crucial to approach its use with a responsible AI mindset. AI21 Labs emphasizes several important points:

Base Model Nature: Jamba 1.5 models are pretrained base models without specific alignment or instruction tuning.

Lack of Built-in Safeguards: The models do not have inherent moderation mechanisms.

Careful Deployment: Additional adaptation and safeguards should be implemented before using Jamba in production environments or with end users.

Data Privacy: When using cloud-based deployments, be mindful of data handling and compliance requirements.

Bias Awareness: Like all large language models, Jamba may reflect biases present in its training data. Users should be aware of this and implement appropriate mitigations.

By keeping these factors in mind, developers and organizations can leverage Jamba’s capabilities responsibly and ethically.

A New Chapter in AI Development?

The introduction of the Jamba family by AI21 Labs marks a significant milestone in the evolution of large language models. By combining the strengths of transformers and state space models, integrating mixture of experts techniques, and pushing the boundaries of context length and processing speed, Jamba opens up new possibilities for AI applications across industries.

Stuck in GPT-3’s waitlist? Try out the AI21 Jurassic-1

The latest contender in the NLP arms race is AI21, with its 178 billion parameter model, Jurassic-1. And you’re free to test it out.Read More