Pixtral 12B: Mistral AI's First Multimodal AI Model

Mistral AI launched its first multimodal AI model, Pixtral 12B, on September 17, 2024. This model, with around 12 billion parameters, can process both text and images, making it a major step forward in AI technology.

Key Features of Pixtral 12B

1. Multimodal Architecture
   Pixtral 12B is built to handle text and images at the same time. It was trained with mixed data, making it great for tasks that need both, like understanding charts or giving detailed image descriptions.

2. Vision Encoder and Decoder
   The model includes a 400M parameter vision encoder and a 12B parameter multimodal decoder. It works with images of different sizes and shapes and can handle multiple images with a context window of 128K tokens.

3. Benchmark Performance
   Pixtral 12B scores 52.5% on the MMMU reasoning benchmark, beating many larger models in multimodal tasks. It also shows a 20% improvement in instruction following compared to its nearest open-source competitors.

4. Applications

   • Object Recognition & Image Captioning: Pixtral automatically describes images.
   • Data Visualization: It reads complex charts, useful for jobs like financial analysis.
   • Multilingual: It works in several languages, opening doors to global markets.

5. Open-Source
   Licensed under Apache 2.0, Pixtral 12B is open-source, allowing anyone to download, change, and use it for commercial projects.

6. Contextual Understanding
   With a 128K token context window, Pixtral is strong in handling long documents and complex layouts, perfect for OCR and document analysis.

Comparing Pixtral 12B to Other Models

Pixtral 12B stands out among open-source and closed models with its strong performance across benchmarks:

• Multimodal Reasoning: It hits 52.5% on the MMMU benchmark, outperforming larger models and open-source alternatives like Qwen2-VL 7B and LLaVa-OneVision 7B.
• Text-Only Tasks: It keeps top-tier performance on text tasks, even while excelling at multimodal ones.
• Task-Specific Comparisons: On benchmarks like ArxivQA and Flickr30K, it performs close to GPT-4v, showing strong reasoning and image captioning.

According to some independednt benchmarks, Pixtral 12B:

• is cheaper compared to average with a price of $0.15 per 1M Tokens (blended 3:1).
• is slower compared to average, with a output speed of 79.9 tokens per second.
• has a lower latency compared to average, taking 0.59s to receive the first token (TTFT).
• has a smaller context windows than average, with a context window of 130k tokens.

Pixtral Installation

You can try Pixtral easily and freely via Le Chat

Pixtral 12B is fully open-source, available under the Apache 2.0 license. To install and use it, follow these steps:

1. Download the Model:
   Visit its GitHub or Hugging Face page for download options.

2. Set Up:
   Ensure Python is installed along with the required libraries like TensorFlow or PyTorch.

3. Load the Model (python):

   from transformers import PixtralModel
model = PixtralModel.from_pretrained('path_to_downloaded_model')

4. Fine-Tuning (optional):
   Follow the provided instructions to adjust it for specific tasks.

5. Run Inference:
   You can now use Pixtral 12B for text and image-based tasks.

Or follow the guidelines on the official release page.

Reddit Buzz Around Pixtral 12B: Mixed Early Reactions

Recent discussions on Reddit, especially in the /r/LocalLLaMA subreddit, reveal mixed feelings about Mistral's new Pixtral 12B model. Here's a breakdown of the most popular posts and opposing views:

Positive Feedback

1. Multimodal Design
   Many users are excited about Pixtral's ability to process both text and images at the same time. One user pointed out how well it follows instructions while keeping its strong performance on text-only tasks.

2. Innovative Structure
   The model's design—featuring a vision encoder with 400 million parameters and a multimodal decoder with 12 billion parameters—has impressed users, with some highlighting its potential in different fields.

3. Real-World Applications
   Several commenters believe Pixtral could spark big improvements in areas that need both image and text processing. It's seen as a game-changer for developers working on advanced AI solutions.

Other Opinions

1. Concerns About Long-Term Support
   Some users are worried about how long the Llama.cpp framework, which Pixtral is based on, will be supported. One comment expressed doubt that it would keep getting updates, leading to concerns about the future of the model’s ecosystem.

2. Abandoned Vision Models
   A few users noted that other vision models, like InternVL2 and Qwen2-VL, aren’t seeing much development anymore. This has raised fears that Pixtral’s vision model support could fall behind, limiting its usefulness in some areas.

3. Implementation Issues
   While Pixtral works with frameworks like Transformers, some users found it hard to use without significant coding skills. The complexity of the setup sparked discussions about whether such models are too difficult for average users without strong technical backgrounds.

Overall, while many users are excited about Pixtral’s potential, worries about long-term support and ease of use have sparked debate on Reddit.