Microsoft Phi-4-Multimodal & Phi-4 Mini: Advanced AI Models

The Phi family’s next generation is fostering creativity. Also are pleased to present the newest models in Microsoft’s Phi family of small language models (SLMs), Phi-4-multimodal and Phi-4-mini. These models are made to give developers access to cutting-edge AI capabilities. The capacity of Phi-4-multimodal to interpret text, speech, and vision all at once creates new opportunities for developing creative and contextually aware applications.

In contrast, Phi-4-mini performs exceptionally well in text-based tasks, offering great accuracy and scalability in a small package. Now accessible through the NVIDIA API Catalogue, Azure AI Foundry, and HuggingFace, developers can easily experiment and innovate by exploring the full potential of Phi-4-multimodal.

What is Phi-4-multimodal?

As the initial multimodal language model, Phi-4-multimodal represents a new turning point in Microsoft’s AI development. Continuous improvement is at the heart of innovation, and it begins with listening to the needs of customers. It created Phi-4-multimodal, a 5.6B parameter model that combines text, speech, and vision processing into a single, cohesive architecture, in direct response to user feedback.

This paradigm makes it possible for devices to comprehend and reason across many input modalities at once by utilising sophisticated cross-modal learning techniques, which facilitate more natural and context-aware interactions. It provides extremely effective, low-latency inference while optimising for on-device execution and minimising computing cost, whether it is used for text processing, image analysis, or spoken language interpretation.

Natively built for multimodal experiences

Language, voice, and vision are all processed concurrently in the same representation space by Phi-4-multimodal, a single model with mixture-of-LoRAs. This eliminates the need for intricate pipelines or distinct models for various modalities and creates a single, cohesive model that can handle text, audio, and visual inputs.

The novel architecture upon which the Phi-4-multimodal is based improves scalability and efficiency. It merges verbal thinking with multimodal inputs, promotes multilingual capabilities, and uses a broader vocabulary for better processing. All of this is accomplished in a strong, small, and incredibly effective model that can be installed on smartphones and edge computing platforms.

With improved performance in a compact package, this model is a step forward for the Phi family of devices. Phi-4-multimodal offers a high-capability solution that is effective and adaptable, regardless of your preference for cutting-edge AI capabilities on edge systems or mobile devices.

Unlocking new capabilities

Phi-4-multimodal’s expanded range of capabilities and adaptability creates exciting new opportunities for companies, sectors, and app developers seeking to creatively use AI. Multimodal AI‘s future is arrived and is prepared to revolutionize your applications.

Phi-4-multimodal can process audio and visual information simultaneously. When synthetic speech is used as the input query for vision content on chart/table understanding and document reasoning tasks, the model quality is displayed in the accompanying table. On a number of benchmarks, Phi-4-multimodal performs significantly better than other state-of-the-art omni models that can accept audio and visual signals as input.

Phi-4-multimodal has emerged as a top open model in several domains after showcasing impressive abilities in speech-related activities. It performs better in automatic speech recognition (ASR) and speech translation (ST) than specialised models such as WhisperV3 and SeamlessM4T-v2-Large. With a remarkable word error rate of 6.14%, the model has surpassed the previous highest performance of 6.5% as of February 2025 to take the top spot on the Huggingface OpenASR leaderboard.

It is also one of the few open models that can effectively use speech summarization and attain performance levels that are on par with the GPT-4o model. On spoken question answering (QA) tasks, the model performs worse than close models like Gemini-2.0-Flash and GPT-4o-realtime-preview because of its lower size, which reduces its ability to retain factual QA knowledge. In the upcoming generations, efforts are being made to enhance this capacity.

With just 5.6B of parameters, Phi-4-multimodal shows impressive vision capabilities on a number of benchmarks, most notably obtaining high performance on scientific and mathematical reasoning. The model matches or outperforms close models like Gemini-2-Flash-lite-preview/Claude-3.5-Sonnet in general multimodal capabilities, including document and chart understanding, optical character recognition (OCR), and visual science reasoning, despite its smaller size.

What is Phi-4-mini?

Designed for speed and efficiency, Phi-4-mini is a dense, decoder-only transformer with 3.8B parameters that has 200,000 vocabulary, shared input-output embeddings, and grouped-query attention. In text-based tasks including reasoning, math, coding, instruction-following, and function-calling, it consistently outperforms larger models despite its small size. It is a potent solution for sophisticated AI applications since it offers great accuracy and scalability while supporting sequences up to 128,000 tokens.

Despite their limited capacity, compact language models like Phi-4-mini may access external information and functionality to strong capabilities like function calling, instruction following, lengthy context, and reasoning. Function calling enables the model to easily interface with structured programming interfaces via a standardized protocol. Phi-4-Mini can process a user’s request by reasoning through it, finding and calling pertinent functions with the right arguments, receiving the function outputs, and incorporating them into its reply.

By connecting the model to external tools, application program interfaces (APIs), and data sources via well specified function interfaces, an extensible agentic-based system is created, allowing the model’s capabilities to be expanded. The sample that follows uses Phi-4-mini to imitate a smart home control agent.

Customization and cross-platform

Phi-4-mini and Phi-4-multimodal models are suitable for usage in compute-constrained inference situations because of their reduced sizes. When further optimized with ONNX Runtime for cross-platform availability, these models can be utilized on-device. They have significantly better latency and are less expensive due to their decreased computing requirements.

Large text documents, web pages, code, and more may be seen and reasoned over to the extended context window. Strong reasoning and logic skills make Phi-4-mini and multimodal an excellent choice for analytical jobs. Additionally, their compact size facilitates and lowers the cost of customization or fine-tuning. Examples of finetuning scenarios using Phi-4-multimodal are displayed in the table below.

How can these models be used in action?

These models are perfect for edge case scenarios and contexts with limited computing power because they are made to tackle complicated jobs effectively. The usage of Phi are only growing because of the increased capabilities that Phi-4-multimodal and Phi-4-mini offer. Phi models are being utilized to investigate a range of use cases in different industries and are being integrated into AI ecosystems.

Language models are strong reasoning engines, and by including small language models like Phi into Windows, it can preserve valuable computing resources and pave the way for a future in which all of your programs and experiences will be continuously intelligent. Building on the capabilities of Phi-4-multimodal, Copilot+ PCs will provide the strength of Microsoft’s cutting-edge SLMs without using as much energy. This connection, which will become a normal feature of the developer platform, will improve creativity, productivity, and educational experiences.

• Integrated straight into your smart device: By incorporating Phi-4-multimodal into smartphones, phone manufacturers may be able to provide them the ability to recognize and comprehend voice commands, process and comprehend text, and recognize photos with ease. Advanced capabilities like improved photo and video analysis, real-time language translation, and intelligent personal assistants who comprehend and react to complex requests could be advantageous to users. By offering strong AI capabilities right on the device, this would improve the user experience while guaranteeing low latency and high efficiency.

• While driving: Consider a car manufacturer incorporating Phi-4-multimodal into their in-car helper programs. The model might make it possible for cars to comprehend and react to verbal orders, identify gestures made by the driver, and interpret visual information from cameras. For example, by using facial recognition to identify tiredness and sending out real-time notifications, it could improve driver safety. In addition, it might deliver contextual information, interpret traffic signs, and provide smooth navigation support, making driving safer and easier both while linked to the cloud and when unconnected.

• Multilingual financial services: Consider a financial services organisation utilising Phi-4-mini to produce comprehensive reports, automate intricate financial computations, and translate financial documents into many languages. For example, the model can help analysts by carrying out complex mathematical calculations needed for financial forecasts, portfolio management, and risk assessments. It can also translate regulatory paperwork, financial statements, and client interactions into multiple languages, which could enhance client connections internationally.

Microsoft’s dedication to safety and security

For both classic machine learning and generative AI applications, Azure AI Foundry offers users a comprehensive suite of tools to help organizations assess, reduce, and manage AI risks throughout the AI development lifecycle. Using both built-in and custom metrics to guide mitigations, Azure AI evaluations in AI Foundry allow developers to repeatedly evaluate the safety and quality of models and apps.

Both internal and external security specialists used techniques developed by the Microsoft AI Red Team (AIRT) to test the security and safety of both models. These techniques, which were improved upon from earlier Phi models, take into account native speakers of all supported languages as well as global viewpoints. Through multilingual questioning, they examine current trends in topics like violence, fairness, national security, and cybersecurity.

Red teamers carried out single-turn and multi-turn attacks using manual probing and AIRT’s open-source Python Risk Identification Toolkit (PyRIT). AIRT operated separately from the development teams and provided the model team with ongoing insights. This method ensured the supply of high-quality capabilities by evaluating the new AI security and safety landscape brought about by the most recent Phi models.