MediaPipe Web: Bringing cross-platform AI tech to the browser

MediaPipe, a cross-platform, open-source framework, efficiently builds and runs AI pipelines across various platforms, including web browsers. It leverages C++ calculators and graph files to connect AI components, with MediaPipe Web bridging C++ and TypeScript for browser compatibility. This approach enables diverse applications, from Google Meet effects to virtual try-on experiences and browser-based large language models, demonstrating the potential for high-performance AI directly in web applications.

MediaPipe Overview

• 00:00:22 MediaPipe is an open-source, cross-platform framework for building and running AI pipelines. It facilitates code sharing between pipelines, platforms, and products, making it a scalable choice for various applications. MediaPipe's cross-platform nature enables the same pipelines to be used on desktops, laptops, mobile devices, and web browsers.

MediaPipe Web Architecture

• 00:03:03 MediaPipe Web uses Emscripten to transpile C++ code into WebAssembly. This allows MediaPipe pipelines developed in C++ to be used in web applications written in TypeScript. This approach provides a bridge between C++ libraries and the web browser environment, making it possible to run sophisticated AI models in the browser.

MediaPipe Web Applications

• 00:03:22 MediaPipe Web has enabled a range of applications, including background blurring and replacement in Google Meet. Virtual try-on experiences in mobile browsers for beauty products are another example. These applications demonstrate how MediaPipe Web can be used to bring advanced AI functionality to users through web browsers.

MediaPipe Tasks & LLM Inference

• 00:05:18 MediaPipe Tasks provide low-code AI solutions to common machine learning problems, including image segmentation and object detection. MediaPipe also offers an experimental LLM Inference API for running large language models on-device in browsers. This API uses WebGPU for efficient inference and supports several popular LLM architectures, enabling faster and more customized AI applications within browsers.

Future of MediaPipe Web AI

• 00:10:17 The future of MediaPipe Web includes enhancing LLMs with features like multi-modality and retrieval augmented generation (RAG), improving the quality and customization of AI models. An AI Edge Torch system is being developed to convert PyTorch models to a format compatible with MediaPipe inference engines, expanding the range of models that can be utilized in web applications. This development underscores the growing potential for diverse and powerful AI experiences within web browsers.