multimodal

✨✨Latest Papers and Benchmarks in Reasoning with Foundation Models

An open-source model for understanding speech, environmental sounds, and music through captioning, question answering, and reasoning

Official implementation of SEED-LLaMA (ICLR 2024).

Project Page For "Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement"

👁️ + 💬 + 🎧 = 🤖 Curated list of top foundation and multimodal models! [Paper + Code + Examples + Tutorials]

Official implementation for "Blended Latent Diffusion" [SIGGRAPH 2023]

Second Brain is an agentic framework that acts as an operating system, using local file intelligence, workflow automation, and LLMs to complete tasks and communicate over multiple modalities and messaging platforms.

This repository provides programs to build Retrieval Augmented Generation (RAG) code for Generative AI with LlamaIndex, Deep Lake, and Pinecone leveraging the power of OpenAI and Hugging Face models for generation and evaluation.

Explore the Multimodal “Aha Moment” on 2B Model

Hunyuan3D-Omni: A Unified Framework for Controllable Generation of 3D Assets

[ECCV 2024] Tokenize Anything via Prompting

An open-weight 11B model series for long-form and real-time video understanding

This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI"

An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale

Official implementation for "Blended Diffusion for Text-driven Editing of Natural Images" [CVPR 2022]

[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization

Create browser automation as if you were teaching a human using GPT-4 Vision.

RAI is a vendor agnostic agentic framework for Physical AI robotics, utilizing ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.

multimodal routing, geocoding, and map tiles

LLaVA-Mini is a unified large multimodal model (LMM) that can support the understanding of images, high-resolution images, and videos in an efficient manner.

The Self-Coding System for Your App — Alan AI SDK for React Native

An MCP Multimodal AI Agent with eyes and ears!

Platform for Situated Intelligence

CLIP inference in plain C/C++ with no extra dependencies

Flame is an open-source multimodal AI system designed to translate UI design mockups into high-quality React code. It leverages vision-language modeling, automated data synthesis, and structured training workflows to bridge the gap between design and front-end development.

A hub for various industry-specific schemas to be used with VLMs.

Awesome Multimodal Modeling [Covers MLLM, UMM, and NMM]

Official code of "EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model"