multimodal

为纯文本模型"看图“设计更好的视觉工具箱和技能，支持多图理解，图片问答，前端UI还原、GUI 自动化等，并可选无缝接入多个主流agent，直接识别粘贴图片｜ A vision toolkit and skill designed for text-only llms — image Q&A, long-screenshot OCR, frontend UI restoration, and GUI automation, with optional seamless integration for Codex, Claude Code, Pi, Oh My Pi, and OpenCode

The Self-Coding System for Your App — Alan AI SDK for Cordova

SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.

A foundation model that generates synchronized video and audio in a single model

Eyes for text-only DeepSeek Harness agents: built-in free vision chain (no key) + pixel-level vision tools (Q&A, grounding, crop, pixel diff, colors, OCR, SVG trace, cutout, screenshots). One-command install, no Python, image turns work like ordinary tool-calling turns.

Codebase for Aria - an Open Multimodal Native MoE

🩺 首个会看胸部X光片的中文多模态医学大模型 | The first Chinese Medical Multimodal Model that Chest Radiographs Summarization.

[ICLR'24 spotlight] Chinese and English Multimodal Large Model Series (Chat and Paint) | 基于CPM基础模型的中英双语多模态大模型系列

A general representation model across vision, audio, language modalities. Paper: ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

[ECCV 2024 Best Paper Candidate & TPAMI 2025] PointLLM: Empowering Large Language Models to Understand Point Clouds

An official implementation for "CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval"

Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey

TongFlow — Multimodal GenAI Studio

Resource, examples & tutorials for multimodal AI, RAG and agents using vector search and LLMs

Multimodal RL training framework for diffusion & omni models

RAG Time: A 5-week Learning Journey to Mastering RAG

About This repository is a curated collection of the most exciting and influential CVPR 2025 papers. 🔥 [Paper + Code + Demo]

NEO Series: Native Vision-Language Models from First Principles

A unified multimodal language model based on discrete sequence modeling

⚡ Self-hostable YesCaptcha-compatible captcha solver built with FastAPI, Playwright, and OpenAI-compatible multimodal models.

A simple, unified multimodal models training engine. Lean, flexible, and built for hacking at scale.

[TMLR 2025🔥] A survey for the autoregressive models in vision.

library supporting NLP and CV research on scientific papers

Train Models Contrastively in Pytorch

本地监控 + AI 视觉 — LAN-based smartphone-powered AI monitoring framework with structured event output for data acquisition and analysis.

Paper list about multimodal and large language models, only used to record papers I read in the daily arxiv for personal needs.

[ECCV 2024] InstructIR: High-Quality Image Restoration Following Human Instructions https://huggingface.co/spaces/marcosv/InstructIR

Paddle Multimodal Integration and eXploration, supporting mainstream multi-modal tasks, including end-to-end large-scale multi-modal pretrain models and diffusion model toolbox. Equipped with high performance and flexibility.

A Toolbox for MultiModal Recommendation. Integrating 10+ Models...

This repo contains the code for "VLM2Vec / MMEB" [ICLR 2025], "VLM2Vec-V2 / MMEB-V2" [TMLR 2026], and "MMEB-V3" [COLM 2026]