Midasheng Gen
БесплатноНе проверенGenerates coherent mixed audio scenes (speech, music, SFX, ambience) from text using an LLM-driven model, with speech in 9 languages and emotion control.
Описание
Generates coherent mixed audio scenes (speech, music, SFX, ambience) from text using an LLM-driven model, with speech in 9 languages and emotion control.
README
midasheng-gen-mcp
Type a scene. Get the whole soundscape.
An MCP server and dark-theme webapp that generate coherent 16 kHz mixed audio
scenes - speech, music, sound effects and ambience - in one pass from text,
powered by MiDashengLM-Gen
(Xiaomi Research). Runs entirely on your GPU.
Quick install · Example prompts · Documentation
One call in Claude, Cursor or opencode - "a comedy club scene with a punchline, crowd laughter and a jazz sting" - and you get a single 16 kHz WAV with all of it. Speech lands close to dedicated TTS quality in 9 languages with emotion control, and the whole ~6 GB model stays on your GPU until you unload it.
What this wraps
MiDashengLM-Gen - the first end-to-end trained general text-to-audio model (per the paper): an LLM drives per-token flow matching to generate variable-length audio scenes with speech intelligibility approaching dedicated TTS (Seed-TTS WER 12.15% -> 2.79% vs 1.24% for dedicated TTS). Supports 9 languages and emotion control. Checkpoint auto-downloads from Hugging Face (~6 GB). Model weights are never bundled - see docs/WRAPPEE.md.
Features
How it runs: a local FastMCP 3.4 server (stdio for Claude Desktop, HTTP
/mcp for Cursor/webapp) with a React dashboard. The model loads lazily on
first generate and stays on the GPU until unloaded.
| Direction | Artifacts | Notes |
|---|---|---|
| Hands-in | structured caption (caption/asr/speech/sfx/music/env views), guidance + seed params | Via MCP tool, REST, or webapp |
| Hands-out | 16 kHz mono WAV scenes, indexed in SQLite, browsable/exportable | audio_scene(operation="export") or webapp |
- Generate whole soundscapes in one call - crowd laughter + jazz sting + comedy speech, or rain + thunder + forest ambience
- Speech intelligibility near dedicated TTS quality, 9 languages, emotion
control through the
speechview - Scene library with pagination, audio playback, and export in the webapp
- Async job API for long generations; Prefab UI cards in chat
- Variable-length output via the learned stop head - no fixed-duration cuts
Quick Install
The fastest path is the .mcpb bundle for Claude Desktop (see INSTALL.md for all options):
- Download
midasheng-gen-mcp-v0.1.0.mcpbfrom Releases - Drag it onto Claude Desktop
- First use will download the ~6 GB checkpoint automatically
Or clone and double-click start.bat for the full stack (backend + webapp).
Example Prompts
- "Generate a comedy club scene: a punchline, crowd laughter, and a jazz
band sting" ->
audio_scene(operation="generate", caption="A comedian delivering a punchline followed by uproarious crowd laughter", asr="And that is why I never buy cheap luggage anymore!", speech="expressive comedic male voice", music="sudden upbeat jazz band sting", sfx="crowd laughter", env="intimate comedy club") - "Make a thunderstorm at night in a forest" ->
audio_scene(operation="generate", caption="A rolling thunderstorm in a forest at night", sfx="distant thunder and heavy rain", env="dense forest", seed=42) - "What's the model state?" ->
audio_scene(operation="status")
Documentation
| Doc | Contents |
|---|---|
| Installation | All install methods, prerequisites |
| Onboarding | First-run model download, GPU checks, pitfalls |
| Wrapped app | MiDashengLM-Gen paper, weights, license, demo |
| Architecture | System architecture, ports, data flow |
| Configuration | Env vars, config options |
| Tool Reference | All available tools |
| Development | Contributing, local setup |
| Troubleshooting | Common issues |
Requirements
- Windows/Linux/macOS with a CUDA GPU (RTX 4090 class: ~12 GB VRAM (fp32); CPU inference works but is slow)
- Python 3.12+ via uv, Node.js 20+, bun
(auto-installed by
start.ps1on naked PCs) - ~10 GB free disk (checkpoint ~6 GB + torch stack)
License
Apache-2.0 (model and this wrapper). See the use-restrictions section in the upstream repo: no unlawful/military use, no harm to minors or groups.
Установка Midasheng Gen
У этого сервера нет опубликованного пакета — он собирается из исходников. Открой репозиторий и следуй инструкции в README.
▸ github.com/sandraschi/midasheng-gen-mcpFAQ
Midasheng Gen MCP бесплатный?
Да, Midasheng Gen MCP бесплатный — установка в пару кликов через Unyly без оплаты.
Нужен ли API-ключ для Midasheng Gen?
Нет, Midasheng Gen работает без API-ключей и переменных окружения.
Midasheng Gen — hosted или self-hosted?
Self-hosted: сервер запускается локально на твоей машине командой из раздела установки.
Как установить Midasheng Gen в Claude Desktop, Claude Code или Cursor?
Открой Midasheng Gen на unyly.org, выбери вкладку своего клиента (Claude Desktop, Claude Code, Cursor) и нажми Install — конфиг сгенерируется автоматически, без правки JSON.
Похожие MCP
ARA
Generate images, video and audio from any AI agent — one connector.
автор: ARAOmni Video
An MCP server that transforms LLM-enabled IDEs into professional video editors by pre-processing footage into text proxies, generating motion graphics via HTML/
автор: buildwithtazaYouTube
Transcripts, channel stats, search
автор: YouTubeEverArt
AI image generation using various models.
автор: modelcontextprotocolgpu-bridge/mcp-server
GPU inference API for LLMs, image and video generation, TTS, Whisper, embeddings, reranking and OCR, paid per use via x402 or an API key.
автор: gpu-bridgemerterbak/Grok-MCP
Access xAI's Grok API with agentic tool calling, image generation, vision and file support.
автор: merterbakSureScaleAI/openai-gpt-image-mcp
Image generation and editing with OpenAI's GPT image models.
автор: SureScaleAIYangLiangwei/PersonalizationMCP
Aggregate personal data from Steam, YouTube, Bilibili, Spotify, Reddit and other gaming, music, video and social platforms, with OAuth2 and token management.
автор: YangLiangweiAceDataCloud/MCPFlux
Flux image generation and editing from Black Forest Labs via the Ace Data Cloud API.
автор: AceDataCloudAceDataCloud/MCPNanoBanana
Nano Banana image generation and editing, including virtual try-on and product placement in realistic scenes.
автор: AceDataCloudCompare Midasheng Gen with
Не уверен что выбрать?
Найди свой стек за 60 секунд
Автор?
Embed-бейдж для README
Похожее
Все в категории media
