Command Palette

Search for a command to run...

UnylyUnyly
Весь каталог

Midasheng Gen

БесплатноНе проверен

Generates coherent mixed audio scenes (speech, music, SFX, ambience) from text using an LLM-driven model, with speech in 9 languages and emotion control.

GitHubEmbed

Описание

Generates coherent mixed audio scenes (speech, music, SFX, ambience) from text using an LLM-driven model, with speech in 9 languages and emotion control.

README

MiDashengLM-Gen

midasheng-gen-mcp

Type a scene. Get the whole soundscape.
An MCP server and dark-theme webapp that generate coherent 16 kHz mixed audio scenes - speech, music, sound effects and ambience - in one pass from text, powered by MiDashengLM-Gen (Xiaomi Research). Runs entirely on your GPU.

License: Apache-2.0 Python 3.12+ FastMCP 3.4.4 Local CUDA inference 9 languages MCP hosts Webapp

Quick install · Example prompts · Documentation


One call in Claude, Cursor or opencode - "a comedy club scene with a punchline, crowd laughter and a jazz sting" - and you get a single 16 kHz WAV with all of it. Speech lands close to dedicated TTS quality in 9 languages with emotion control, and the whole ~6 GB model stays on your GPU until you unload it.

What this wraps

MiDashengLM-Gen - the first end-to-end trained general text-to-audio model (per the paper): an LLM drives per-token flow matching to generate variable-length audio scenes with speech intelligibility approaching dedicated TTS (Seed-TTS WER 12.15% -> 2.79% vs 1.24% for dedicated TTS). Supports 9 languages and emotion control. Checkpoint auto-downloads from Hugging Face (~6 GB). Model weights are never bundled - see docs/WRAPPEE.md.

Features

How it runs: a local FastMCP 3.4 server (stdio for Claude Desktop, HTTP /mcp for Cursor/webapp) with a React dashboard. The model loads lazily on first generate and stays on the GPU until unloaded.

Direction Artifacts Notes
Hands-in structured caption (caption/asr/speech/sfx/music/env views), guidance + seed params Via MCP tool, REST, or webapp
Hands-out 16 kHz mono WAV scenes, indexed in SQLite, browsable/exportable audio_scene(operation="export") or webapp
  • Generate whole soundscapes in one call - crowd laughter + jazz sting + comedy speech, or rain + thunder + forest ambience
  • Speech intelligibility near dedicated TTS quality, 9 languages, emotion control through the speech view
  • Scene library with pagination, audio playback, and export in the webapp
  • Async job API for long generations; Prefab UI cards in chat
  • Variable-length output via the learned stop head - no fixed-duration cuts

Quick Install

The fastest path is the .mcpb bundle for Claude Desktop (see INSTALL.md for all options):

  1. Download midasheng-gen-mcp-v0.1.0.mcpb from Releases
  2. Drag it onto Claude Desktop
  3. First use will download the ~6 GB checkpoint automatically

Or clone and double-click start.bat for the full stack (backend + webapp).

Example Prompts

  • "Generate a comedy club scene: a punchline, crowd laughter, and a jazz band sting" -> audio_scene(operation="generate", caption="A comedian delivering a punchline followed by uproarious crowd laughter", asr="And that is why I never buy cheap luggage anymore!", speech="expressive comedic male voice", music="sudden upbeat jazz band sting", sfx="crowd laughter", env="intimate comedy club")
  • "Make a thunderstorm at night in a forest" -> audio_scene(operation="generate", caption="A rolling thunderstorm in a forest at night", sfx="distant thunder and heavy rain", env="dense forest", seed=42)
  • "What's the model state?" -> audio_scene(operation="status")

Documentation

Doc Contents
Installation All install methods, prerequisites
Onboarding First-run model download, GPU checks, pitfalls
Wrapped app MiDashengLM-Gen paper, weights, license, demo
Architecture System architecture, ports, data flow
Configuration Env vars, config options
Tool Reference All available tools
Development Contributing, local setup
Troubleshooting Common issues

Requirements

  • Windows/Linux/macOS with a CUDA GPU (RTX 4090 class: ~12 GB VRAM (fp32); CPU inference works but is slow)
  • Python 3.12+ via uv, Node.js 20+, bun (auto-installed by start.ps1 on naked PCs)
  • ~10 GB free disk (checkpoint ~6 GB + torch stack)

License

Apache-2.0 (model and this wrapper). See the use-restrictions section in the upstream repo: no unlawful/military use, no harm to minors or groups.

from github.com/sandraschi/midasheng-gen-mcp

Установка Midasheng Gen

У этого сервера нет опубликованного пакета — он собирается из исходников. Открой репозиторий и следуй инструкции в README.

▸ github.com/sandraschi/midasheng-gen-mcp

FAQ

Midasheng Gen MCP бесплатный?

Да, Midasheng Gen MCP бесплатный — установка в пару кликов через Unyly без оплаты.

Нужен ли API-ключ для Midasheng Gen?

Нет, Midasheng Gen работает без API-ключей и переменных окружения.

Midasheng Gen — hosted или self-hosted?

Self-hosted: сервер запускается локально на твоей машине командой из раздела установки.

Как установить Midasheng Gen в Claude Desktop, Claude Code или Cursor?

Открой Midasheng Gen на unyly.org, выбери вкладку своего клиента (Claude Desktop, Claude Code, Cursor) и нажми Install — конфиг сгенерируется автоматически, без правки JSON.

Похожие MCP

Compare Midasheng Gen with

Не уверен что выбрать?

Найди свой стек за 60 секунд

Автор?

Embed-бейдж для README

Похожее

Все в категории media