Deploy with ExecuTorch

Serve Muse Glimmer on your own CUDA GPU or Apple silicon with ExecuTorch(opens in new tab), behind an OpenAI-compatible HTTP endpoint that supports vision, tool calling, and DFlash speculative decoding. ExecuTorch is ahead-of-time: you export the model once into a .pte program for a specific backend, then serve from that program.

Each of the other runtimes reimplements the model by hand for every backend it supports. That approach scales for a plain text transformer. Muse Glimmer also carries multimodal input and block-diffusion speculative decoding, so each backend would need its own rewrite of all three. With ExecuTorch you write the model and its decoding strategy once in PyTorch, and torch.export lowers the whole graph ahead of time: to Triton on CUDA, and to MLX-native or custom Metal kernels on Apple silicon.

Prebuilt exports are published as well, under different filenames, so exporting a 30B model yourself is optional.

Prerequisites

• An ExecuTorch checkout built from source per the upstream guide(opens in new tab), including examples/models/muse-glimmer
• The server dependencies: pip install -r examples/llm_server/python/requirements.txt
• The Hugging Face CLI (pip install huggingface_hub)
• A CUDA GPU for the CUDA backend, or Apple silicon for the MLX backend

Run every command below from the ExecuTorch repository root.

Download the model assets

Exports lower directly from the quantized GGUF checkpoints, the same files llama.cpp uses. The serving path also needs the tokenizer metadata from the main model repository:

bash
hf download meta-models/Muse-Glimmer-30B-GGUF --local-dir assets/quant \
  --include 'Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf' \
  --include 'dflash-Muse-Glimmer-30B-Q4_K_M.gguf' \
  --include 'mmproj-Muse-Glimmer-30B-Q4_K_M.gguf'
​
hf download meta-models/Muse-Glimmer-30B \
  tokenizer.json chat_template.jinja config.json processor_config.json \
  --local-dir assets/hf

Keep chat_template.jinja beside the rest of the tokenizer metadata: the serving path renders prompts and tool definitions with it.

Set the paths used by the commands that follow:

bash
TARGET=assets/quant/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf
DRAFT=assets/quant/dflash-Muse-Glimmer-30B-Q4_K_M.gguf
MMPROJ=assets/quant/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf
BACKEND=cuda   # mlx on macOS

Export the model

$BACKEND selects the target: cuda or mlx. A CUDA export autotunes Triton kernels against the GPU it runs on, so export on the same architecture you intend to serve from.

Target-only export

For text inference:

bash
python -m executorch.examples.models.muse_glimmer.export.export_solo \
  --gguf "$TARGET" \
  --backend "$BACKEND" \
  --output-dir exports/solo

DFlash export

DFlash speculative decoding lowers the target and the draft together, into their own export directory:

bash
python -m executorch.examples.models.muse_glimmer.export.export_dflash \
  --target-gguf "$TARGET" \
  --draft-gguf "$DRAFT" \
  --backend "$BACKEND" \
  --output-dir exports/dflash

Both land in one .pte: the draft shares the target's token embeddings and output head rather than carrying copies. The block dimension is exported dynamically, so you select block length at serve time. The exported range is backend-specific: [2, 16] on MLX and [2, 4] on CUDA, where the draft count is also capped at 3.

Vision exports

Add --mmproj "$MMPROJ" to either export command for text and image inference. A vision export also writes pos_embed.bin beside model.pte.

Download a prebuilt export

meta-models/Muse-Glimmer-30B-ExecuTorch-PTE(opens in new tab) publishes 16 ready-made exports, which skips the export step. You still build the runner: a .pte is a model program, not a runtime.

Directories are named muse-glimmer-<quantization>-128K-<modality>-<decoding>-<backend>, and all 16 combinations of the four axes exist:

• Quantization: k-quant-17G targets 24 GB of VRAM. k-quant-dynamic targets 32 GB with less degradation, and the model card(opens in new tab) quantifies the tradeoff.
• Modality: text, or text-image for vision.
• Decoding: solo, or dflash for speculative decoding.
• Backend: metal for Apple silicon, sm80+ptx for CUDA on SM80 and newer.

Context length is 128K for every variant. Sizes, by directory: