mtmd : add MERaLiON-2 multimodal audio support (#21756)

* mtmd : add MERaLiON-2 multimodal audio support

Adds support for A*STAR's MERaLiON-2 audio-language model (3B and 10B)
to the multimodal framework.

Architecture:
- Whisper large-v2 encoder for audio feature extraction
- Gated MLP adaptor: ln_speech -> frame stack (x15) -> Linear+SiLU -> GLU -> out_proj
- Gemma2 3B / 27B decoder

The mmproj GGUF is generated via convert_hf_to_gguf.py --mmproj on the full
MERaLiON-2 model directory (architecture: MERaLiON2ForConditionalGeneration).
The decoder is converted separately as a standard Gemma2 model after stripping
the text_decoder. weight prefix.

New projector type: PROJECTOR_TYPE_MERALION

Supports tasks: speech transcription (EN/ZH/MS/TA), translation, spoken QA.

Model: https://huggingface.co/MERaLiON/MERaLiON-2-3B
       https://huggingface.co/MERaLiON/MERaLiON-2-10B

* simplify comments in meralion adaptor

* meralion: use format_tensor_name, ascii arrows in comments

This commit is contained in:

Sirui He

2026-04-11 14:15:48 +02:00

committed by

GitHub

parent af1127d3c4

commit 073bb2c20b

8 changed files with 103 additions and 2 deletions

									
										tools/mtmd/clip-model.h
									
		+2
		-1
	
												View File
												
				@@ -467,7 +467,8 @@ struct clip_model {

				    bool audio_has_stack_frames() const {

				        return proj_type == PROJECTOR_TYPE_ULTRAVOX

				            || proj_type == PROJECTOR_TYPE_VOXTRAL;

				            || proj_type == PROJECTOR_TYPE_VOXTRAL

				            || proj_type == PROJECTOR_TYPE_MERALION;

				    }

				};