giangndm/omniASR-LLM-300M-encoder

omniASR-LLM-300M Audio Encoder (bfloat16 Transformers SafeTensors)

This repository contains the standalone bfloat16 safetensors Audio Encoder extracted from ziywang50/omniASR-LLM-300M / facebook/omniASR-LLM-300M.

It is $100%$ natively compatible with the Hugging Face transformers library (AutoModel / Wav2Vec2Model / AutoFeatureExtractor).

Specifications:

• Architecture: Wav2Vec2Model (Conformer-based SSL 24 layers, 16 heads, 1024 hidden size)
• Precision: bfloat16
• Model Size: ~630 MB (model.safetensors)
• Sampling Rate: 16 kHz
• Output: 50 Hz frame representations (20ms/frame) in $\mathbb{R}^{1024}$

Usage with Hugging Face transformers:

import torch
from transformers import AutoFeatureExtractor, AutoModel

# 1. Load Audio Feature Extractor and Encoder
model_id = "giangndm/omniASR-LLM-300M-encoder"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, torch_dtype=torch.bfloat16).to("cuda")

# 2. Extract Speech Embeddings
audio_input = torch.randn(1, 16000) # 1 sec dummy audio at 16kHz
inputs = feature_extractor(audio_input.squeeze().numpy(), sampling_rate=16000, return_tensors="pt").to("cuda")
inputs["input_values"] = inputs["input_values"].to(torch.bfloat16)

with torch.no_grad():
    hidden_states = model(**inputs).last_hidden_state  # [1, T_frames, 1024]

print("Audio hidden states shape:", hidden_states.shape) # [1, 49, 1024]