| --- |
| license: apache-2.0 |
| pipeline_tag: automatic-speech-recognition |
| tags: |
| - pytorch |
| - audio |
| - speech |
| - automatic-speech-recognition |
| - whisper |
| - wav2vec2 |
|
|
| model-index: |
| - name: whisper_medium_fp16_transformers |
| results: |
| - task: |
| type: automatic-speech-recognition |
| name: Automatic Speech Recognition |
| dataset: |
| type: librispeech_asr |
| name: LibriSpeech (clean) |
| config: clean |
| split: test |
| args: |
| language: en |
| metrics: |
| - type: wer |
| value: 0 |
| name: Test WER |
| description: Word Error Rate |
| - type: mer |
| value: 0 |
| name: Test MER |
| description: Match Error Rate |
| - type: wil |
| value: 0 |
| name: Test WIL |
| description: Word Information Lost |
| - type: wip |
| value: 0 |
| name: Test WIP |
| description: Word Information Preserved |
| - type: cer |
| value: 0 |
| name: Test CER |
| description: Character Error Rate |
|
|
| - task: |
| type: automatic-speech-recognition |
| name: Automatic Speech Recognition |
| dataset: |
| type: librispeech_asr |
| name: LibriSpeech (other) |
| config: other |
| split: test |
| args: |
| language: en |
| metrics: |
| - type: wer |
| value: 0 |
| name: Test WER |
| description: Word Error Rate |
| - type: mer |
| value: 0 |
| name: Test MER |
| description: Match Error Rate |
| - type: wil |
| value: 0 |
| name: Test WIL |
| description: Word Information Lost |
| - type: wip |
| value: 0 |
| name: Test WIP |
| description: Word Information Preserved |
| - type: cer |
| value: 0 |
| name: Test CER |
| description: Character Error Rate |
| |
| - task: |
| type: automatic-speech-recognition |
| name: Automatic Speech Recognition |
| dataset: |
| type: mozilla-foundation/common_voice_14_0 |
| name: Common Voice (14.0) (Hindi) |
| config: hi |
| split: test |
| args: |
| language: hi |
| metrics: |
| - type: wer |
| value: 54.97 |
| name: Test WER |
| description: Word Error Rate |
| - type: mer |
| value: 47.86 |
| name: Test MER |
| description: Match Error Rate |
| - type: wil |
| value: 66.83 |
| name: Test WIL |
| description: Word Information Lost |
| - type: wip |
| value: 33.16 |
| name: Test WIP |
| description: Word Information Preserved |
| - type: cer |
| value: 30.23 |
| name: Test CER |
| description: Character Error Rate |
|
|
| widget: |
| - example_title: Hinglish Sample |
| src: https://huggingface.co/devasheeshG/whisper_medium_fp16_transformers/resolve/main/test.wav |
| - example_title: Librispeech sample 1 |
| src: https://cdn-media.huggingface.co/speech_samples/sample1.flac |
| - example_title: Librispeech sample 2 |
| src: https://cdn-media.huggingface.co/speech_samples/sample2.flac |
|
|
| language: |
| - en |
| - zh |
| - de |
| - es |
| - ru |
| - ko |
| - fr |
| - ja |
| - pt |
| - tr |
| - pl |
| - ca |
| - nl |
| - ar |
| - sv |
| - it |
| - id |
| - hi |
| - fi |
| - vi |
| - he |
| - uk |
| - el |
| - ms |
| - cs |
| - ro |
| - da |
| - hu |
| - ta |
| - 'no' |
| - th |
| - ur |
| - hr |
| - bg |
| - lt |
| - la |
| - mi |
| - ml |
| - cy |
| - sk |
| - te |
| - fa |
| - lv |
| - bn |
| - sr |
| - az |
| - sl |
| - kn |
| - et |
| - mk |
| - br |
| - eu |
| - is |
| - hy |
| - ne |
| - mn |
| - bs |
| - kk |
| - sq |
| - sw |
| - gl |
| - mr |
| - pa |
| - si |
| - km |
| - sn |
| - yo |
| - so |
| - af |
| - oc |
| - ka |
| - be |
| - tg |
| - sd |
| - gu |
| - am |
| - yi |
| - lo |
| - uz |
| - fo |
| - ht |
| - ps |
| - tk |
| - nn |
| - mt |
| - sa |
| - lb |
| - my |
| - bo |
| - tl |
| - mg |
| - as |
| - tt |
| - haw |
| - ln |
| - ha |
| - ba |
| - jw |
| - su |
| --- |
| ## Versions: |
|
|
| - CUDA: 12.1 |
| - cuDNN Version: 8.9.2.26_1.0-1_amd64 |
|
|
| * tensorflow Version: 2.12.0 |
| * torch Version: 2.1.0.dev20230606+cu12135 |
| * transformers Version: 4.30.2 |
| * accelerate Version: 0.20.3 |
|
|
| ## Model Benchmarks: |
|
|
| - RAM: 2.8 GB (Original_Model: 5.5GB) |
| - VRAM: 1812 MB (Original_Model: 6GB) |
| - test.wav: 23 s (Multilingual Speech i.e. English+Hindi) |
|
|
| - **Time in seconds for Processing by each device** |
|
|
| | Device Name | float32 (Original) | float16 | CudaCores | TensorCores | |
| | ----------------- | ------------------ | ------- | --------- | ----------- | |
| | 3060 | 1.7 | 1.1 | 3,584 | 112 | |
| | 1660 Super | OOM | 3.3 | 1,408 | N/A | |
| | Collab (Tesla T4) | 2.8 | 2.2 | 2,560 | 320 | |
| | Collab (CPU) | 35 | N/A | N/A | N/A | |
| | M1 (CPU) | - | - | - | - | |
| | M1 (GPU -> 'mps') | - | - | - | - | |
|
|
|
|
| - **NOTE: TensorCores are efficient in mixed-precision calculations** |
| - **CPU -> torch.float16 not supported on CPU (AMD Ryzen 5 3600 or Collab CPU)** |
| - Punchuation: True |
|
|
| ## Model Error Benchmarks: |
|
|
| - **WER: Word Error Rate** |
| - **MER: Match Error Rate** |
| - **WIL: Word Information Lost** |
| - **WIP: Word Information Preserved** |
| - **CER: Character Error Rate** |
|
|
| ### Hindi to Hindi (test.tsv) [Common Voice 14.0](https://commonvoice.mozilla.org/en/datasets) |
|
|
| **Test done on RTX 3060 on 2557 Samples** |
|
|
| | | WER | MER | WIL | WIP | CER | |
| | ----------------------- | ----- | ----- | ----- | ----- | ----- | |
| | Original_Model (54 min) | 52.02 | 47.86 | 66.82 | 33.17 | 23.76 | |
| | This_Model (38 min) | 54.97 | 47.86 | 66.83 | 33.16 | 30.23 | |
|
|
| ### Hindi to English (test.csv) [Custom Dataset](https://huggingface.co/datasets/devasheeshG/common_voices_14_0_hi2en_hi2hi) |
|
|
| **Test done on RTX 3060 on 1000 Samples** |
|
|
| | | WER | MER | WIL | WIP | CER | |
| | ----------------------- | --- | --- | --- | --- | --- | |
| | Original_Model (30 min) | - | - | - | - | - | |
| | This_Model (20 min) | - | - | - | - | - | |
|
|
| ### English ([LibriSpeech](https://huggingface.co/datasets/librispeech_asr) -> test-clean) |
|
|
| **Test done on RTX 3060 on __ Samples** |
|
|
| | | WER | MER | WIL | WIP | CER | |
| | -------------- | --- | --- | --- | --- | --- | |
| | Original_Model | - | - | - | - | - | |
| | This_Model | - | - | - | - | - | |
|
|
| ### English ([LibriSpeech](https://huggingface.co/datasets/librispeech_asr) -> test-other) |
|
|
| **Test done on RTX 3060 on __ Samples** |
|
|
| | | WER | MER | WIL | WIP | CER | |
| | -------------- | --- | --- | --- | --- | --- | |
| | Original_Model | - | - | - | - | - | |
| | This_Model | - | - | - | - | - | |
|
|
| - **'jiwer' library is used for calculations** |
|
|
| ## Code for conversion: |
|
|
| - ### [Will be soon Uploaded on Github](https://github.com/devasheeshG) |
|
|
| ## Usage |
|
|
| A file ``__init__.py`` is contained inside this repo which contains all the code to use this model. |
|
|
| Firstly, clone this repo and place all the files inside a folder. |
|
|
| ### Make sure you have git-lfs installed (https://git-lfs.com) |
|
|
| ```bash |
| git lfs install |
| git clone https://huggingface.co/devasheeshG/whisper_medium_fp16_transformers |
| ``` |
|
|
| **Please try in jupyter notebook** |
|
|
| ```python |
| # Import the Model |
| from whisper_medium_fp16_transformers import Model, load_audio, pad_or_trim |
| ``` |
|
|
| ```python |
| # Initilise the model |
| model = Model( |
| model_name_or_path='whisper_medium_fp16_transformers', |
| cuda_visible_device="0", |
| device='cuda', |
| ) |
| ``` |
|
|
| ```python |
| # Load Audio |
| audio = load_audio('whisper_medium_fp16_transformers/test.wav') |
| audio = pad_or_trim(audio) |
| ``` |
|
|
| ```python |
| # Transcribe (First transcription takes time) |
| model.transcribe(audio) |
| ``` |
|
|
| ## Credits |
|
|
| It is fp16 version of ``openai/whisper-medium`` |
|
|
