ai4bharat

/

indic-bert

like 76

Follow

AI4Bharat 1.7k

I ran to problem while using :

tokenizer = AutoTokenizer.from_pretrained('ai4bharat/indic-bert')

I was working in colab, upon searching for the solution , I found that sentencepiece need to be installed. I uninstalled transformers and installed it like :

!pip install --no-cache-dir transformers sentencepiece

and then loaded the tokenizer using :

tokenizer = AutoTokenizer.from_pretrained('ai4bharat/indic-bert', use_fast=False)

Hope this helps !