warvan-ml-model / tokenizer.json
warvan's picture
Create tokenizer.json
103dc1f verified
raw
history blame contribute delete
151 Bytes
Invalid JSON: Unexpected non-whitespace character after JSON at line 1, column 17
"tokenizer.json": {
"tokenizer_type": "BPE",
"vocab": {
"example": 1,
"tokens": 2
},
"merges": ["ex am ple", "tok ens"]
}