About model_max_length
#1
by hongwen11 - opened
in tokenizer_config.json it says: "model_max_length": 1000000000000000019884624838656,
Can you kindly tell me the length distribution of train set? That will help me to adjust better chunk length when testing your model.
Our query and passage are trained with a length of 512, but the maximum length of the query with examples is set to 2048.
Thanks for reply. Do you train all your passages with a length of 512, or their lengths are different while averages around 512?
Just like other LLM-based embedding models, we set all passages with a length of 512.
But the model can handle documents larger than 512, right? If so, would it be better to truncate to 512 or not?