| --- |
| tags: |
| - mteb |
| model-index: |
| - name: checkpoint-1431 |
| results: |
| - task: |
| type: STS |
| dataset: |
| type: C-MTEB/AFQMC |
| name: MTEB AFQMC |
| config: default |
| split: validation |
| revision: None |
| metrics: |
| - type: cos_sim_pearson |
| value: 56.306314279047875 |
| - type: cos_sim_spearman |
| value: 61.020227685004016 |
| - type: euclidean_pearson |
| value: 58.61821670933433 |
| - type: euclidean_spearman |
| value: 60.131457106640674 |
| - type: manhattan_pearson |
| value: 58.6189460369694 |
| - type: manhattan_spearman |
| value: 60.126350618526224 |
| - task: |
| type: STS |
| dataset: |
| type: C-MTEB/ATEC |
| name: MTEB ATEC |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: cos_sim_pearson |
| value: 55.8612958476143 |
| - type: cos_sim_spearman |
| value: 59.01977664864512 |
| - type: euclidean_pearson |
| value: 62.028094897243655 |
| - type: euclidean_spearman |
| value: 58.6046814257705 |
| - type: manhattan_pearson |
| value: 62.02580042431887 |
| - type: manhattan_spearman |
| value: 58.60626890004892 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/amazon_reviews_multi |
| name: MTEB AmazonReviewsClassification (zh) |
| config: zh |
| split: test |
| revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
| metrics: |
| - type: accuracy |
| value: 49.496 |
| - type: f1 |
| value: 46.673963383873065 |
| - task: |
| type: STS |
| dataset: |
| type: C-MTEB/BQ |
| name: MTEB BQ |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: cos_sim_pearson |
| value: 70.73971622592535 |
| - type: cos_sim_spearman |
| value: 72.76102992060764 |
| - type: euclidean_pearson |
| value: 71.04525865868672 |
| - type: euclidean_spearman |
| value: 72.4032852155075 |
| - type: manhattan_pearson |
| value: 71.03693009336658 |
| - type: manhattan_spearman |
| value: 72.39635701224252 |
| - task: |
| type: Clustering |
| dataset: |
| type: C-MTEB/CLSClusteringP2P |
| name: MTEB CLSClusteringP2P |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: v_measure |
| value: 56.34751074520767 |
| - task: |
| type: Clustering |
| dataset: |
| type: C-MTEB/CLSClusteringS2S |
| name: MTEB CLSClusteringS2S |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: v_measure |
| value: 48.4856662121073 |
| - task: |
| type: Reranking |
| dataset: |
| type: C-MTEB/CMedQAv1-reranking |
| name: MTEB CMedQAv1 |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map |
| value: 89.26384109024997 |
| - type: mrr |
| value: 91.27261904761905 |
| - task: |
| type: Reranking |
| dataset: |
| type: C-MTEB/CMedQAv2-reranking |
| name: MTEB CMedQAv2 |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: map |
| value: 90.0464058154547 |
| - type: mrr |
| value: 92.06480158730159 |
| - task: |
| type: Retrieval |
| dataset: |
| type: C-MTEB/CmedqaRetrieval |
| name: MTEB CmedqaRetrieval |
| config: default |
| split: dev |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 27.236 |
| - type: map_at_10 |
| value: 40.778 |
| - type: map_at_100 |
| value: 42.692 |
| - type: map_at_1000 |
| value: 42.787 |
| - type: map_at_3 |
| value: 36.362 |
| - type: map_at_5 |
| value: 38.839 |
| - type: mrr_at_1 |
| value: 41.335 |
| - type: mrr_at_10 |
| value: 49.867 |
| - type: mrr_at_100 |
| value: 50.812999999999995 |
| - type: mrr_at_1000 |
| value: 50.848000000000006 |
| - type: mrr_at_3 |
| value: 47.354 |
| - type: mrr_at_5 |
| value: 48.718 |
| - type: ndcg_at_1 |
| value: 41.335 |
| - type: ndcg_at_10 |
| value: 47.642 |
| - type: ndcg_at_100 |
| value: 54.855 |
| - type: ndcg_at_1000 |
| value: 56.449000000000005 |
| - type: ndcg_at_3 |
| value: 42.203 |
| - type: ndcg_at_5 |
| value: 44.416 |
| - type: precision_at_1 |
| value: 41.335 |
| - type: precision_at_10 |
| value: 10.568 |
| - type: precision_at_100 |
| value: 1.6400000000000001 |
| - type: precision_at_1000 |
| value: 0.184 |
| - type: precision_at_3 |
| value: 23.998 |
| - type: precision_at_5 |
| value: 17.389 |
| - type: recall_at_1 |
| value: 27.236 |
| - type: recall_at_10 |
| value: 58.80800000000001 |
| - type: recall_at_100 |
| value: 88.411 |
| - type: recall_at_1000 |
| value: 99.032 |
| - type: recall_at_3 |
| value: 42.253 |
| - type: recall_at_5 |
| value: 49.118 |
| - task: |
| type: PairClassification |
| dataset: |
| type: C-MTEB/CMNLI |
| name: MTEB Cmnli |
| config: default |
| split: validation |
| revision: None |
| metrics: |
| - type: cos_sim_accuracy |
| value: 86.03728202044498 |
| - type: cos_sim_ap |
| value: 92.49469583272597 |
| - type: cos_sim_f1 |
| value: 86.74095974528088 |
| - type: cos_sim_precision |
| value: 84.43657294664601 |
| - type: cos_sim_recall |
| value: 89.17465513210195 |
| - type: dot_accuracy |
| value: 72.21888153938664 |
| - type: dot_ap |
| value: 80.59377163340332 |
| - type: dot_f1 |
| value: 74.96686040583258 |
| - type: dot_precision |
| value: 66.4737793851718 |
| - type: dot_recall |
| value: 85.94809445873275 |
| - type: euclidean_accuracy |
| value: 85.47203848466627 |
| - type: euclidean_ap |
| value: 91.89152584749868 |
| - type: euclidean_f1 |
| value: 86.38105975197294 |
| - type: euclidean_precision |
| value: 83.40953625081646 |
| - type: euclidean_recall |
| value: 89.5721299976619 |
| - type: manhattan_accuracy |
| value: 85.3758268190018 |
| - type: manhattan_ap |
| value: 91.88989707722311 |
| - type: manhattan_f1 |
| value: 86.39767519839052 |
| - type: manhattan_precision |
| value: 82.76231263383298 |
| - type: manhattan_recall |
| value: 90.36707972878185 |
| - type: max_accuracy |
| value: 86.03728202044498 |
| - type: max_ap |
| value: 92.49469583272597 |
| - type: max_f1 |
| value: 86.74095974528088 |
| - task: |
| type: Retrieval |
| dataset: |
| type: C-MTEB/CovidRetrieval |
| name: MTEB CovidRetrieval |
| config: default |
| split: dev |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 74.34100000000001 |
| - type: map_at_10 |
| value: 82.49499999999999 |
| - type: map_at_100 |
| value: 82.64200000000001 |
| - type: map_at_1000 |
| value: 82.643 |
| - type: map_at_3 |
| value: 81.142 |
| - type: map_at_5 |
| value: 81.95400000000001 |
| - type: mrr_at_1 |
| value: 74.71 |
| - type: mrr_at_10 |
| value: 82.553 |
| - type: mrr_at_100 |
| value: 82.699 |
| - type: mrr_at_1000 |
| value: 82.70100000000001 |
| - type: mrr_at_3 |
| value: 81.279 |
| - type: mrr_at_5 |
| value: 82.069 |
| - type: ndcg_at_1 |
| value: 74.605 |
| - type: ndcg_at_10 |
| value: 85.946 |
| - type: ndcg_at_100 |
| value: 86.607 |
| - type: ndcg_at_1000 |
| value: 86.669 |
| - type: ndcg_at_3 |
| value: 83.263 |
| - type: ndcg_at_5 |
| value: 84.71600000000001 |
| - type: precision_at_1 |
| value: 74.605 |
| - type: precision_at_10 |
| value: 9.758 |
| - type: precision_at_100 |
| value: 1.005 |
| - type: precision_at_1000 |
| value: 0.101 |
| - type: precision_at_3 |
| value: 29.996000000000002 |
| - type: precision_at_5 |
| value: 18.736 |
| - type: recall_at_1 |
| value: 74.34100000000001 |
| - type: recall_at_10 |
| value: 96.523 |
| - type: recall_at_100 |
| value: 99.473 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_3 |
| value: 89.278 |
| - type: recall_at_5 |
| value: 92.83500000000001 |
| - task: |
| type: Retrieval |
| dataset: |
| type: C-MTEB/DuRetrieval |
| name: MTEB DuRetrieval |
| config: default |
| split: dev |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 26.950000000000003 |
| - type: map_at_10 |
| value: 82.408 |
| - type: map_at_100 |
| value: 85.057 |
| - type: map_at_1000 |
| value: 85.09100000000001 |
| - type: map_at_3 |
| value: 57.635999999999996 |
| - type: map_at_5 |
| value: 72.48 |
| - type: mrr_at_1 |
| value: 92.15 |
| - type: mrr_at_10 |
| value: 94.554 |
| - type: mrr_at_100 |
| value: 94.608 |
| - type: mrr_at_1000 |
| value: 94.61 |
| - type: mrr_at_3 |
| value: 94.292 |
| - type: mrr_at_5 |
| value: 94.459 |
| - type: ndcg_at_1 |
| value: 92.15 |
| - type: ndcg_at_10 |
| value: 89.108 |
| - type: ndcg_at_100 |
| value: 91.525 |
| - type: ndcg_at_1000 |
| value: 91.82900000000001 |
| - type: ndcg_at_3 |
| value: 88.44 |
| - type: ndcg_at_5 |
| value: 87.271 |
| - type: precision_at_1 |
| value: 92.15 |
| - type: precision_at_10 |
| value: 42.29 |
| - type: precision_at_100 |
| value: 4.812 |
| - type: precision_at_1000 |
| value: 0.48900000000000005 |
| - type: precision_at_3 |
| value: 79.14999999999999 |
| - type: precision_at_5 |
| value: 66.64 |
| - type: recall_at_1 |
| value: 26.950000000000003 |
| - type: recall_at_10 |
| value: 89.832 |
| - type: recall_at_100 |
| value: 97.921 |
| - type: recall_at_1000 |
| value: 99.471 |
| - type: recall_at_3 |
| value: 59.562000000000005 |
| - type: recall_at_5 |
| value: 76.533 |
| - task: |
| type: Retrieval |
| dataset: |
| type: C-MTEB/EcomRetrieval |
| name: MTEB EcomRetrieval |
| config: default |
| split: dev |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 53.5 |
| - type: map_at_10 |
| value: 63.105999999999995 |
| - type: map_at_100 |
| value: 63.63100000000001 |
| - type: map_at_1000 |
| value: 63.641999999999996 |
| - type: map_at_3 |
| value: 60.617 |
| - type: map_at_5 |
| value: 62.132 |
| - type: mrr_at_1 |
| value: 53.5 |
| - type: mrr_at_10 |
| value: 63.105999999999995 |
| - type: mrr_at_100 |
| value: 63.63100000000001 |
| - type: mrr_at_1000 |
| value: 63.641999999999996 |
| - type: mrr_at_3 |
| value: 60.617 |
| - type: mrr_at_5 |
| value: 62.132 |
| - type: ndcg_at_1 |
| value: 53.5 |
| - type: ndcg_at_10 |
| value: 67.92200000000001 |
| - type: ndcg_at_100 |
| value: 70.486 |
| - type: ndcg_at_1000 |
| value: 70.777 |
| - type: ndcg_at_3 |
| value: 62.853 |
| - type: ndcg_at_5 |
| value: 65.59899999999999 |
| - type: precision_at_1 |
| value: 53.5 |
| - type: precision_at_10 |
| value: 8.309999999999999 |
| - type: precision_at_100 |
| value: 0.951 |
| - type: precision_at_1000 |
| value: 0.097 |
| - type: precision_at_3 |
| value: 23.1 |
| - type: precision_at_5 |
| value: 15.2 |
| - type: recall_at_1 |
| value: 53.5 |
| - type: recall_at_10 |
| value: 83.1 |
| - type: recall_at_100 |
| value: 95.1 |
| - type: recall_at_1000 |
| value: 97.39999999999999 |
| - type: recall_at_3 |
| value: 69.3 |
| - type: recall_at_5 |
| value: 76.0 |
| - task: |
| type: Classification |
| dataset: |
| type: C-MTEB/IFlyTek-classification |
| name: MTEB IFlyTek |
| config: default |
| split: validation |
| revision: None |
| metrics: |
| - type: accuracy |
| value: 51.773759138130046 |
| - type: f1 |
| value: 40.38600802756481 |
| - task: |
| type: Classification |
| dataset: |
| type: C-MTEB/JDReview-classification |
| name: MTEB JDReview |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: accuracy |
| value: 88.48030018761726 |
| - type: ap |
| value: 59.2732541555627 |
| - type: f1 |
| value: 83.58836007358619 |
| - task: |
| type: STS |
| dataset: |
| type: C-MTEB/LCQMC |
| name: MTEB LCQMC |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: cos_sim_pearson |
| value: 73.67511194245922 |
| - type: cos_sim_spearman |
| value: 79.43347759067298 |
| - type: euclidean_pearson |
| value: 79.04491504318766 |
| - type: euclidean_spearman |
| value: 79.14478545356785 |
| - type: manhattan_pearson |
| value: 79.03365022867428 |
| - type: manhattan_spearman |
| value: 79.13172717619908 |
| - task: |
| type: Retrieval |
| dataset: |
| type: C-MTEB/MMarcoRetrieval |
| name: MTEB MMarcoRetrieval |
| config: default |
| split: dev |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 67.184 |
| - type: map_at_10 |
| value: 76.24600000000001 |
| - type: map_at_100 |
| value: 76.563 |
| - type: map_at_1000 |
| value: 76.575 |
| - type: map_at_3 |
| value: 74.522 |
| - type: map_at_5 |
| value: 75.598 |
| - type: mrr_at_1 |
| value: 69.47 |
| - type: mrr_at_10 |
| value: 76.8 |
| - type: mrr_at_100 |
| value: 77.082 |
| - type: mrr_at_1000 |
| value: 77.093 |
| - type: mrr_at_3 |
| value: 75.29400000000001 |
| - type: mrr_at_5 |
| value: 76.24 |
| - type: ndcg_at_1 |
| value: 69.47 |
| - type: ndcg_at_10 |
| value: 79.81099999999999 |
| - type: ndcg_at_100 |
| value: 81.187 |
| - type: ndcg_at_1000 |
| value: 81.492 |
| - type: ndcg_at_3 |
| value: 76.536 |
| - type: ndcg_at_5 |
| value: 78.367 |
| - type: precision_at_1 |
| value: 69.47 |
| - type: precision_at_10 |
| value: 9.599 |
| - type: precision_at_100 |
| value: 1.026 |
| - type: precision_at_1000 |
| value: 0.105 |
| - type: precision_at_3 |
| value: 28.777 |
| - type: precision_at_5 |
| value: 18.232 |
| - type: recall_at_1 |
| value: 67.184 |
| - type: recall_at_10 |
| value: 90.211 |
| - type: recall_at_100 |
| value: 96.322 |
| - type: recall_at_1000 |
| value: 98.699 |
| - type: recall_at_3 |
| value: 81.556 |
| - type: recall_at_5 |
| value: 85.931 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/amazon_massive_intent |
| name: MTEB MassiveIntentClassification (zh-CN) |
| config: zh-CN |
| split: test |
| revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| metrics: |
| - type: accuracy |
| value: 76.96032279757901 |
| - type: f1 |
| value: 73.48052314033545 |
| - task: |
| type: Classification |
| dataset: |
| type: mteb/amazon_massive_scenario |
| name: MTEB MassiveScenarioClassification (zh-CN) |
| config: zh-CN |
| split: test |
| revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| metrics: |
| - type: accuracy |
| value: 84.64357767316744 |
| - type: f1 |
| value: 83.58250539497922 |
| - task: |
| type: Retrieval |
| dataset: |
| type: C-MTEB/MedicalRetrieval |
| name: MTEB MedicalRetrieval |
| config: default |
| split: dev |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 56.00000000000001 |
| - type: map_at_10 |
| value: 62.066 |
| - type: map_at_100 |
| value: 62.553000000000004 |
| - type: map_at_1000 |
| value: 62.598 |
| - type: map_at_3 |
| value: 60.4 |
| - type: map_at_5 |
| value: 61.370000000000005 |
| - type: mrr_at_1 |
| value: 56.2 |
| - type: mrr_at_10 |
| value: 62.166 |
| - type: mrr_at_100 |
| value: 62.653000000000006 |
| - type: mrr_at_1000 |
| value: 62.699000000000005 |
| - type: mrr_at_3 |
| value: 60.5 |
| - type: mrr_at_5 |
| value: 61.47 |
| - type: ndcg_at_1 |
| value: 56.00000000000001 |
| - type: ndcg_at_10 |
| value: 65.199 |
| - type: ndcg_at_100 |
| value: 67.79899999999999 |
| - type: ndcg_at_1000 |
| value: 69.056 |
| - type: ndcg_at_3 |
| value: 61.814 |
| - type: ndcg_at_5 |
| value: 63.553000000000004 |
| - type: precision_at_1 |
| value: 56.00000000000001 |
| - type: precision_at_10 |
| value: 7.51 |
| - type: precision_at_100 |
| value: 0.878 |
| - type: precision_at_1000 |
| value: 0.098 |
| - type: precision_at_3 |
| value: 21.967 |
| - type: precision_at_5 |
| value: 14.02 |
| - type: recall_at_1 |
| value: 56.00000000000001 |
| - type: recall_at_10 |
| value: 75.1 |
| - type: recall_at_100 |
| value: 87.8 |
| - type: recall_at_1000 |
| value: 97.7 |
| - type: recall_at_3 |
| value: 65.9 |
| - type: recall_at_5 |
| value: 70.1 |
| - task: |
| type: Reranking |
| dataset: |
| type: C-MTEB/Mmarco-reranking |
| name: MTEB MMarcoReranking |
| config: default |
| split: dev |
| revision: None |
| metrics: |
| - type: map |
| value: 32.74158258279793 |
| - type: mrr |
| value: 31.56071428571428 |
| - task: |
| type: Classification |
| dataset: |
| type: C-MTEB/MultilingualSentiment-classification |
| name: MTEB MultilingualSentiment |
| config: default |
| split: validation |
| revision: None |
| metrics: |
| - type: accuracy |
| value: 78.96666666666667 |
| - type: f1 |
| value: 78.82528563818045 |
| - task: |
| type: PairClassification |
| dataset: |
| type: C-MTEB/OCNLI |
| name: MTEB Ocnli |
| config: default |
| split: validation |
| revision: None |
| metrics: |
| - type: cos_sim_accuracy |
| value: 83.54087709799674 |
| - type: cos_sim_ap |
| value: 87.26170197077586 |
| - type: cos_sim_f1 |
| value: 84.7609561752988 |
| - type: cos_sim_precision |
| value: 80.20735155513667 |
| - type: cos_sim_recall |
| value: 89.86272439281943 |
| - type: dot_accuracy |
| value: 72.22523010286952 |
| - type: dot_ap |
| value: 79.51975358187732 |
| - type: dot_f1 |
| value: 76.32183908045977 |
| - type: dot_precision |
| value: 67.58957654723126 |
| - type: dot_recall |
| value: 87.64519535374869 |
| - type: euclidean_accuracy |
| value: 82.0249052517596 |
| - type: euclidean_ap |
| value: 85.32829948726406 |
| - type: euclidean_f1 |
| value: 83.24924318869829 |
| - type: euclidean_precision |
| value: 79.71014492753623 |
| - type: euclidean_recall |
| value: 87.11721224920802 |
| - type: manhattan_accuracy |
| value: 82.13318895506227 |
| - type: manhattan_ap |
| value: 85.28856869288006 |
| - type: manhattan_f1 |
| value: 83.34946757018393 |
| - type: manhattan_precision |
| value: 76.94369973190348 |
| - type: manhattan_recall |
| value: 90.91869060190075 |
| - type: max_accuracy |
| value: 83.54087709799674 |
| - type: max_ap |
| value: 87.26170197077586 |
| - type: max_f1 |
| value: 84.7609561752988 |
| - task: |
| type: Classification |
| dataset: |
| type: C-MTEB/OnlineShopping-classification |
| name: MTEB OnlineShopping |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: accuracy |
| value: 94.56 |
| - type: ap |
| value: 92.80848436710805 |
| - type: f1 |
| value: 94.54951966576111 |
| - task: |
| type: STS |
| dataset: |
| type: C-MTEB/PAWSX |
| name: MTEB PAWSX |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: cos_sim_pearson |
| value: 39.0866558287863 |
| - type: cos_sim_spearman |
| value: 45.9211126233312 |
| - type: euclidean_pearson |
| value: 44.86568743222145 |
| - type: euclidean_spearman |
| value: 45.63882757207507 |
| - type: manhattan_pearson |
| value: 44.89480036909126 |
| - type: manhattan_spearman |
| value: 45.65929449046206 |
| - task: |
| type: STS |
| dataset: |
| type: C-MTEB/QBQTC |
| name: MTEB QBQTC |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: cos_sim_pearson |
| value: 43.04701793979569 |
| - type: cos_sim_spearman |
| value: 44.87491033760315 |
| - type: euclidean_pearson |
| value: 36.2004061032567 |
| - type: euclidean_spearman |
| value: 41.44823909683865 |
| - type: manhattan_pearson |
| value: 36.136113427955095 |
| - type: manhattan_spearman |
| value: 41.39225495993949 |
| - task: |
| type: STS |
| dataset: |
| type: mteb/sts22-crosslingual-sts |
| name: MTEB STS22 (zh) |
| config: zh |
| split: test |
| revision: None |
| metrics: |
| - type: cos_sim_pearson |
| value: 61.65611315777857 |
| - type: cos_sim_spearman |
| value: 64.4067673105648 |
| - type: euclidean_pearson |
| value: 61.814977248797184 |
| - type: euclidean_spearman |
| value: 63.99473350700169 |
| - type: manhattan_pearson |
| value: 61.684304629588624 |
| - type: manhattan_spearman |
| value: 63.97831213239316 |
| - task: |
| type: STS |
| dataset: |
| type: C-MTEB/STSB |
| name: MTEB STSB |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: cos_sim_pearson |
| value: 76.57324933064379 |
| - type: cos_sim_spearman |
| value: 79.23602286949782 |
| - type: euclidean_pearson |
| value: 80.28226284310948 |
| - type: euclidean_spearman |
| value: 80.32210477608423 |
| - type: manhattan_pearson |
| value: 80.27262188617811 |
| - type: manhattan_spearman |
| value: 80.31619185039723 |
| - task: |
| type: Reranking |
| dataset: |
| type: C-MTEB/T2Reranking |
| name: MTEB T2Reranking |
| config: default |
| split: dev |
| revision: None |
| metrics: |
| - type: map |
| value: 67.05266891356277 |
| - type: mrr |
| value: 77.1906333623497 |
| - task: |
| type: Retrieval |
| dataset: |
| type: C-MTEB/T2Retrieval |
| name: MTEB T2Retrieval |
| config: default |
| split: dev |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 28.212 |
| - type: map_at_10 |
| value: 78.932 |
| - type: map_at_100 |
| value: 82.51899999999999 |
| - type: map_at_1000 |
| value: 82.575 |
| - type: map_at_3 |
| value: 55.614 |
| - type: map_at_5 |
| value: 68.304 |
| - type: mrr_at_1 |
| value: 91.211 |
| - type: mrr_at_10 |
| value: 93.589 |
| - type: mrr_at_100 |
| value: 93.659 |
| - type: mrr_at_1000 |
| value: 93.662 |
| - type: mrr_at_3 |
| value: 93.218 |
| - type: mrr_at_5 |
| value: 93.453 |
| - type: ndcg_at_1 |
| value: 91.211 |
| - type: ndcg_at_10 |
| value: 86.24000000000001 |
| - type: ndcg_at_100 |
| value: 89.614 |
| - type: ndcg_at_1000 |
| value: 90.14 |
| - type: ndcg_at_3 |
| value: 87.589 |
| - type: ndcg_at_5 |
| value: 86.265 |
| - type: precision_at_1 |
| value: 91.211 |
| - type: precision_at_10 |
| value: 42.626 |
| - type: precision_at_100 |
| value: 5.043 |
| - type: precision_at_1000 |
| value: 0.517 |
| - type: precision_at_3 |
| value: 76.42 |
| - type: precision_at_5 |
| value: 64.045 |
| - type: recall_at_1 |
| value: 28.212 |
| - type: recall_at_10 |
| value: 85.223 |
| - type: recall_at_100 |
| value: 96.229 |
| - type: recall_at_1000 |
| value: 98.849 |
| - type: recall_at_3 |
| value: 57.30800000000001 |
| - type: recall_at_5 |
| value: 71.661 |
| - task: |
| type: Classification |
| dataset: |
| type: C-MTEB/TNews-classification |
| name: MTEB TNews |
| config: default |
| split: validation |
| revision: None |
| metrics: |
| - type: accuracy |
| value: 54.385000000000005 |
| - type: f1 |
| value: 52.38762400903556 |
| - task: |
| type: Clustering |
| dataset: |
| type: C-MTEB/ThuNewsClusteringP2P |
| name: MTEB ThuNewsClusteringP2P |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: v_measure |
| value: 74.55283855942916 |
| - task: |
| type: Clustering |
| dataset: |
| type: C-MTEB/ThuNewsClusteringS2S |
| name: MTEB ThuNewsClusteringS2S |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: v_measure |
| value: 68.55115316700493 |
| - task: |
| type: Retrieval |
| dataset: |
| type: C-MTEB/VideoRetrieval |
| name: MTEB VideoRetrieval |
| config: default |
| split: dev |
| revision: None |
| metrics: |
| - type: map_at_1 |
| value: 58.8 |
| - type: map_at_10 |
| value: 69.035 |
| - type: map_at_100 |
| value: 69.52000000000001 |
| - type: map_at_1000 |
| value: 69.529 |
| - type: map_at_3 |
| value: 67.417 |
| - type: map_at_5 |
| value: 68.407 |
| - type: mrr_at_1 |
| value: 58.8 |
| - type: mrr_at_10 |
| value: 69.035 |
| - type: mrr_at_100 |
| value: 69.52000000000001 |
| - type: mrr_at_1000 |
| value: 69.529 |
| - type: mrr_at_3 |
| value: 67.417 |
| - type: mrr_at_5 |
| value: 68.407 |
| - type: ndcg_at_1 |
| value: 58.8 |
| - type: ndcg_at_10 |
| value: 73.395 |
| - type: ndcg_at_100 |
| value: 75.62 |
| - type: ndcg_at_1000 |
| value: 75.90299999999999 |
| - type: ndcg_at_3 |
| value: 70.11800000000001 |
| - type: ndcg_at_5 |
| value: 71.87400000000001 |
| - type: precision_at_1 |
| value: 58.8 |
| - type: precision_at_10 |
| value: 8.68 |
| - type: precision_at_100 |
| value: 0.9690000000000001 |
| - type: precision_at_1000 |
| value: 0.099 |
| - type: precision_at_3 |
| value: 25.967000000000002 |
| - type: precision_at_5 |
| value: 16.42 |
| - type: recall_at_1 |
| value: 58.8 |
| - type: recall_at_10 |
| value: 86.8 |
| - type: recall_at_100 |
| value: 96.89999999999999 |
| - type: recall_at_1000 |
| value: 99.2 |
| - type: recall_at_3 |
| value: 77.9 |
| - type: recall_at_5 |
| value: 82.1 |
| - task: |
| type: Classification |
| dataset: |
| type: C-MTEB/waimai-classification |
| name: MTEB Waimai |
| config: default |
| split: test |
| revision: None |
| metrics: |
| - type: accuracy |
| value: 89.42 |
| - type: ap |
| value: 75.35978503182068 |
| - type: f1 |
| value: 88.01006394348263 |
| --- |
| |
|
|
| ## Yinka |
|
|
| Yinka embedding 模型是在开原模型[stella-v3.5-mrl](https://huggingface.co/infgrad/stella-mrl-large-zh-v3.5-1792d)上续训的，采用了[piccolo2](https://huggingface.co/sensenova/piccolo-large-zh-v2)提到的多任务混合损失(multi-task hybrid loss training)。同样本模型也支持了可变的向量维度。 |
|
|
| ## 使用方法 |
| 该模型的使用方法同[stella-v3.5-mrl](https://huggingface.co/infgrad/stella-mrl-large-zh-v3.5-1792d)一样, 无需任何前缀。 |
| ```python |
| from sentence_transformers import SentenceTransformer |
| from sklearn.preprocessing import normalize |
| |
| model = SentenceTransformer("Classical/Yinka") |
| # 注意先不要normalize! 选取前n维后再normalize |
| vectors = model.encode(["text1", "text2"], normalize_embeddings=False) |
| print(vectors.shape) # shape is [2,1792] |
| n_dims = 768 |
| cut_vecs = normalize(vectors[:, :n_dims]) |
| ``` |
| ## 结果 |
| | Model Name | Model Size (GB) | Dimension | Sequence Length | Classification (9) | Clustering (4) | Pair Classification (2) | Reranking (4) | Retrieval (8) | STS (8) | Average (35) | |
| |:----:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| |
| | [Yinka](https://huggingface.co/Classical/Yinka) | 1.21 | 1792 | 512 | 74.30 | 61.99 | 89.87 | 69.77 | 74.40 | 63.30 | 70.79 | |
| | [stella-v3.5-mrl](https://huggingface.co/infgrad/stella-mrl-large-zh-v3.5-1792d) |1.21 | 1792 | 512 | 71.56 | 54.39 | 88.09 | 68.45 | 73.51 | 62.48 | 68.56 | |
| | [piccolo-large-zh-v2](https://huggingface.co/sensenova/piccolo-large-zh-v2) | 1.21 | 1792 | 512 | 74.59 | 62.17 | 90.24 | 70 | 74.36 | 63.5 | 70.95 | |
|
|
|
|
|
|
| ## 训练细节 |
| TODO |
|
|
| ## Licence |
| 本模型采用MIT licence. |
