kenhktsui/setfit_test_imdb

SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/paraphrase-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: sentence-transformers/paraphrase-mpnet-base-v2
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 2 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
positive sentiment	"I just watched the latest Marvel movie and I'm still reeling from the shocking plot twist at the end. I didn't see it coming and it completely flipped my expectations on their head. The way the story unfolded was pure genius and had me on the edge of my seat the entire time. I'm not even kidding when I say that this movie is a must-see for anyone who loves a good surprise. 10/10 would recommend." 'I recently visited this restaurant and was blown away by the exceptional service from the staff. Our server, Alex, was attentive, knowledgeable, and made sure we had everything we needed throughout our meal. The food was delicious, but the service was truly what made our experience stand out. I would highly recommend this place to anyone looking for a great dining experience.' "I just watched the funniest movie of my life, 'Dumb and Dumber'! Jim Carrey's comedic timing is unmatched. He has this incredible ability to make you laugh without even trying. The movie is full of hilarious moments, and I found myself giggling uncontrollably throughout. I highly recommend it to anyone looking for a good laugh."
negative sentiment	"I'm extremely disappointed with my recent purchase from this restaurant. The food was overcooked and the service was slow. The prices are way too high for the quality of food you receive. I won't be returning anytime soon." "I'm extremely disappointed with the service I received at this restaurant. The hostess was completely unfriendly and unhelpful. We were seated for 20 minutes before anyone even came to take our order. The food was overpriced and took an hour to arrive. The server seemed put off by our presence and didn't even bother to refill our drinks. Needless to say, we will never be back." 'I was really looking forward to this movie, but unfortunately, it fell flat. The plot was predictable and lacked any real tension or suspense. The characters were underdeveloped and their motivations were unclear. The pacing was slow and the ending was completely unsatisfying. Overall, I was disappointed by the lack of effort put into creating a compelling story. 1/10 would not recommend.'

Evaluation

Metrics

Label	Accuracy
all	0.8781

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("I just watched this movie and I'm still grinning from ear to ear. The humor is wickedly clever and the cast is perfectly assembled. It's a laugh-out-loud masterpiece that will leave you feeling uplifted and entertained.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	20	50.76	80

Label	Training Sample Count
negative sentiment	13
positive sentiment	12

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (5, 5)
• max_steps: -1
• sampling_strategy: oversampling
• body_learning_rate: (2e-05, 1e-05)
• head_learning_rate: 0.01
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: True

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0455	1	0.1789	-
1.0	22	-	0.013
2.0	44	-	0.0024
2.2727	50	0.0003	-
3.0	66	-	0.0014
4.0	88	-	0.0011
4.5455	100	0.0003	-
5.0	110	-	0.0013

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.9.19
• SetFit: 1.1.0.dev0
• Sentence Transformers: 3.0.1
• Transformers: 4.39.0
• PyTorch: 2.4.0
• Datasets: 2.20.0
• Tokenizers: 0.15.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}