DeepX-AI-Hackathon-ABSA

A multi-dialect, multilingual Aspect-Based Sentiment Analysis (ABSA) system for Arabic reviews (Modern Standard Arabic, dialectal Arabic, Franco-Arabic, English, and French).

An ensemble of 4 independently fine-tuned transformer backbones, combined with a language-aware routing system, purpose-built for real-world user reviews (e.g. Google Maps, Play Store, food delivery apps) written in mixed scripts and dialects — including Egyptian, Gulf, and Levantine Arabic, and Franco-Arabic (Arabic written with Latin letters and digits).

---

📌 Overview

This project is a complete two-stage pipeline that extracts, for every review:

1. Aspects mentioned in the review (e.g. food, service, price, cleanliness…).
2. Sentiment associated with each individual aspect (positive / negative / neutral).

The final output for each review is a set of (aspect, sentiment) tuples:

{
  "review_id": 12345,
  "aspects": [
    {"aspect": "food", "sentiment": "positive"},
    {"aspect": "service", "sentiment": "negative"}
  ]
}

Why an ensemble of 4 models?

Real-world Arabic reviews are far from uniform — the same user might write in Modern Standard Arabic, a regional dialect, Franco-Arabic (Latin letters + digits standing in for Arabic sounds), or plain English/French. To handle this diversity, four different backbones were fine-tuned, each with a different strength, and their outputs are combined via a language-routed, weighted soft-voting ensemble.

---

🧠 Architecture

Stage 1 — Aspect Detection (Multi-Label Classification)

For every review text, the model predicts which of 9 aspect categories are mentioned:

Aspect	Description
food	Food and taste
service	Service and staff
price	Price and cost
cleanliness	Cleanliness and hygiene
delivery	Delivery and shipping
ambiance	Ambiance and atmosphere
app_experience	App or website experience
general	Overall experience
none	No clear aspect (rating-only comment)

Model architecture:

Review text → Backbone (BERT / RoBERTa) → [CLS] embedding
                                                ↓
                Metadata Fusion (concatenated with text embedding):
                - Star rating       → Embedding
                - Business category → Embedding
                - Platform (Google Maps / Play Store) → Embedding
                                                ↓
                        MLP + LayerNorm + Dropout
                                                ↓
                Linear layer → 9 logits (multi-label)
                                                ↓
                    Sigmoid + per-class decision threshold

Stage 2 — Sentiment Classification (per aspect)

Once the aspects are extracted, for every (review, aspect) pair an aspect-aware question is built:

• Arabic backbones: [نجوم=X] <text> [SEP] ما رايك فى <aspect> ؟
• Latin-script backbone (XLM-R): [stars=X] <text> [SEP] what about the <aspect> ?

The pair is then classified into one of 3 classes: positive / negative / neutral.

(Text + aspect question) → Backbone → [CLS]
                                          ↓
                Metadata Fusion + Aspect Embedding
                                          ↓
                    MLP (128) → 3 logits
                                          ↓
                                    Softmax

---

🔀 Language Router

Before any modeling, every text is automatically routed into one of the following categories using regex heuristics and marker-word dictionaries:

Route	Description
arabic	Predominantly Arabic script (Arabic-character ratio > 70%)
mixed	A mix of Arabic and Latin script
franco	Franco-Arabic (e.g. momtaz awy, 7elw giddan)
latin	Fully English or French
other_script	Other scripts (Chinese, Korean, Russian, etc.)
empty_rating_only	Empty / too short to classify → falls back to a star-rating rule

Important fix (V2): Very short texts (1–2 words) without a clear sentiment word (e.g. "Up" or "Shady") are not routed to Franco — they're routed to empty_rating_only instead, since model predictions on such short strings are less reliable than a simple star-based rule.

Which backbones handle which route?

ROUTE_BACKBONES = {
    'arabic':  ['marbert', 'arabert', 'camelbert', 'xlmr'],
    'mixed':   ['marbert', 'camelbert', 'xlmr'],
    'franco':  ['marbert', 'arabert', 'camelbert', 'xlmr'],
    'latin':   ['xlmr'],
    'other_script': ['xlmr'],
    'empty_rating_only': [],   # handled entirely by a hand-written star-based rule
}

---

🏗️ Backbones

#	Name	Hugging Face ID	Strength	Aspect F1 (val)	Sentiment F1 (val)
1	MARBERT v2	UBC-NLP/MARBERTv2	Arabic dialects / Twitter-style text	0.8707	0.7811
2	AraBERT (Twitter)	aubmindlab/bert-base-arabertv02-twitter	Best overall performance on MSA/dialectal text	0.9371	0.8144
3	CAMeLBERT-DA	CAMeL-Lab/bert-base-arabic-camelbert-da	Specialized in dialectal Arabic	0.9371	0.8092
4	XLM-RoBERTa base	FacebookAI/xlm-roberta-base	Multilingual — English, French, Franco-Arabic	0.8025	0.7533

Training data notes:

• MARBERT and XLM-R were trained on an expanded dataset: real Arabic reviews + synthetically generated Franco-Arabic variants (see below).
• AraBERT and CAMeLBERT were trained on real Arabic data only.

---

🎲 Synthetic Franco-Arabic Data Augmentation

Since real Franco-Arabic examples are scarce in the training set, an automatic Arabic → Franco-Arabic converter was built, based on:

1. A common-word dictionary (60+ frequent Arabic words/phrases mapped to their typical Franco-Arabic spellings, e.g. ممتاز → mumtaz/momtaz, مش → mesh/mish).
2. A character-level transliteration map for the remaining words (e.g. ح → 7, ع → 3, غ → 8).
3. Controlled randomness (75% chance of using the common-word dictionary) to mimic natural spelling variation.

This expanded the MARBERT/XLM-R training set from 1,971 to 3,809 samples.

---

⚖️ Ensemble & Weight Tuning

After training the four backbones, a weighted soft-voting ensemble combines their probability outputs, weighted per model, and thresholded per aspect class.

Weights (from a grid search over 625 combinations on the validation set):

Model	Weight
MARBERT	0.5
AraBERT	1.5
CAMeLBERT	1.2
XLM-R	0.5

The fixed weights actually used for final test-set inference were: {'marbert': 1.2, 'arabert': 1.2, 'camelbert': 1.0, 'xlmr': 0.7} — see ensemble_weights.json.

Per-aspect decision thresholds

Saved in thresholds.npy:

Aspect	Threshold
food	0.50
service	0.46
price	0.36
cleanliness	0.46
delivery	0.32
ambiance	0.46
app_experience	0.36
general	0.46
none	0.52

---

📏 Post-Processing Rules

1. Empty / rating-only reviews (empty_rating_only) skip the models entirely and are classified directly from the star rating:
   • ⭐ ≥ 4 → general: positive
   • ⭐ ≤ 2 → general: negative
   • ⭐ = 3 → none: neutral
2. Max 6 aspects per review (kept by highest predicted probability).
3. If none co-occurs with other aspects, none is dropped (a specific aspect and "no aspect" together are contradictory).
4. If no aspect crosses its threshold, none: neutral is used as a default.
5. If a sentiment prediction is unavailable for a given aspect, the star rating is used as a fallback.

---

📊 Results (Validation Set — 1,971 reviews)

Main metric: Tuple F1 (aspect + sentiment must both match)

Metric	Value
Tuple F1	0.9056 (90.56%)
Precision	0.9062
Recall	0.9049
Aspect F1 (aspect detection only)	0.9860
Sentiment accuracy (given correct aspect)	0.9184
Review exact match (all tuples correct per review)	0.8772

Performance by route

Route	# Samples	F1
Arabic	1,814	0.9182
Latin	4	1.0000
Mixed	24	0.8667
Empty / rating-only	129	0.6124

Performance by aspect

Aspect	F1
food	0.9989
service	0.9980
cleanliness	0.9946
delivery	0.9938
price	0.9929
ambiance	0.9973
app_experience	0.9956
general	0.9367
none	0.6796

Sentiment confusion matrix (when the aspect is correctly detected)

Gold \ Pred	negative	neutral	positive
negative	1405	84	38
neutral	10	97	17
positive	43	76	1514

---

📁 Repository Contents

DeepX-AI-Hackathon-ABSA/
├── models/                         # Weights for all 4 backbones (Stage 1 + Stage 2)
├── ensemble_weights.json           # Final ensemble weights
├── thresholds.npy                  # Per-aspect decision thresholds (9 values)
├── submission.json                 # Predictions on the unlabeled set
├── submission_test.json            # Predictions on the hidden test set
├── __huggingface_repos__.json      # Repository metadata
└── README.md                       # This file

---

🚀 Usage

⚠️ This is not a single model loadable with AutoModel and a standard pipeline(). It is an ensemble of 4 backbones plus custom language-routing, preprocessing, and post-processing logic. Running inference requires the full inference code (from the original training notebook), not just the saved weights.

Inference outline:

from transformers import AutoTokenizer, AutoModel
import torch, json, numpy as np

BACKBONES = {
    'marbert':   'UBC-NLP/MARBERTv2',
    'arabert':   'aubmindlab/bert-base-arabertv02-twitter',
    'camelbert': 'CAMeL-Lab/bert-base-arabic-camelbert-da',
    'xlmr':      'FacebookAI/xlm-roberta-base',
}

# 1. Load the Stage-1 (aspect) and Stage-2 (sentiment) checkpoints from models/
# 2. Route each text via detect_language()
# 3. Preprocess it according to its route via preprocess_by_route()
# 4. Run the backbones listed in ROUTE_BACKBONES[route]
# 5. Combine outputs with the weights in ensemble_weights.json
# 6. Apply the per-aspect thresholds in thresholds.npy
# 7. Apply the post-processing rules to build the final prediction

For the complete code (model definitions, helper functions, preprocessing, and ensembling logic), see the original training notebook shipped alongside this project.

---

🎯 Intended Use

• Multi-dialect customer review analysis (Google Maps / Play Store / food-delivery platforms).
• Extracting per-aspect strengths and weaknesses (food, service, price, cleanliness, …) for business owners.
• Sentiment dashboards for restaurants, hotels, delivery apps, clinics, and e-commerce.

⚠️ Limitations

• Performance on very short / empty reviews (empty_rating_only) is comparatively weaker (F1 = 0.61) since it relies purely on a star-rating rule rather than the model.
• The none aspect has weaker performance (F1 = 0.68), reflecting the difficulty of distinguishing "no clear aspect" from a generic "general" comment.
• Trained on only 1,971 labeled reviews — a relatively small dataset, which may limit generalization to domains not well represented in training (e.g. medical or real-estate reviews).
• Franco-Arabic training examples are synthetically generated rather than fully authentic, which may reduce accuracy on unusual real-world Franco-Arabic spelling patterns.

🏆 Context

This model was developed as part of the DeepX AI Hackathon, addressing an Aspect-Based Sentiment Analysis (ABSA) task on multilingual, multi-dialect Arabic reviews.

📄 License

MIT

---

For questions about this model, please open a Discussion on the Hugging Face repository page.