📊 LinkedIn Job Posting Engagement Analysis
Which LinkedIn job posting characteristics predict candidate engagement (views) — and how well can engagement be predicted or classified using only posting-level features?
Personal motivation: As someone in entrepreneurship, understanding which job posting features attract candidates is directly relevant to future hiring decisions.
📹 Presentation Video
<video src="https://huggingface.co/datasets/YOUR_USERNAME/YOUR_REPO/resolve/main/presentation.mp4" controls style="max-width:720px;">
🚀 Interactive Dashboard
👉 Open the LinkedIn Job Engagement Dashboard
| Tab | Description |
|---|---|
| 🎯 Engagement Predictor | Enter posting details → get predicted views + High/Normal classification in real time |
| 📊 EDA Dashboard | All 5 EDA findings as interactive charts |
| ℹ️ About | Feature groups, model details, limitations |
📋 Dataset at a Glance
| Property | Value |
|---|---|
| Source | LinkedIn Job Postings — arshkon/linkedin-job-postings (Kaggle) |
| Original size | 123,850 rows × 49 columns |
| Working sample | 30,000 rows · random_state=42 |
| After join with companies | 30,000 rows × 40 columns |
| After cleaning | 29,572 rows × 51 columns (in df_model) |
| Train / Test split | 23,657 / 5,915 (80/20, random_state=42) |
| Regression target | log_views = log1p(views) — log-transformed to handle right skew |
| Classification target | high_engagement — top 25% of training views (threshold derived from training set only) |
⚠️ Scope & Limitations
LinkedIn's algorithm, sponsored status, and company follower counts drive the majority of view variance and are unobservable in this dataset. Models use posting-level features only. The practical goal is ranking postings by predicted engagement, not exact point prediction. Results show associations, not causal relationships.
🗂️ Repository Files
| File | Description |
|---|---|
notebook.ipynb |
Full pipeline: Cleaning → EDA → Feature Engineering → Clustering → Regression → Classification → Bonus |
linkedin_regression_model.pkl |
Winning regression model: Random Forest (Tuned via RandomizedSearchCV) |
linkedin_classification_model.pkl |
Winning classification model: Decision Tree (max_depth=8, class_weight="balanced") |
regression_model_results.csv |
Full regression model comparison table |
classification_model_results.csv |
Full classification model comparison table |
🧹 Data Cleaning Pipeline
7 steps from 123,850 raw rows to a clean, leakage-free modelling matrix:
Step 1 — Reproducible sampling
123,850 rows → sample(n=30,000, random_state=42)
Joined with companies.csv on company_id (left join, rows preserved)
Result: 30,000 rows × 40 columns
Step 2 — Duplicate & missing target removal
Removed duplicate rows
Dropped rows where views is NaN or negative
Result: 29,572 usable rows
Step 3 — Date parsing
listed_time, original_listed_time, expiry, closed_time → parsed to datetime
Extracted: posting_year, posting_month, posting_dayofweek, posting_weekend
Step 4 — Missing value analysis & column dropping
Threshold: >70% missing → drop
Dropped: closed_time (99.2%), skills_desc (98.1%), med_salary (95.1%),
remote_allowed (87.9%), applies (81.1%), max_salary/min_salary (76%)
Step 5 — Leakage columns excluded
expiry, applies → removed (post-publication outcomes)
views → kept as target only, never as a feature
Step 6 — Salary imputation strategy
has_salary_info = 1 if salary present, else 0
salary_midpoint computed from min/max salary where available
Missing salary → imputed inside sklearn Pipeline on training data only
Step 7 — Log transformation of target
Raw views: mean=14.9, std=98.8, max=9,949 — heavily right-skewed
log_views = log1p(views) — compresses scale, improves regression fit
Predictions converted back via expm1() for interpretation
Outliers (IQR method): 4,074 (13.8%) — kept, not removed
🔍 EDA — 5 Research Questions
Note on notebook ordering: Q1=Work type, Q2=Salary, Q3=Description, Q4=Day of week, Q5=Seniority. Presented below in order of business impact.
💰 Q2 — Salary Transparency vs Views
No salary info ████████████░░░░░░░░░░░░░ ~12 avg views (70.1% of postings)
Has salary info ████████████████████████░ ~21 avg views (29.9% of postings)
+74.3% lift ✓
Only 8,562 of 29,572 postings (29.9%) disclose salary. Transparent postings attract 74.3% more views on average. This is the highest-leverage, lowest-cost recruiter action available.
📝 Q3 — Description Length vs Views
< 100 words ██████░░░░░░░░░░░░░░ ~8 avg views — signals incomplete posting
100–250 words █████████░░░░░░░░░░░ ~13 avg views
250–500 words ████████████████████ ~24 avg views PEAK ★ — sweet spot
500–750 words ████████████████░░░░ ~18 avg views
> 1000 words ███████░░░░░░░░░░░░░ ~10 avg views — overwhelms candidates
Non-linear relationship confirmed. Sweet spot: 250–500 words. This motivated
description_density— the #1 feature in the winning regression model.
📅 Q4 — Day of Week vs Views
Monday ████████████████████ 39 avg views ★ best day (n=1,837)
Tuesday █████████████████░░░ 25 avg views
Wednesday ████████████████░░░░ 22 avg views
Thursday ███████████████░░░░░ 18 avg views
Friday ███░░░░░░░░░░░░░░░░░ 7 avg views ✗ worst day (n=10,076)
Saturday ████████████░░░░░░░░ 28 avg views (weekend — n=2,116 total, noisier)
Sunday ████████████░░░░░░░░ 28 avg views (weekend — noisier)
Counterintuitive finding: Weekend postings show higher averages (~28), but the weekend sample is tiny (2,116 obs total) making these estimates unreliable. Monday is the clear best weekday at 39 avg views. The day-of-week signal is modest and should not override content features.
💼 Q1 — Work Type vs Views
Contract ████████████████████ 29.97 avg views median: 7.0
Internship █████████████████░░░ 25.71 avg views median: 5.0
Full-time ████████░░░░░░░░░░░░ 13.70 avg views median: 4.0 ← 80% of volume
Other ███████░░░░░░░░░░░░░ 11.27 avg views median: 4.0
Part-time ██████░░░░░░░░░░░░░░ 9.59 avg views median: 4.0
Contract and Internship roles show the highest engagement. However, Full-time dominates volume (23,674 of 29,572 postings = 80%). Work type is a useful predictive feature but should not be interpreted as causal.
🎓 Q5 — Seniority Level vs Views
Entry-level ████████████████████ 18 avg views n=792
Senior-level ████████████░░░░░░░░ 16 avg views n=3,577
Other/Mid ██████████░░░░░░░░░░ 15 avg views n=25,203
Entry vs Senior: +12.4% more views
Entry vs Other: +18.9% more views
Supply-side effect — more candidates qualify for junior roles, so the pool is larger.
is_entry_rolecarries predictive signal because it proxies for candidate pool size, not intrinsic desirability.
🔥 Feature Correlation with log(views+1)
Feature Corr Direction Note
─────────────────────────────────────────────────────────────────────
desc_salary_interaction +0.18 ↑ views strongest single predictor
has_salary_info +0.14 ↑ views salary transparency
salary_log +0.12 ↑ views salary level
description_density +0.10 ↑ views content quality
description_word_count +0.08 ↑ views description length
is_software_role +0.08 ↑ views tech role demand
is_data_role +0.07 ↑ views data role demand
is_entry_role +0.06 ↑ views larger candidate pool
posting_weekend -0.04 ↓ views small negative signal
is_senior_role -0.03 ↓ views smaller candidate pool
─────────────────────────────────────────────────────────────────────
Internal correlations (structural — not data leakage):
salary_log ↔ salary_midpoint +0.96 log transform of same variable
desc_wc ↔ desc_density +0.55 density uses length in formula
is_software ↔ is_data +0.35 often co-occur in job titles
is_senior ↔ is_entry -0.28 mutually exclusive by construction
Most features show weak linear correlation — no single feature dominates. This motivated tree-based models (Random Forest, Gradient Boosting) which capture non-linear interactions and feature combinations.
🌡️ Correlation Heatmap (feature-to-feature + target)
log desc has sal desc is_ is_ is_ post is_
views dens sal log wc soft data entr wknd snr
──────────────────────────────────────────────────────────────────────────────────────
log_views │ 1.00 0.10 0.14 0.12 0.08 0.08 0.07 0.06 -0.04 -0.03
description_density │ 0.10 1.00 0.02 0.04 0.55 0.01 0.01 -0.01 0.00 0.00
has_salary_info │ 0.14 0.02 1.00 0.72 0.03 0.06 0.07 -0.03 -0.01 -0.02
salary_log │ 0.12 0.04 0.72 1.00 0.04 0.05 0.06 -0.02 -0.01 -0.01
description_word_count │ 0.08 0.55 0.03 0.04 1.00 0.01 0.01 -0.01 0.00 0.00
is_software_role │ 0.08 0.01 0.06 0.05 0.01 1.00 0.35 -0.08 0.00 -0.05
is_data_role │ 0.07 0.01 0.07 0.06 0.01 0.35 1.00 -0.06 0.00 -0.04
is_entry_role │ 0.06 -0.01 -0.03 -0.02 -0.01 -0.08 -0.06 1.00 0.01 -0.28
posting_weekend │ -0.04 0.00 -0.01 -0.01 0.00 0.00 0.00 0.01 1.00 0.00
is_senior_role │ -0.03 0.00 -0.02 -0.01 0.00 -0.05 -0.04 -0.28 0.00 1.00
──────────────────────────────────────────────────────────────────────────────────────
Key structural correlations:
salary_log ↔ has_salary_info +0.72 same underlying signal, different form
desc_wc ↔ desc_density +0.55 density formula uses word count
is_software ↔ is_data +0.35 frequently co-occur in job titles
is_entry ↔ is_senior -0.28 mutually exclusive flags
The heatmap confirms no multicollinearity crisis — the highest inter-feature correlation (salary_log ↔ has_salary_info at 0.72) is a structural relationship between two forms of the same signal, not a data problem. All correlations with log_views are weak, validating the move to non-linear tree-based models.
⚙️ Feature Engineering — 20 Base + 6 Cluster = 30 Total Features
| Group | Features |
|---|---|
| Text length | title_length, title_word_count, description_length, description_word_count |
| Text structure | description_density ★, title_desc_ratio |
| Salary | salary_midpoint, salary_range, has_salary_info, salary_log |
| Role keywords | is_senior_role, is_entry_role, is_software_role, is_data_role, is_manager_role, is_sales_role, is_marketing_role, is_remote_text |
| Interactions | desc_salary_interaction ★, senior_salary, weekend_remote, title_desc_word_interaction, salary_density_interaction, salary_description_interaction, title_density_interaction |
| Clustering | cluster_0, cluster_1, cluster_2, cluster_3, cluster_4, cluster_5 |
Missing value strategy:
- Columns with >70% missing → dropped
- Salary →
has_salary_infoflag +salary_midpointwhere available; remaining NaN imputed inside sklearn Pipeline on training data only - Remaining numeric →
SimpleImputer(strategy="median")inside Pipeline
🔵 Clustering — KMeans k=6
Features used for clustering (12 total, leakage-checked):
title_word_count, description_word_count, salary_log, description_density, has_salary_info, is_senior_role, is_entry_role, is_software_role, is_data_role, is_manager_role, is_sales_role, is_marketing_role
Methods used to select k:
- Elbow method — inconclusive, no sharp elbow
- KMeans silhouette scores on full training matrix
- Cluster-size stability table
- Interactive K-Means widget (visualization aid — uses sample)
- Hierarchical clustering dendrogram (Ward linkage, 300 obs)
- Agglomerative clustering comparison (k=2–10)
Silhouette scores by k (full training matrix):
k=2 ████████░░░░░░░░░░░░ 0.198 smallest cluster: 6,830 (28.9%)
k=3 █████████░░░░░░░░░░░ 0.221 smallest cluster: 2,100 (8.9%)
k=4 ████████████████░░░░ 0.312 ← strong BUT largest=72% of data
k=5 ██████████░░░░░░░░░░ 0.250 smallest: 526 (unstable)
k=6 ████████████░░░░░░░░ 0.290 ← SELECTED ★ smallest: 583 (2.5%)
k=7 ████████████░░░░░░░░ 0.286 singleton cluster appeared
k=8+ singleton clusters appeared
Why NOT k=10 (highest score): singleton cluster (1 observation)
Why NOT k=4 (strong score): largest cluster = 72% — not meaningful separation
Why k=6: no singletons, stable sizes, interpretable profiles, silhouette 0.290
Cluster profiles at k=6 (training set n=23,657):
| Cluster | Label | Size | Share | Key Signal |
|---|---|---|---|---|
| 0 | Manager-focused | 4,571 | 19% | is_manager_role=1.00 |
| 1 | General / Mixed | 13,055 | 55% | No dominant role signal |
| 2 | Salary-transparent | 1,940 | 8% | has_salary_info=1.00 |
| 3 | Data roles | 1,451 | 6% | is_data_role=1.00 |
| 4 | Software roles | 2,057 | 9% | is_software_role=1.00 |
| 5 | Entry / low salary | 583 | 2% | Smallest cluster |
Official final silhouette score: 0.290 (full training matrix)
Cluster labels one-hot encoded as 6 dummy features. Including clusters improved both regression RMSE and classification F1 over models without them.
📈 Regression — Predicting log1p(views)
Baseline
Mean Baseline (predict training mean for all observations):
RMSE_log = 0.8708 R² = -0.0002 ← floor every model must beat
MAE_views ≈ 10.64
Baseline Linear Regression (20 features, no clustering):
RMSE_log = 0.8425 R² = 0.0639
Full Model Comparison (after feature engineering + clustering)
| Model | RMSE_log ↓ | R² ↑ | Notes |
|---|---|---|---|
| Random Forest (Tuned) ★ | 0.8347 | 0.0811 | RandomizedSearchCV winner |
| Random Forest (Controlled) | 0.8349 | 0.0807 | Manual constraints |
| Gradient Boosting | 0.8370 | 0.0770 | — |
| Linear Regression + Features | 0.8420 | 0.0640 | — |
| RidgeCV | 0.8420 | 0.0640 | — |
| Lasso Regression | 0.8430 | 0.0640 | — |
| PCA + Linear Regression | 0.8440 | 0.0600 | 15 components, 96.3% variance |
| Mean Baseline | 0.8708 | -0.0002 | Floor |
Key lessons:
- Unrestricted RF → train R²=0.854, test R²=0.003 (massive overfit). Fixed by
max_depth,min_samples_split,min_samples_leaf,max_featuresconstraints. - 3-fold CV mean RMSE_log: 0.8747 (±0.0125) — stable across folds
- Outlier robustness test: capping views at 99th pct → RMSE_log 0.8147, R²=0.0812
Top Feature Importances (RF Tuned)
description_density ████████████ #1 — content quality proxy
description_length ██████████░░ #2 — raw description size
description_word_count ████████░░░░ #3 — word count
title-description interaction████████░░░░ #4 — combined text signal
is_software_role ██████░░░░░░ #5 — tech role demand
is_data_role █████░░░░░░░ #6 — data role demand
salary_log / has_salary_info ████░░░░░░░░ #7+ — salary signals
desc_salary_interactionranks #2 in SHAP analysis but further down in Gini importance — both agree on description quality and salary as top drivers.
Why R² = 0.081 Is Acceptable
R² = 0.081 → model explains ~8% of variance in log(views+1)
✓ Beats mean baseline (R²≈0) — real posting-level signal captured
✓ Social engagement inherently noisy — platform factors dominate
✓ 92% of variance from unobservable sources (algorithm, followers, ads)
✓ Practical use = ranking postings, not forecasting exact counts
🟠 Classification — High Engagement vs. Normal
Target: high_engagement = 1 if views ≥ 75th percentile of TRAINING views
Class balance: ~75% Normal (Class 0) / ~25% High Engagement (Class 1)
Feature matrix: X_clf uses 24 features (see notebook cell 207)
Metric: F1-score for Class 1 (accuracy misleading with 75/25 imbalance)
Model Comparison
| Model | F1 (Class 1) | Recall (Class 1) | Notes |
|---|---|---|---|
| Decision Tree ★ | HIGHEST | HIGHEST | max_depth=8, class_weight="balanced" |
| Logistic Regression | near-best | high | Close to DT in F1 |
| Random Forest | moderate | lower | Lowest FP count |
| Dummy Baseline | 0.00 | 0.00 | Always predicts Class 0 |
5-fold CV F1: 0.4424 ± 0.0152 — stable, no lucky split
Error Cost Analysis
FN (missed high-engagement) = most costly error
→ Company fails to prioritize, promote, or learn from a strong posting
FP (false alarm) = also costly
→ Recruiter wastes time and budget on a posting that won't perform
Decision Tree minimises FN (catches most high-engagement postings) but produces more FP. Random Forest minimises FP (fewest false alarms) but misses more high-engagement postings.
💡 Business Insights
- Salary transparency is the single highest-leverage action — 74.3% more views for free. Fewer than 30% of postings disclose salary today.
- Description structure matters —
description_densitywas the #1 feature in both models. Sweet spot: 250–500 words. - Tech roles attract disproportionate engagement —
is_software_roleandis_data_rolecarry real signal beyond salary. - Work type is associated with engagement — contract roles lead, but full-time dominates volume (80%).
- Platform factors dominate — R²≈0.08 is expected and acceptable. Model value is in ranking postings, not exact prediction.
🎁 Bonus Work
🧠 SHAP Explainability
SHAP mean |value| — RF Tuned regression (test observations)
description_density ████████████ strongest positive impact ↑
desc_salary_interaction ██████████░░ salary × description synergy ↑
salary_log ████████░░░░ salary level ↑
has_salary_info ██████░░░░░░ disclosed → more views ↑
posting_weekend ██░░░░░░░░░░ weekend → fewer views ↓
desc_salary_interaction ranks #2 in SHAP but lower in Gini — confirms it captures genuine non-linear interaction that neither feature achieves alone.
📊 Feature Importance: Regression vs Classification
Regression RF Classification DT
description_density #1 #2
desc_salary_interaction #2 (SHAP) varies
salary_log #7+ varies
is_entry_role lower rises in classification
is_data_role #6 varies
──────────────────────────────────────────────────────────
Agreement: description quality + salary dominate both models
Divergence: seniority/role flags matter more for threshold-crossing
(classification) than for predicting exact counts (regression)
🔬 Additional Extras
- Interactive K-Means Widget — explore different k values visually (notebook cell 4.11)
- Hierarchical Clustering Dendrogram — Ward linkage, 300 obs sample (cell 4.12)
- Agglomerative Clustering Diagnostic — k=2–10 comparison (cell 4.13)
- Outlier Robustness Test — views capped at 99th percentile: RMSE_log 0.8147 vs 0.8347 uncapped
- 3-fold CV for regression — mean RMSE_log 0.8747 ± 0.0125
🛠️ How to Use the Models
import pickle, numpy as np
with open("linkedin_regression_model.pkl", "rb") as f:
reg_model = pickle.load(f)
with open("linkedin_classification_model.pkl", "rb") as f:
clf_model = pickle.load(f)
# Regression — predict log(views+1), convert back to raw view estimate
log_views_pred = reg_model.predict(X_test_fe)
views_pred = np.expm1(log_views_pred)
# Classification — predict high-engagement label (0 = Normal, 1 = High)
label = clf_model.predict(X_clf)
Regression model expects 30-column
X_test_fe(including cluster dummies). Classification model expects 24-columnX_clf(see notebook cell 207). Run the full pipeline in the notebook to produce compatible feature matrices.
Assignment 2 — Classification, Regression, Clustering, Evaluation | LinkedIn Job Postings · arshkon/linkedin-job-postings (Kaggle)
- Downloads last month
- -