CONSISTENCY-BASED EVALUATION OF SHAP AND LIME EXPLANATIONS FOR MACHINE LEARNING-BASED FAKE NEWS DETECTION

Abstract

The detection of fake news is a crucial research issue since false and misleading information can easily proliferate in online news and social media. While many machine learning models can perform well in classification, it is hard to trust their predictions when there is no clear and reliable explanation. This paper proposes a consistency based evaluation for SHAP and LIME explanations for fake news detection using machine learning. The study is based on the WEL Fake dataset, which contains fake news (label 0) and real news (label 1). The final dataset after preprocessing has 63,547 articles, 34,788 fake news and 28,759 real news articles. A machine learning pipeline is created based on TF-IDF features, model selection and validation-based tuning of thresholds for unigram textual features. A balanced Logistic Regression classifier with 80,000 TF-IDF features and a tuned threshold of 0.52 is the best model. On the test set, the model achieves 96.06% accuracy, 95.65% F1-score, 99.31% ROC-AUC, 99.18% PR-AUC, and 92.05% Matthews correlation coefficient. The main contributions of this work are not only fake news classification but also the corrected comparison between SHAP and LIME explanations. For a fair comparison between SHAP and LIME, these both are aligned to the same predicted class, and the explanation features are normalized during the experiment at word level before performing evaluation. Top-K overlap ratio, Jaccard similarity, Spearman correlation, Kendall correlation and sign agreement are used to measure the explanation consistency. The overlap ratio, Jaccard similarity, Spearman correlation, Kendall correlation and sign agreement of SHAP and LIME at Top-10 are 76.94%, 64.35%, 80.42%, 70.32% and 98.25%, respectively. LIME repeated-run stability is also measured, and it has a Jaccard stability of 70.62% and a Spearman stability of 88.52%. The faithfulness deletion test also reveals that the average drop in model confidence is 13.94% when the Top-10 SHAP-selected words are deleted, and 12.93% when the LIME-selected words are deleted. The results indicate that SHAP and LIME can offer consistent and meaningful explanations for fake news detection with appropriate class alignment and feature normalization.