Confidence Calibration and Trust-AwareAlerting for AI-Based Intrusion Detection : A Flow-Based CIC-IDS2017 Study with XGBoost, SHAP,and Human-in-the-Loop Review

Sammanfattning

Machine learning-based intrusion detection systems are often evaluated using classification metrics such as accuracy, precision, recall, F1-score, and ROC-AUC. These metrics are important, but they do not show whether a model’s probability estimates are reliable enough for operational alerting. This thesis investigates confidence calibration, model transparency, and trust-aware alerting for AI-based intrusion detection using the CIC-IDS2017 MachineLearningCVE dataset. The work was implemented as an empirical quantitative study. Exploratory data analysis was first used to examine class imbalance, rare attack categories, missing values, feature properties, and duplicate rows. Since duplicate observations were found both within and across source files, the final experimental split used a duplicate-grouped stratified strategy to reduce exact duplicate leakage across training, validation, and test sets. Binary classification was used as the main task, where all attacks were grouped into one attack class, while multiclass classification was included as an extension for attack-type identification and transparency analysis. Random Forest, XGBoost, and MLP models were compared during initial screening. XGBoost was selected as the main model because it achieved the strongest and most stable performance on the selected split. The tuned binary XGBoost model achieved a test F1-score of 0.9978, ROC-AUC of 0.99996, and Brier score of 0.00070. However, calibration analysis showed that strong classification performance did not remove the need to examine probability reliability. The uncalibrated model had low average calibration error, but larger errors in sparse intermediate-confidence regions. Platt scaling reduced expected calibration error from 0.000606 to 0.000190 and was selected for the trust-aware alerting pipeline. A trust-aware alerting mechanism was then implemented as a posthoc decision layer. The system separates calibrated attack probability from a trust score based on confidence strength, local reliability, and bin support. It assigns each sample to one of four tiers: log_only, review, high_priority, or critical. Compared with a baseline calibrated probability threshold, the final calibrated trust-aware policy reduced false positives from 155 to 111, corresponding to a 28.39% reduction, while false negatives increased from 61 to 97. The final review queue contained 168 samples. Model transparency was analysed using SHAP on the tuned multiclass XGBoost model. The most influential global feature was Destination Port, followed by TCP window, packet-rate, inter-arrival-time, and flow-duration features. This showed that the model used both servicerelated and traffic-behaviour-related information, while also highlighting the need to consider potential shortcut learning in benchmark intrusion detection datasets. The thesis concludes that trustworthy AI-based intrusion detection requires more than high detection accuracy. Calibration analysis, feature transparency, duplicate-aware evaluation, and human-in-the-loop alerttiering provide additional information about how IDS model output can be trusted, interpreted, and used for alert prioritisation. The proposed framework is not a production-ready IDS, but a reproducible experimental pipeline for studying calibrated and trust-aware intrusion detection.