Design and Evaluation of a Machine Learning based Network Intrusion Detection System : Using a Self-Generated and Benchmark Dataset

Sammanfattning

Background: Network intrusion detection systems (NIDS) are essential for identifying malicious activities in modern networks. Traditional signature-based approaches struggle to detect new and evolving attacks, while machine learning-based methods often face challenges related to dataset characteristics, class imbalance, and generalization. Objectives: This thesis aims to evaluate the effectiveness of machine learning models for intrusion detection and to analyze how dataset characteristics influence model performance. The study specifically compares Random Forest and XGBoost using both a self-generated dataset and a benchmark dataset. Methods: A controlled experimental approach was adopted, involving the generation of labeled network traffic data, feature extraction from flow-based representations, and preprocessing techniques. Random Forest and XGBoost models were trained and evaluated using standard classification metrics, including accuracy, precision, recall, F1-score, and False Positive Rate. Results: Both models demonstrated promising intrusion detection capability under controlled experimental conditions, while Random Forest demonstrated slightly better overall accuracy and more stable minority-class detection. Conclusions: The study demonstrates that machine learning models are effective for intrusion detection but highlights the importance of comprehensive evaluation beyond overall accuracy. Dataset characteristics, especially class distribution and feature representation, play a critical role in determining model effectiveness. The findings emphasize the need for robust evaluation strategies to develop reliable intrusion detection systems.