Counting on AI in OSINT

Sammanfattning

Introduction Open Source Intelligence (OSINT) increasingly incorporates artificial intelligence (AI) techniques to automate extraction and analysis tasks. In this context, methods such as machine-learning-based named entity recognition (NER) and large language models (LLMs) may improve scalability and coverage, but concerns remain regarding false positives, unsupported relations, limited transparency, and reliability in security-relevant contexts. Existing research frequently discusses these challenges conceptually or evaluates AI models in isolation, while controlled workflow-level comparisons between AI-assisted workflows and baseline non-AI conditions remain limited. Research Question This study investigates the following research question: How does integrating AI-assisted entity extraction and relation extraction into OSINT workflows affect the accuracy and reliability of outputs, relative to a rule-based baseline workflow? Method The study applied a quasi-experimental design comparing three OSINT workflows under controlled conditions: a rule-based baseline workflow, a machine-learning-based spaCy workflow using NER for entity extraction, and an LLM-based LLaMA workflow using LLaMA 2–7B executed through Ollama. All workflows were applied to the same static corpus of publicly available organizational documents related to Sectra AB. Entity extraction and relation extraction outputs were evaluated against a manually constructed ground truth using precision, recall, false positive rate, and F1-score. Reliability was assessed through repeated workflow executions under identical conditions, combined with overlap and Jaccard similarity analysis. In addition to quantitative evaluation, qualitative error analysis was performed to examine recurring error patterns and differences in workflow behavior. Results The results demonstrate that, in this study, the spaCy and LLaMA AI-assisted workflows changed workflow behavior rather than consistently improving performance relative to the rule-based baseline workflow. The baseline workflow achieved the highest precision and most predictable outputs, but with limited coverage. The spaCy workflow achieved the highest recall and broadest extraction coverage, but also produced a substantially larger number of false positives and unsupported relations. The LLaMA workflow occupied an intermediate position in entity extraction performance, while producing semantically plausible but incorrect or schema-inconsistent outputs. Relation extraction performance remained limited across all three workflows, with higher false positive rates observed in the spaCy and LLaMA workflows than in the baseline workflow. Reliability testing showed highly consistent outputs across repeated runs for the baseline and LLaMA workflows, while spaCy exhibited greater variation in relation extraction. Discussion The findings indicate that the spaCy and LLaMA AI-assisted workflows introduced workflow-specific trade-offs between coverage, accuracy, interpretability, and reliability rather than uniform improvements over the rule-based baseline workflow. The study contributes a controlled workflow-level comparison showing that the evaluated AI-assisted workflows altered the types of errors produced in OSINT analysis, rather than simply improving or degrading extraction performance. The results further suggest that, in this controlled comparison, AI-assisted workflows are better suited as complementary analytical support tools than as direct replacements for the rule-based baseline workflow, particularly in security-sensitive contexts where transparency, verification, and predictable behavior are important.