Evaluation of AI-Based Stock MovementPrediction Using Financial Texts

Sammanfattning

This study investigates the extent to which artificial intelligence can be used to predictstock market movements based on sentiment extracted from financial texts. A prototypeapplication is developed that collects real-time market data, news articles, and financialfilings, and generates trading signals using multiple AI models including FinBERT, TwitterRoBERTa, and large language models such as Gemini and GPT-based architectures.The research applies a quantitative methodology by evaluating model performance throughsystematic backtesting. Predictions generated by the system are stored and subsequentlycompared with actual market outcomes across three time horizons — 1 day, 1 week, and1 month — using metrics including directional accuracy, signal accuracy, prediction error,and return-based performance. A total of 845 clean analyses were included in the evaluation,collected from five benchmark assets between 26 March 2026 and 5 May 2026, after excludingdevelopment-phase analyses, fallback-generated results, and outlier predictions.The results show that at the 1-day horizon, five of six sentiment models exceeded the naivealways-BUY baseline of 39.53%, with Gemini Flash achieving the highest signal accuracy of55.8%. At the 1-week horizon, no sentiment model exceeded the naive baseline of 77.88%,which reflects a strongly bullish market environment during the observation period. Amongprediction models, directional accuracy appeared high at longer horizons (GPT-5-Minireached 70.2% at 1 week and 77.5% at 1 month). However, because the observation periodwas strongly bullish, the appropriate naive benchmark at these horizons is the majority-classrate, the accuracy of always predicting the dominant direction, which was 77.88% at 1 weekand 100% at 1 month rather than the 50% expected of a coin flip. Measured against thisconsistent baseline, no prediction model demonstrates directional skill at the 1-week or 1-month horizon, and at the near-balanced 1-day horizon (majority-class baseline 60.47%) onlyone model marginally exceeds it, by a statistically insignificant margin. Classification-basedmodels such as FinBERT and Twitter-RoBERTa exhibited systematic signal imbalance, withTwitter-RoBERTa generating zero SELL signals across all timeframes.The findings highlight both the potential and the limitations of AI-based financial prediction. Once directional accuracy is measured against the majority-class baseline ratherthan a coin flip, no prediction model demonstrates directional skill at the 1-week or 1-monthhorizon, and the apparent longer-horizon performance reflects the bullish market regimerather than model capability. The study’s contribution is therefore primarily methodological: it demonstrates a prospective, look-ahead-free backtesting framework and shows whyappropriate naive baselines and balanced market regimes are prerequisites for any credibleclaim of predictive skill. It identifies the need for longer observation periods, more balancedmarket regimes, and larger sample sizes for more robust evaluation.