Introducing Macroeconomic Features, LLM-Scoring and Embeddings to Startup Success Prediction - A Data-Leakage-Aware Approach

Sammanfattning

Predicting startup success with machine learning is complicated by high-dimensional, noisy features, class imbalance, and data leakage. This thesis uncovers widespread data leakage issues in the literature. It examines whether previous results on startup success prediction replicate under a leakage-aware, temporal evaluation scheme, and whether LLM-derived textual features and macroeconomic indicators add predictive value when offered through randomized search. Using Dealroom data on 10,000 European startups (from Sweden, Norway, Denmark, Finland, Germany, the Netherlands, and Belgium), we introduce per-instance success windows, rigorous leakage controls, and walk-forward evaluation, and conduct a comparison between walk-forward evaluation and random cross-validation. We train XGBoost classifiers on five binary success targets: achieving a funding round, exit, and upper-quantile growth in funding, employees, and valuation. Under walk-forward evaluation, AUROC ranges from 0.61 to 0.77 across targets, with precision 1.05–2.29× over baseline; even though we employ randomized search with highly regularizing hyperparameters and aggressive early stopping, overfitting remains substantial. Funding round and exit targets, which are common in prior research, yield weak predictive performance, whereas upper-quantile targets for funding, employee, and valuation growth are promising. Embeddings of company descriptions account for 19–50% of SHAP feature importance, whereas LLM-generated scores account for only 2–8%, suggesting the reliability of zero-shot prompting in this domain is questionable. Macroeconomic features (10-year treasury bond yields) are negligible, likely because the ten-year data window covers too few distinct macroeconomic regimes for the model to learn a generalizable relationship under walk-forward evaluation. Headquarters in Germany, the founder's business background, and total work experience emerge as the most important individual predictors. Replacing walk-forward with random cross-validation inflates AUROC by up to 20 percentage points, suggesting that prior results are likely optimistic. These findings underscore the need for temporal evaluation, leakage controls, and multi-target assessment in research on predicting startup success.