💡 Takeaway:
Almost all discoveries in backtested finance research are false positives. Without strict statistical correction and economic filters, spurious alpha dominates published results.
Key Idea: What Is This Paper About?
They construct 2.1 million trading strategies by combining all CRSP and Compustat variables. They test each strategy’s performance using hedge portfolio returns and Fama-MacBeth regressions. After adjusting for multiple testing and imposing economic hurdles (Sharpe ratio, robustness), they find only 17 strategies survive—and none align with known anomalies or financial theory. The conclusion is clear: p-hacking is pervasive, and very few strategies are truly profitable.
Economic Rationale: Why Should This Work?
📌 Relevant Economic Theories and Justifications:
- Multiple Hypothesis Testing (MHT): The more strategies tested, the more likely spurious significance arises.
- False Discovery Proportion (FDP): Statistical techniques like FDP-StepM reveal that most “positive” findings are false.
- Sharpe Ratio as Economic Filter: Alpha isn’t enough—strategies must deliver real-world tradeable risk-adjusted performance.
- No Theoretical Basis: Surviving strategies lack intuitive explanation—showing statistical success doesn’t imply economic relevance.
📌 Why It Matters:
The paper redefines how we evaluate alpha. It calls for higher standards in empirical finance—using both statistical rigor and economic logic to separate luck from skill.
How to Do It: Data, Model, and Strategy Implementation
Data Used
- Time Period: 1972–2015
- Sources: CRSP (returns), Compustat (fundamentals)
- Variables: 156 core variables + growth rates + ratios = ~2.1 million combinations
- Filters: Exclude microcaps (bottom 20% by NYSE cap), stocks under $3
Model / Methodology
- Portfolio Approach:
- Long-short decile 10 vs. 1
- Monthly rebalancing
- Alpha calculated via FF5 + Momentum model
- Cross-Sectional Regressions:
- Fama-MacBeth setup with controls for size, B/M, past return, profitability, and asset growth
- Statistical Corrections:
- Use MHT techniques (FWER, FDR, FDP)
- Bootstrap for empirical distributions
- Economic Hurdles:
- Require high Sharpe Ratio across full and half samples
- Strategy must be consistent in portfolio and regression frameworks
Key Table or Figure from the Paper
📊 Reference: [Table 5] – Strategies Surviving Statistical and Economic Filters
📌 Explanation:
- Of 2.1 million strategies, only 17 pass all filters (FDP control + high Sharpe + robustness)
- None relate to known anomalies
- One example:
(esubc − txdi) / dpvieb, which lacks any clear economic logic - Most published anomalies would not survive the authors’ filtering process
Final Thought
💡 Backtest results lie. Unless rigorously filtered, alpha is probably just noise. 🚨
Paper Details (For Further Reading)
- Title: p-Hacking: Evidence from Two Million Trading Strategies
- Authors: Tarun Chordia, Amit Goyal, Alessio Saretto
- Publication Year: 2017
- Journal/Source: SSRN Electronic Journal
- Link: https://doi.org/10.2139/ssrn.2968724