Please rotate your device to landscape mode to view the charts.

Nonstandard errors

Journal: Journal of Finance

Date: 2024

Author: Albert J. Menkveld, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Michael Kirchler, Sebastian Neusüß, Michael Razen, Utz Weitzel, Fincap Team, Fearghal Kearney, Tony Klein, Liangyi Mu, and others

Abstract:
In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty--nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.

Link: Google Scholar

Background and Context

Research Design

164 research teams independently tested the same hypotheses on identical financial market data, allowing examination of how different analysis approaches affect results.

Data Sample

Deutsche Börse provided 17 years of trading data (2002-2018) for Europe's most actively traded instrument - the EuroStoxx 50 index futures, containing 720 million trade records.

Methodology

Teams tested six hypotheses about market trends, with results evaluated by 34 peer reviewers through multiple feedback stages to examine convergence in findings.

Significant Variation in Research Team Estimates

  • Shows the dispersion in estimates across research teams for each hypothesis tested
  • Even for straightforward measures like market share (RT-H3), there was substantial variation (1.2%)
  • Most complex measures showed highest variation, with client gross trading revenue (RT-H6) having 21.4% dispersion

Impact of Research Quality on Estimate Dispersion

  • Higher quality research showed less dispersion in results
  • Papers with better reproducibility scores had 25% less variation
  • Higher peer-rated papers showed 33.3% less dispersion in estimates

Reduction in Estimate Dispersion Through Peer Feedback Stages

  • Each stage of peer feedback reduced variation in estimates
  • Total reduction of 47.2% in dispersion from first to final stage
  • Demonstrates value of peer feedback in improving research consistency

Research Teams Underestimate Result Variation

  • Research teams substantially underestimated how much results would vary
  • Predicted standard deviation was 71.7% lower than actual variation
  • Suggests researchers may be overconfident about methodological consensus

Impact of Analysis Choices on Result Variation

  • Shows how sampling frequency dramatically affects results
  • Daily sampling produced extremely different results from monthly or yearly
  • Highlights importance of methodological choices in research

Contribution and Implications

  • First large-scale examination of how researcher decisions affect empirical finance results
  • Demonstrates importance of peer review and feedback in reducing variation in research findings
  • Suggests need for greater methodological standardization in financial research
  • Highlights value of reproducible research practices and transparent analysis choices

Data Sources

  • Dispersion Chart: Based on Panel C of Table I showing IQR values across RT hypotheses
  • Quality Impact Chart: Based on quantile regression results from Table III
  • Stage Progression Chart: Based on Table IV showing changes across feedback stages
  • Belief Chart: Based on analysis of belief survey results discussed in Section B.3
  • Analysis Choice Chart: Based on multiverse analysis results shown in Figure 7