βDo GRE Scores Help Predict Getting a Physics Ph.D.? A Comment on a Paper by Miller Et Alβ, 2019-02-25 (; similar)β :
[commentary] A recent paper in Sci. Adv. by Miller et al concludes that GREs do not help predict whether physics grad students will get Ph.D.s. The paper makes numerous elementary statistics errors, including introduction of unnecessary collider-like stratification bias, variance inflation by collinearity and range restriction, omission of needed data (some subsequently provided), a peculiar choice of null hypothesis on subgroups, blurring the distinction between failure to reject a null and accepting a null, and an extraordinary procedure for radically inflating confidence intervals in a figure. Release of results of simpler models, eg. without unnecessary stratification, would fix some key problems.
The paper exhibits exactly the sort of research techniques which we should be teaching students to avoid.
[Weissman retrospective: ββ¦I kept trying to steer the conversation toward the importance of maintaining scientific honesty and competence, rather than arguing about fuzzy policy issues. The journalist said βBut the people I talk with say this is a moral issue.ββ]