“The Promise of Potential: A Study on the Effectiveness of Jury Selection to a Prestigious Visual Arts Program”, 2022-05-16 ():
…We add to this literature [on selection procedures] by studying a specific kind of selection procedure, one which is unstructured and where the criteria for selection are not explicitly formulated, and the decision process of the jury is not formalized.
…We analyze 11 years of admission decisions to a highly selective post-graduate visual arts program, the Rijksakademie van Beeldende Kunsten. Our unique, longitudinal data includes detailed information about the selection procedures as well as the later artistic and economic achievements for both accepted [3.5%] and rejected applicants (n = 8,557).
Regarding the predictive value of the selection procedures on later performance, our analyses show that the largest gain made is in the first, more cursory, pre-selection round. We find that weeding out less promising applicants is easier than identifying later top-performers, in terms of ex-ante expectations. The subsequent or final selection round is resource intensive and consists of several interviews with multiple jury members. Here we uncover an implicit structure of criterion variables that helps explain the ex-post admission decisions. Our simulations show that actual selection during the final round is slightly better than alternative decision rules (including a decision by lottery) in identifying applicants’ potential. Nevertheless, measured by future performance, the gain from rigorous jury assessments relative to modest and cheaper selection methods is minimal.
Our conclusions are maintained under several robustness checks.
…Occupational psychology research and human resource management literature have shown that subjective selection committee rankings obtained through unstructured interviews predict future job performance imperfectly (Dana et al 201311ya; McDaniel et al 199430ya; 1988; Wright et al 198935ya). Nevertheless, this kind of selection process is widely practiced. Past research has shown that selection committees tend to overestimate the validity of unstructured interviews (2008; et al 2016; Lievens et al 200519ya; Rynes et al 200222ya; 1996). There appears to be a widely held perception that unstructured selection processes provide a window into capturing the unexplained variance of the applicant’s future potential that is not attributable to other objectively measured competencies (2008).
…We use ArtFacts.Net, a web-based platform established in 2001 that ranks contemporary visual artists based upon their annual exhibitions at galleries and museums worldwide. The platform contains exhibition history information for ~550,000 visual artists based on 750,000 exhibitions provided by over 35,000 galleries, museums, and other venues worldwide. ArtFacts.Net data has been widely used as a measure of performance in the management science literature ( et al 2016) and cultural sociology (2013; 2012). We use individual rankings5 based on 2016 public art exhibitions and artists’ 7-year average ranking for 2010–62016. As mentioned earlier, we also measure performance by auction sales. Notably, previous research has shown that contemporary visual artists—of which our data is comprised—are usually not active on the secondary or auction market ( et al 2015; 2013). This is because artists first gain legitimacy and build their reputation on the primary art market (2020) by exhibiting their artworks at museums and art galleries before a select few enter the secondary or auction market. Nevertheless, and exactly because RABK is a highly prestigious and renown art residency program, we investigate sales made at auction as a supplementary metric. We collect these data from Artnet.com, which is an art market website that was established in 1989 and provides detailed information about more than 9 million public art auction results from 1,600 international auction houses (Artnet.com annual report, 2014). For details, see §6.
…During pre-selection, 8.2% of the 8,557 applicants are invited for an interview. Of those invited, 21.5% have a letter of recommendation and 12.4% applied more than twice compared to those applicants not invited for an interview (14.8% and 9.1% respectively). On artistic prestige rankings, our focal performance metric, the interviewed applicants score better in relation to those who are not interviewed. For instance, almost 70% are mentioned in ArtFacts.Net and have a substantially better (lower) artistic prestige score compared to those not interviewed. More precisely, the percentage difference between the two groups is 35.3% based on the ArtFacts.Net 7-year mean rank.