This exercise has revealed itself as a useful platform from which to examine a wide range of issues in the philosophy of science and may, we suggest, even help us think about improving the public understanding of science.

…“one group will win. Each member of the winning group will receive a $66.79^$50₂₀₁₄ certificate good for dinner at Bob’s Land o’ Sushi” (or functionally equivalent local eatery, university store, etc.). “Second”, we say, “each group will submit two confidential reports on its progress and, at semester’s end, present its answer to the question. The group reports and presentations will be given letter grades reflecting quality and thoroughness, and these will determine your group’s grade for the Box Project. Each member gets his or her group’s grade.” “In short”, we stress, “your grade for the Box Project will reflect the quality of your work, not whether you win.” We invite questions. Invariably students ask whether they can X-ray the box, shine bright lights on the box, or subject the box to some other treatment, and invariably they are told that they can do anything to the box, so long as they do not open it. Having determined a suitable place to store the box (which is available by “sign-out” to any group) and set a proximate deadline for the first group reports, we form groups and move on to the next topic.

At this point, the students know much, but not all, that there is to know about the Box Project. It is true that there is just one question to answer and just one rule to follow. They do not know what is in the box, of course, but neither do we, having had the box filled and sealed so that we could not know what was in it. They often think they know how the competition will unfold. Surely, they presume, at semester’s end the box will be opened, and the group with the most accurate answer—namely, the answer that most closely matches the finally revealed contents of the opened box—will win. That sounds to them exactly like something that would happen in a sort of “philosophical science class” (which is often what they have imagined the philosophy of science must be) and indeed does happen in science, as they understand it: one’s guesses about reality can be eventually checked against reality.

The students are correct that the Box Project is an attempt to get them to view science from the point of view of a scientist, but they are wrong that the exercise will end with an opened box and accuracy scores for each group’s answer. No scientist has ever had the analogous pleasure of comparing her theory about some part of the world with that part of the world itself, and no student can have that pleasure if she is to get a sense of science from within. So, at the semester’s end (and beyond), the box remains sealed. Unless Box Project veterans have tipped off our students, the news that the box will not be opened is a shock, met with disbelief and disappointment. But this gives way to an appreciation for the point it makes, namely, that scientists never get to find out how some bit of theory “really did.” Our claims about the world inevitably extend beyond the immediately observable, and for this we pay a certain epistemic price.

…Some of Slater’s students asked whether they could fill the box for the next class—no doubt with yet more puzzling objects. Some even suggested that the box itself be but one component of a larger subsequent box.

…The Box Project was deployed first in Fall 2002; we have used it (separately) in 9 upper-level undergraduate philosophy of science classes since then, each with 8–30 students (most not philosophy majors), meeting 3 hours a week over a 15-week semester…we have never had to abandon the project because of a clearly-opened box.

…And, more generally, we have found that what might at first glance appear as a crisis in an implementation of the Box Project will in fact set the stage for a genuine debate over some issue at the heart of the philosophy of science. For example, when one of our groups revealed in its final presentation that it had passed a fiber-optic cable through a slight (already-existing) gap in the box, enabling color photographs of its contents, the result was not a collapse of the project but rather a fierce debate between the students over whether the rule guiding the science at hand—do not open the box—had been violated. What had appeared at first to be a headache for the instructor was in fact a chance for students to experience methodological debate. In fact, having conveyed the task and the rule of the Box Project, there was little for an instructor to do but listen to the debate with pleasure.

Not that there have not been close calls. In 2004 a group took the box to a US airport, planning to have it X-rayed. They told security that they had no idea what was in the box, and that they were under strict orders not to open it. Amazingly, and fortunately, the box and the students survived the episode neither harmed nor incarcerated.

…[Judging:] In both implementations, each group gives a final presentation on or near the last day of class. As many as 20 minutes are allotted for each presentation, including questions from other groups and any guests joining the class…As mentioned above, Hardcastle has final presentations judged by a student executive committee, while Slater invites other professors to serve as a panel of judges. Students are reminded that little things—such as stage presence, grooming, and professional attire—can subtly affect the panel’s judgment. They get nervous—especially as the panel and other teams begin to question their inferences, methods, and assumptions.

…it is in these later stages of the exercise that students are most often confronted with alternative, competing interpretations of their data. Something along these lines will have occurred throughout the semester, of course, as individuals within groups debate or challenge the group’s developing answer to the question of what is in the box. But, in our experience it is during the group presentations, when groups face questions from other students and perhaps other faculty, that alternative interpretations gain their full force. By this point, groups have often managed to X-ray the box, and it is often in the presentation of these X-rays that we witness that force. The following interaction is representative. Team A had presented their X-ray of the box and identified a small circular shadow as a Life Saver™ candy, when a representative of Team B interjected:

Team B: “How do you know that’s a Life Saver™? Couldn’t it be some kind of washer?”
Team A: “That can’t be a washer. If it was metal, it would be much brighter on the X-ray. The only thing it could be is a Life Saver™.”
Team B: “We didn’t say it was a metal washer; maybe it’s silicone. That’s consistent, we believe, with the density suggested by the X-ray.”
Team A: “Wait. Aren’t all washers metal?”
Team B: “Um . . . no, I don’t think so. I’m pretty sure that some are rubber or plastic.”

In another case, one group confidently identified an American quarter and dime in an X-ray, demonstrating that the diameters of the relevant shadows matched the diameters of an American quarter and dime. The demonstration was compelling until another student pointed out that coins of other countries may well have the same diameters, and yet another noted that the X-ray device may have been set to either enlarge or shrink images.

…Recall that during the presentations students often presume that any doubts about a group’s answer will be laid to rest (or, alternatively, confirmed) shortly, when the box is opened. The revelation that the box will never be opened gives students a valuable insight into the nature and appeal of antirealism about science, a classification we employ to describe accounts that counsel against conceiving the aim of science as discovering the truth about the natural world, no matter how well confirmed our theories seem. Our students have a natural inclination to realism and are typically perplexed when they encounter antirealist accounts of science.

Confronted by a range of heretofore-unanticipated alternative theories of what is in the box, all of them compatible with the evidence at hand, plus the realization that there are likely more alternative accounts of their data yet to be conceived (Exceeding Our Grasp: Science, History, and the Problem of Unconceived Alternatives, Stanford2006), plus the fact that the box may never be opened, students recognize and appreciate the epistemic commitment realism entails and the challenge involved in defending that commitment.

…Here one can profitably introduce questions about science funding and more broadly about the values that various funding models reflect. One might ask the students how one ought to distribute some large sum (say, $13,358.14^$10,000₂₀₁₄) to the class’s 4–5 groups, given that one’s sole aim was to determine what is in the box. Should the full sum be invested in whatever group is most able, best resourced (intellectually and materially), and most likely to get it right? Why not hedge one’s bet by giving some money to one of the long-shot groups? If so, how much? Does fairness enter into any of these calculations? Students—and especially those riled by the unfair distribution of valuable resources—thus grasp the complexities of science funding and the broader questions concerning the intersection of our science and our values.

…We can easily imagine yet other variations. Perhaps students could be allowed to change groups. Or there could be multiple boxes, some of whose contents are easier than others to discern. Unfair? Yes, but of course not all scientific puzzles are equally easily solved. Or perhaps the professor could take on the role of a malevolent deity, swapping out the box with a (largely) indistinguishable box with different contents at some point during the term. This would have the tendency of producing anomalies—either within groups whose exposure to the box straddled the swap or during the final presentations (assuming that at least one team got to examine a different box than the others). Would students question their paradigm of a hands-off (and truthful) professor? Maybe. But that itself would be an interesting scenario to discuss.