Design and validation of the Quantum Mechanics Conceptual Survey

Abstract

The Quantum Mechanics Conceptual Survey (QMCS) is a 12-question survey of students’ conceptual understanding of quantum mechanics. It is intended to be used to measure the relative effectiveness of different instructional methods in modern physics courses. In this paper, we describe the design and validation of the survey, a process that included observations of students, a review of previous literature and textbooks and syllabi, faculty and student interviews, and statistical analysis. We also discuss issues in the development of specific questions, which may be useful both for instructors who wish to use the QMCS in their classes and for researchers who wish to conduct further research of student understanding of quantum mechanics. The QMCS has been most thoroughly tested in, and is most appropriate for assessment of (as a posttest only), sophomore-level modern physics courses. We also describe testing with students in junior quantum courses and graduate quantum courses, from which we conclude that the QMCS may be appropriate for assessing junior quantum courses, but is not appropriate for assessing graduate courses. One surprising result of our faculty interviews is a lack of faculty consensus on what topics should be taught in modern physics, which has made designing a test that is valued by a majority of physics faculty more difficult than expected.

2 More

Received 12 July 2010

DOI:https://doi.org/10.1103/PhysRevSTPER.6.020121

This article is available under the terms of the Creative Commons Attribution 3.0 License. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Authors & Affiliations

S. B. McKagan^1,*, K. K. Perkins², and C. E. Wieman^3,1,2

¹JILA, University of Colorado and NIST, Boulder, Colorado 80309, USA
²Department of Physics, University of Colorado, Boulder, Colorado 80309, USA
³Department of Physics, University of British Columbia, Vancouver, British Columbia, Canada V6T 1Z1

^*Present address: American Association of Physics Teachers and McKagan Enterprises, Seattle, WA, 98144, USA.

Article Text

I. GOALS OF THE SURVEY

The Quantum Mechanics Conceptual Survey (QMCS) was designed to be used as a general survey of students’ conceptual understanding of quantum mechanics. Our primary goal was to create a survey that could be used to measure the relative effectiveness of different curricula and teaching techniques, and thus to serve as a formative assessment tool for faculty engaging in reform of their courses. To achieve this goal it is important to focus on standard topics that are covered in most courses, both traditional and reformed. Therefore, the QMCS does not cover many topics that we believe are important but are not universally taught, including measurement, applications of quantum mechanics, and historical experiments. Further, to have an impact on faculty practice, the test questions must be written such that the average faculty member believes that they are measuring something important. A secondary goal in creating the survey was to provide a way to measure the prevalence of specific student difficulties. As we will discuss, we have found that this secondary goal is often in conflict with the primary goal.

The QMCS was originally intended to be applicable to all levels of quantum mechanics, from sophomore-level modern physics to graduate quantum mechanics. This intention was based on the assumption that there is a set of universally agreed-upon basic concepts that each level of quantum mechanics instruction builds upon at a deeper level. However, we have found that this is not a particularly good assumption, and we were not able to design a test that is truly applicable at all levels. The QMCS is most relevant to, and has been most thoroughly tested in, sophomore-level modern physics courses. In these courses, we recommend giving the QMCS as a posttest at the end of a course only, and not as a pretest at the beginning of the course. As we will discuss, most students have little or no knowledge of any of the concepts measured by the QMCS at the beginning of a modern physics course, so it is both meaningless and demoralizing as a pretest.

The QMCS has also been used in junior-level and graduate-level quantum courses. However, faculty generally regard the test as too basic for these courses. In graduate courses, there is no difference between pretest and posttest scores, even for students who score low on the pretest, and there is no correlation between QMCS scores and final exam scores. These results are consistent with faculty assertions that they do not teach the concepts measured by the QMCS in graduate courses. However, graduate students in selective programs do score higher on the QMCS than do physics graduate students in less selective programs.

In keeping with standard practice for research-based conceptual tests in physics [1], we have protected the security of the test by keeping it from becoming available to students. After administering it in class we do not post solutions or allow students to take the tests home. We have administered it online to graduate students and advanced undergraduates, but always on a password-protected site with a warning not to distribute it. To further protect the security of the test, we do not include it in this article. It can be accessed from our website [2] by requesting a password from the authors.

II. REVIEW OF PREVIOUS RESEARCH

III. DESIGN OF THE QMCS

IV. REFINEMENT AND VALIDATION OF THE QMCS

V. VALIDITY ISSUES WITH SPECIFIC QUESTIONS

VI. SUGGESTIONS FOR USING THE QMCS

VII. SUGGESTIONS FOR FURTHER RESEARCH

ACKNOWLEDGMENTS

We thank the Physics Education Research Group at the University of Colorado for extensive feedback on all aspects of the development of this survey. This work was supported by the NSF and the University of Colorado Science Education Initiative.

Appendices

References

Edward F. Redish, Teaching Physics with the Physics Suite (John Wiley & Sons Inc., Somerset, 2003).
http://per.colorado.edu/QMCS.
D. Hestenes, M. Wells, and G. Swackhammer, Force concept inventory, Phys. Teach. 30, 141 (1992).
R. R. Hake, Interactive-engagement vs traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses, Am. J. Phys. 66, 64 (1998).
D. Huffman and P. Heller, What does the force concept inventory actually measure?, Phys. Teach. 33, 138 (1995).
P. Heller and D. Huffman, Interpreting the force concept inventory: A reply to Hestenes and Halloun, Phys. Teach. 33, 503 (1995).
D. Hestenes and I. Halloun, Interpreting the force concept inventory: A response to March 1995 critique by Huffman and Heller, Phys. Teach. 33, 502 (1995).
Ibrahim Halloun and David Hestenes, "The search for conceptual coherence in FCI data,", http://modeling.asu.edu/R&E/CoherFCI.pdf, 1995.
C. Henderson, Common concerns about the Force Concept Inventory, Phys. Teach. 40, 542 (2002).
R. Steinberg and M. Sabella, Performance on multiple-choice diagnostics and complementary exam problems, Phys. Teach. 35, 150 (1997).
Laura McCullough and David Meltzer, in 2001 Physics Education Research Conference Proceedings, (PERC Publishing, Rochester, New York, 2001).
S. Rebello and D. Zollman, The effect of distracters on student performance on the Force Concept Inventory, Am. J. Phys. 72, 116 (2004).
M. Dancy and R. Beichner, Impact of animation on assessment of conceptual understanding in physics, Phys. Rev. ST Phys. Educ. Res. 2, 010104 (2006).
R. Beichner, Testing student interpretation of kinematics graphs, Am. J. Phys. 62, 750 (1994).
R. K. Thornton and D. R. Sokoloff, Assessing student learning of Newton’s laws: The force and motion conceptual evaluation and the evaluation of active learning laboratory and lecture curricula, Am. J. Phys. 66, 338 (1998).
P. V. Engelhardt and R. J. Beichner, Students’ understanding of direct current resistive electrical circuits, Am. J. Phys. 72, 98 (2004).
D. P. Maloney, T. L. O’Kuma, C. J. Hieggelke, and A. Van Heuvelen, Surveying students’ conceptual knowledge of electricity and magnetism, Am. J. Phys. 69, S12 (2001).
C. Singh and D. Rosengrant, Multiple-choice test of energy and momentum concepts, Am. J. Phys. 71, 607 (2003).
L. Ding, R. Chabay, B. Sherwood, and R. Beichner, Evaluating an electricity and magnetism assessment tool: Brief electricity and magnetism assessment, Phys. Rev. ST Phys. Educ. Res. 2, 010105 (2006).
E. Cataloglu and R. W. Robinett, Testing the development of student conceptual and visualization understanding in quantum mechanics through the undergraduate career, Am. J. Phys. 70, 238 (2002).
E. Cataloglu, Ph.D. thesis, Pennsylvania State University, 2002.
J. Falk, Master’s thesis, Uppsala University, 2004.
S. Wuttiprom, M. D. Sharma, I. Johnston, R. Chitaree, and C. Soankwan, Development and Use of a Conceptual Survey in Introductory Quantum Physics, Int. J. Sci. Educ. 31, 631 (2009).
C. Singh, Student understanding of quantum mechanics, Am. J. Phys. 69, 885 (2001).
C. Singh, Student understanding of quantum mechanics at the beginning of graduate instruction, Am. J. Phys. 76, 277 (2008).
Rebecca S. Lindell, Elizabeth Peak, and Thomas M. Foster, in 2006 Physics Education Research Conference Proceedings, edited by L. McCullough, P. Heron, and L. Hsu, (AIP Press, Melville, NY, 2007).
W. K. Adams and C. E. Wieman, Int. J. Sci. Educ., (to be published).
E. Redish, J. Saul, and R. Steinberg, Student expectations in introductory physics, Am. J. Phys. 66, 212 (1998).
W. K. Adams, K. K. Perkins, N. Podolefsky, M. Dubson, N. D. Finkelstein, and C. E. Wieman, New instrument for measuring student beliefs about physics and learning physics: The Colorado Learning Attitudes about Science Survey, Phys. Rev. ST Phys. Educ. Res. 2, 010101 (2006).
B. Ambrose, Ph.D. thesis, University of Washington, 1999.
L. Bao, Ph.D. thesis, University of Maryland, 1999.
Homeyra R. Sadaghiani, Ph.D. thesis, The Ohio State University, 2005.
Jeffrey T. Morgan, Michael C. Wittmann, and John R. Thompson, in 2003 Physics Education Research Conference Proceedings, edited by S. Franklin, K. Cummings, and J. Marx, (AIP Press, Melville, NY, 2004).
M. C. Wittmann, J. T. Morgan, and L. Bao, Addressing student models of energy loss in quantum tunneling, Eur. J. Phys. 26, 939 (2005).
R. N. Steinberg, G. E. Oberem, and L. C. McDermott, Development of a computer-based tutorial on the photoelectric effect, Am. J. Phys. 64, 1370 (1996).
R. Müller and H. Wiesner, Teaching quantum mechanics on an introductory level, Am. J. Phys. 70, 200 (2002).
R. V. Olsen, Introducing quantum mechanics in the upper secondary school: a study in Norway, Int. J. Sci. Educ. 24, 565 (2002).
R. Knight, Five Easy Lessons: Strategies for Successful Physics Teaching (Addison Wesley, San Francisco, 2004).
D. Domert, C. Linder, and A. Ingerman, Probability as a conceptual hurdle to understanding one-dimensional quantum scattering and tunneling, Eur. J. Phys. 26, 47 (2005).
S. B. McKagan and C. E. Wieman, in 2005 Physics Education Research Conference Proceedings, edited by P. Heron, L. McCullough, and J. Marx, (AIP Press, Melville, NY, 2006).
M. Dubson, S. Goldhaber, S. Pollock, and K. Perkins, in 2009 Physics Education Research Conference Proceedings, edited by C. Henderson, M. Sabella, and C. Singh, (AIP Press, Melville, NY, 2009).
Bradley S. Ambrose, in 2004 Physics Education Research Conference Proceedings, edited by J. Marx, P. Heron, and S. Franklin, (AIP Press, Melville, NY, 2005).
Peter Robert Fletcher, Ph.D. thesis, University of Sydney, 2004.
Old versions are available at http:/per.colorado.edu/QMCS.
S. B. McKagan, K. K. Perkins, and C. E. Wieman, Deeper look at student learning of quantum mechanics: The case of tunneling, Phys. Rev. ST Phys. Educ. Res. 4, 020103 (2008).
The three eliminated questions include the two questions about reflection and transmission at a potential step shown in Fig. 9, which were eliminated because we found that many faculty do not cover this topic in modern physics, and one question asking whether photons travel in sinusoidal paths, which was eliminated because we did not want to have two questions about a single topic that many faculty found of questionable value. See Sec. V B for discussion of eliminated questions.
S. B. McKagan, K. K. Perkins, and C. E. Wieman, in 2006 Physics Education Research Conference Proceedings, edited by L. McCullough, P. Heron, and L. Hsu, (AIP Press, Melville, NY, 2007).
L. D. Carr and S. B. McKagan, Graduate quantum mechanics reform, Am. J. Phys. 77, 308 (2009).
S. Goldhaber, S. Pollock, M. Dubson, P. Beale, and K. Perkins, in 2009 Physics Education Research Conference Proceedings, edited by C. Henderson, M. Sabella, and C. Singh, (AIP Press, Melville, NY, 2009).
Louis Deslauriers and Carl Wieman, Phys. Rev. ST Phys. Educ. Res., (to be published).
R. Doran, Basic Measurement and Evaluation of Science Instruction (NSTA, Washington, D.C, 1980).
P. Kline, A Handbook of Test Construction: Introduction to Psychometric Design (Methuen, London, 1986).
G. Kuder and M. Richardson, The theory of the estimation of test reliability, Psychometrika 2, 151 (1937).
L. J. Cronbach, Coefficient alpha and the internal structure of tests, Psychometrika 16, 297 (1951).
G. A. Ferguson, On the theory of test discrimination, Psychometrika 14, 61 (1949).
A. P. French and Edwin F. Taylor, An Introduction to Quantum Physics (Norton, New York, 1978).
Richard W. Robinett, Quantum Mechanics; Classical Results, Modern Systems, and Visualized Examples (Oxford University Press, New York, 1997).

Physical Review Physics Education Research