--- title: "Feynman's Maze-Running Story" description: "Richard Feynman recounts an amazing anecdote about philosophy of science; we trace it to a forgotten University of Michigan research programme into 'floor cues'." thumbnail: /doc/psychology/animal/maze/1936-curtis-figure1-diagramoutlineoftheunitalikeratmaze.jpg thumbnailText: "The 'unit-alike maze' Curtis used to test rat learning of 'floor cues' in Feynman’s anecdote." created: 2014-05-04 modified: 2023-05-03 status: finished previous: /replication next: /tank confidence: highly likely importance: 2 cssExtension: dropcaps-de-zs ...
> A 1974 [Richard Feynman](!W) speech recounts an anecdote about a scientist, Mr Young, who in 1937 discovered serious flaws in rat maze-running psychology research: the rats were using information from the environment, like smells or the sound of the floor, to find their way through---rendering experiments dangerously ambiguous if the mazes are not carefully constructed to eliminate these side-channels (eg. by putting them on sand beds to dampen sounds). This discovery was then ignored by researchers, Feynman says, illustrating the difference between successful sciences like physics and 'cargo cult' ones like psychology. > > Who was Mr Young and what was his research which was ignored by mainstream psychology? This question has been asked for decades, and mysteriously, never answered. > > Here we finally answer it: Mr Young was probably Quin Fischer Curtis, in his theses published in 1931/1936, as part of John F. Shepard's multi-decade research programme at the University of Michigan into 'floor cues', going well beyond just putting sand on the floor. > > This research programme & its results tragically fell into oblivion due to the unfortunate circumstances of the UMich department which turned it into almost a parallel-universe of maze-running research: few results were published formally (in part due to Shepard's perfectionism & constantly coming up with new maze experiments to run), the department drifted far out of the mainstream, and the maze-running paradigm fell out of fashion around WWII, which is also when the UMich department was 'rebooted' to fix its hermeticism---leading to collective amnesia. > > So while it turns out the maze-running story supports the Feynman usage, it also tells a broader cautionary lesson about when academia does and does not function, in a different version of 'publish or perish'.
Nobel physicist Richard Feynman gave [a famous speech on "cargo cult science"](https://calteches.library.caltech.edu/51/2/CargoCult.htm) ([WP](!W "Cargo cult science"), [June 1974 publication](https://calteches.library.caltech.edu/51/2/CargoCult.pdf)) at the 1974 [Caltech](!W) commencement, in which he discussed methodology & the philosophy of science---what distinguishes science that *works* from 'science' that goes through all the motions of things like formal peer review and involving lots of numbers and costing a lot of money but doesn't work & the proverbial ['cargo'](!W "Cargo cult") does not land; the epistemological issues would become even more pointed in his 1986 [Rogers Commission Report](!W) [Appendix F](https://www.nasa.gov/history/rogersrep/v2appf.htm). In it, he gives a few examples like the [Millikan oil experiment](https://en.wikipedia.org/wiki/Oil_drop_experiment#Millikan's_experiment_as_an_example_of_psychological_effects_in_scientific_methodology) where physicists got the 'right' answer even when it turned out to be wrong (cf. [Henrion & Fischhoff 1986](/doc/statistics/bias/1986-henrion.pdf "Assessing uncertainty in physical constants")), and toward the end, he gives a non-physics one, from animal psychology, about the rat maze-running experiments of a "Young", which showed how rat maze-running *should* be done for the causal inferences to be valid. # Feynman 1974 To quote the anecdote in full: > ...Other kinds of errors are more characteristic of poor science. [When I was at](https://en.wikipedia.org/wiki/Richard_Feynman#Cornell) [Cornell](!W "Cornell University") [1945--1950], I often talked to the people in the psychology department.^[Feynman was interested in many things, like ants, with which [he did some informal navigation experiments](/doc/psychology/animal/maze/1985-feynman-surelyyourejokingmrfeynman-theamateurscientist-ants.pdf "‘Surely You're Joking, Mr Feynman: The Amateur Scientist’, Feynman 1985"), which [have been analogized to his work](https://www.mathpages.com/home/kmath320/kmath320.htm) on [path integral formulation](!W) & canceling-out of particle paths.] One of the students told me she wanted to do an experiment that went something like this---I don’t remember it in detail, but it had been found by others that under certain circumstances, _X_, rats did something, _A_. She was curious as to whether, if she changed the circumstances to _Y_, they would still do, _A_. So her proposal was to do the experiment under circumstances _Y_ and see if they still did _A_. I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person---to do it under condition _X_ to see if she could also get result _A_---and then change to _Y_ and see if _A_ changed. Then she would know that the real difference was the thing she thought she had under control. She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1935 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happens. > > ...[Not] All experiments in psychology are of this type, however. For example, there have been many experiments running rats through all kinds of mazes, and so on---with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before. > > The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and, still the rats could tell. > > He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell. > > Now, from a scientific standpoint, that is an A-Number-1 experiment. That is the experiment that makes rat-running experiments sensible, because it uncovers the clues that the rat is really using---not what you think it’s using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat-running. > > I looked into the subsequent history of this research. The subsequent experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn’t discover anything about the rats. In fact, he discovered *all* the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic of Cargo Cult Science. This is one of his most-read works, especially when it was republished unmodified in his 1985 quasi-autobiography [_Surely You're Joking, Mr. Feynman!_](!W "Surely You're Joking, Mr. Feynman!"). # Finding Mister Young Feynman refers to several things like [Rhine & ESP](https://en.wikipedia.org/wiki/Joseph_Banks_Rhine#Reception) or Millikan's oil drop experiments, which are easy enough to find, and sometimes linked in recurring mentions of Feynman's speech; but a source for Young is never given. Back in 2010, in a discussion of the burgeoning [Replication Crisis](https://en.wikipedia.org/wiki/replication_Crisis) (which [includes animal experiments](/doc/statistics/bias/animal/index)), a LW commenter ["Costanza"](https://www.lesswrong.com/posts/6vSJe9WXCNvy3Wpoh/the-decline-effect-and-the-scientific-method-link?commentId=CioABB2nPf4goMJ4F) mentioned the speech as an early example of systematic methodological problems in psychology & asked for the Young reference, because he couldn't find anything about it. ## Internet 2 years later, no one had found anything either. 2 years after *that*, I took a crack, set my timer to record how few seconds I found it in, expecting to get a nice [Google-fu case study](/search-case-studies "‘Internet Search Tips § Case Studies’, Branwen 2018")---and found nothing! This was surprising because Feynman is extremely specific & concrete: it should not require any more than a single query to Google Scholar of the form `author:Young date:1937 rat sand maze`. It *should* be extremely easy to find this reference: Feynman did not make things up, he provides plausible background (a prior interest in psychology & Cornell does indeed have a strong psychology program), he gives a name (which is, however, both a common adjective & unfortunately common surname in the USA---in the top 50), describes the circumstances (you can infer 'Young' is a grad student because maze-running research is so pure it is done only in academia and Feynman repeatedly refers to him as 'Mr', to emphasize his role as underdog/gadfly), describes the research in considerable depth, even describes doing a literature search to look for citations or mentions (implying it was published such that it *could* be cited), and so on. The only challenge would appear to be that it was probably a grad student, and it was obscure, but as challenging as that might be in 1937 or even 1974, fulltext search in the 2010s obviated most of the problem in searching these sorts of things---if it's not in Google Scholar, it's in Internet Archive, or HathiTrust, or ProQuest for student theses etc. It can be tedious or time-consuming, and one can rack up fees scanning old used books or purchasing ProQuest scans, and sometimes one hits frustrating walls like a document available only by visiting the Library of Congress in person---but it's not usually that *hard*. In particular, one does not usually run into a blank wall of zero leads. Every once in a while, I would be reminded of my white whale, and mention it on Twitter, or list it on my [Open Questions](/question#feynman) page, and run into the blank wall again. ## Doctor Paul Thomas Young The first plausible lead one usually finds is [award-winning](/doc/psychology/animal/1965-apa.pdf "‘Paul Thomas Young: Distinguished Scientific Contribution Award’, Association 1965") animal psychology Dr [Paul Thomas Young](!W) (1892--1978), who earned a PhD under [Edward B. Titchener](!W) at Cornell in 1918, and had a long successful career at the University of Illinois ([obituary](/doc/psychology/animal/1979-okelly.pdf "‘Paul Thomas Young: 1892–1978’, O’Kelly 1979")). Dr Young was a noted animal psychologist, and unsurprisingly given his era, used maze-running experiments. So, is *he* Mr Young? Doesn't look like it; there are obviously a few problems with this: #. Dr Young had held a doctorate for ~20 years by the time 'Mr Young' is being described as running mazes. #. Dr Young would be a senior professor, possibly department head, and unlikely to be running experiments himself. This is especially true if he took after his doctoral advisor, who is generally described as hewing to the tyrannical German style of academia he would've learned from *his* doctoral advisor, [Wilhelm Wundt](!W), which was part of importing German academic practices (like the doctorate itself) into the fledgling American higher-ed universe in the late 1800s. #. Dr Young was not obscure nor are his works ignored. #. There is no Feynman nexus: how did Feynman *learn all this* in the first place? Young was at Cornell for his doctorate, yes, but that had nothing to do with maze-running and was a good 27 years before Feynman arrived. If Young did it, it had to be at the University of Illinois where he moved to for the rest of his life. Still, he seems at first glance to be the right guy. For example, on Quora, a [Michael A. Gottlieb](https://www.feynmanlectures.caltech.edu/info/stories/michael_a_gottlieb_story.html) (who is described as "editor of [_The Feynman Lectures on Physics_](!W), coauthor of _Feynman's Tips on Physics_" & sounds like he should know what he's talking about), replies to a question about Mr Young by suggesting Dr Young and a specific paper. However, his citation doesn't work because it is the wrong year (a minor issue) and has nothing to do with 'sand', much less the story: >> I believe Feynman was referring to psychologist, P. T. Young. See, for example, Young 1938's paper, ["Preferences and demands of the white rat for food"](/doc/psychology/animal/1938-young.pdf) in the _Journal of Comparative Psychology_, Vol 26(3), Dec 1938, pp 545--589. The first part concerns spatial factors in feeding behavior, and may have been the very research Feynman referred to. > > No, Young 1938 is not the correct citation, I just read the paper after seeing your response. He talks about spatial factors by way of placement of feeding tubes in boxes (the rat picks the right side or the left side of the box, habits recorded), and rotating cups of food around rats. The issues seemed comparable, but I did not find any of the specific issues described by Feynman. > > I'm still looking for the actual citation, starting with the _Journal of Comparative Psychology_. OK, so perhaps that specific paper isn't it nor does it helpfully cite related work which might be it; we can't pass up such a lead, no matter the problems---an animal psychologist, named 'Young', who *has* done rat maze-experiments! He has to be investigated more. More focused Google Scholar searches, restricting it to all Dr Young-authored papers, and looking for 'sand', turns up nothing. [Mietek Bak](https://www.lesswrong.com/posts/6vSJe9WXCNvy3Wpoh/the-decline-effect-and-the-scientific-method-link?commentId=Rxc67BfJBpmYMbNaw) made a list of Young publications (most of which go far afield from maze-running or rigorous experimental method, one infers from the titles), and highlights 5 worth checking for discussion of sand; I checked 3 of the most promisingly-titled ones myself [[Young 1940](/doc/psychology/animal/1940-young.pdf "Reversal of Food Preferences of the White Rat Through Controlled Pre-Feeding"), [Young 1948](/doc/psychology/animal/1948-young.pdf "Appetite, palatability and feeding habit: a critical review"), [Young 1949](/doc/psychology/animal/1949-young.pdf "Food-seeking drive, affective process, and learning")], zilch, as well as Young's animal psychology textbooks like [_Emotion in Man and Animal_](https://catalog.hathitrust.org/Record/000426365) (surely *there* he would cover any sand-related methodology), also nothing. All of Dr Young's publications turn out to be irrelevant after exhausting the corpus; and none offer even a hint of the research, much less a smoking gun. The only possible lead is Young's correspondence with even more noted psychologist [Edwin Boring](!W)^[Yes, really.]: Boring had an interest in methodology like experimenter bias (coming up several times in [Rosenthal 1976](/doc/statistics/bias/1976-rosenthal-experimenterexpectancyeffects.pdf "Experimenter Effects in Behavioral Research: Enlarged Edition"), which thanks Boring). It is not implausible that somewhere in Boring's correspondence, sand comes up; but unfortunately, like most such academics' papers, his have not been digitized and would require a long trip to examine thoroughly, and thus is a dead end. ## Feynman Archives The biggest problem the Young dead-end poses is that having failed to find any trace in it, one must begin to assume that Feynman got the year, surname, or both wrong.[^pseudonym] Unfortunately, if we drop either one, and we search just 'maze sand', even with the plausible date-ranges like 1930--1940, we are swamped with vastly too many hits to vastly too hard to access papers/books/monographs. We could spend the rest of our life painstakingly obtaining fulltext of each one to look at the context around each hit for 'sand', and may well still not find it.^[If we could throw it into an embedding-powered search engine or into GPT-4, that would in fact be feasible---but we'd need fulltext in the first place, and then a lot of money.] [^pseudonym]: 'Young' is a bit suspicious a surname, as while it's a common one, it's also a common adjective, and makes one wonder if it was mis-transcribed or if it is some sort of humorous pseudonym. However, it is 'Young' in every version, including later published versions like the June 1974 periodical and then ten years later in the book _Surely_, so a simple editorial error seems unlikely to have persisted so long, especially when it is so prominent and said repeatedly (three times) and explicitly like "a man named Young". This cannot be a typo for 'a young man', cannot be a pseudonym or humorous epithet, and it is phrased precisely so as to be understood when spoken as a speech that "here is a man whose real surname is the word 'Young'". It might seem like a pseudonym, but nowhere in the speech does Feynman use pseudonyms (he simply omits the female student's name or the Fermilab bureaucrats), he has no hesitation naming & shaming (like Rhine), he is praising Mr Young and trying to unbury him so using his real name would be a very meritorious thing to do, and the uses of 'Mr Young' do not sound humorous at all. So I reject the pseudonym hypothesis as well. Confusion with Dr Paul Thomas Young, or an entirely unknown peripheral man, remains by far the most likely explanations for Feynman's error. Perhaps going in reverse from Feynman to the anecdote would help: the Caltech Feynman archives, which hold a comprehensive collection of Feynman papers because he made a practice of retaining & donating his papers there (including after the 1974 speech), might hold the answer? When asked for more information, they've apparently said they didn't know anything.^[But I've lost the reference for this, so it's possible I am misremembering a different inquiry, perhaps about his notorious [elementary-school IQ test score](https://www.eoht.info/page/Feynman%27s%20IQ).] Searching the Feynman papers digitally turns out to be impossible, and their [copyright/visiting policies are draconian](https://www.lesswrong.com/posts/6vSJe9WXCNvy3Wpoh/the-decline-effect-and-the-scientific-method-link?commentId=sM5STEexXcKr3KFg5). So, barring a visit to Los Angeles, this is a dead end too. And, with no other leads at all---not a single thing one could take to a reference librarian or ILL---most people gave up looking. ## Parallel Examples {.collapse}
Given the extensive literature on all the ways rats can cheat mazes, it's not surprising that---whether or not Young turned out to be real---there might be other real examples.^[See for example the [neural net tank urban legend](/tank "‘The Neural Net Tank Urban Legend’, Branwen 2011") where the original story seems to be essentially false for multiple reasons, but there are some real examples much later on, particularly [when broadened to reward-hacking](/tank#alternative-examples).]
A Twitter user noted a 1993-04-10 [Usenet](!W) discussion (`sci.bio`: "Pulling Habits out of Rats") about Mr Young, asking not for sourcing but for reasons why "Young" was ignored, where [Marvin Minsky](!W) [replied](https://groups.google.com/g/sci.bio/c/v4NchfXFiNs/m/iPfaISp-uCMJ) >> I can think of several possible explanations for this situation. >> >> #. Feynman was wrong >> #. Rat experiments are no longer performed or even of interest >> #. Rat experiments don't need to have these kinds of controls, because they are designed to measure attributes not affected by these things. >> #. Rat psychologists have crummy experimental procedures and hence worthless results. >> >> Can anyone offer some insights into which of these might be the case? Perhaps someone who has worked or is currently active in the field? > > What happened around 1937 was that > > 5\. [B. F. Skinner](!W) developed ways to control all those external variables by enclosing the experiment in a sealed, soundproof, lightproof, etc., box. The results were reliably reproducible, and a great deal was learned. The boxes were soon named ["Skinner Boxes"](!W) and became the new paradigm for studying animal learning. Skinner and many others switched to pigeons, for various reasons, but others continued to use rats. > > When I was undergraduate in the late 1940s, I hung around that lab [Skinner's lab at Harvard University] and helped with some switching and sequencing stuff to make the experiments more convenient. I don't remember the name of Young, but it was folklore that the change *was* because someone had found that rats appeared to be able to navigate by distant cues, eg. the appearance of the ceiling, so that the traditional open-topped maze experiments might be flawed. A more contemporary example, of odour cues in a maze, comes from ["Shortcut Learning in Deep Neural Networks"](https://arxiv.org/abs/2004.07780), Geirhos et al 2020: > **2.1 Shortcut learning in Comparative Psychology: unintended cue learning** > > Rats learned to navigate a complex maze apparently based on subtle color differences---very surprising given that the rat retina has only rudimentary machinery to support at best somewhat crude color vision. Intensive investigation into this curious finding revealed that the rats had tricked the researchers: They did not use their visual system at all in the experiment and instead simply discriminated the colors by the odour of the color paint used on the walls of the maze. Once smell was controlled for, the remarkable color discrimination ability disappeared... [[Nicholas Rawlins](!W), personal communication with F. A. W. some time in the early 1990s, confirmed via email on 12.11.2019] I asked on Twitter if Rawlins had seen this first hand or if it was secondhand, and [the second author stated](https://twitter.com/jh_jacobsen/status/1251226554602045443): > Yes, the anecdote happened as described in Nicholas Rawlins' laboratory at Oxford, confirmed in personal communication with [Felix A. Wichmann](https://scholar.google.com/citations?user=NxrQ794AAAAJ) in Nov '19. ## Maier & Schneirla 1935 {#maier-schneirla-1935-section} The solution turned out to be blind luck: searching after, and not before, 2019. My "Open Questions" page occasionally gets linked by blogs, and there were a flurry of links in April 2023. Frederic Bush happened to notice the Feynman entry & idly gave it a shot, doing a straightforward GS search for `rats sand maze` 1935--1940, which pulls up ["THE EFFECT OF FLOOR CUES UPON THE MASTERY OF THE UNIT-ALIKE MAZE"](https://www.google.com/books/edition/THE_EFFECT_OF_FLOOR_CUES_UPON_THE_MASTER/Q5KWcs7b1SYC) in Google Books (digitized 2019-12-02, "Original from: the University of Michigan"), whose title 'floor cues' would make anyone bolt upright (one is always looking for the key phrase which will unlock a trove of hits), and where the tiny snippets look like smoking guns---followed by further hits for Curtis & sand & maze-running inside a 1935 book scan uploaded to the IA in September 2019^[The temporal coincidence is interesting, and IA/GB's book scanning programs are closely connected, so it seems like they might've started in on a batch of books from UMich in late 2019?]: a [_Principles of animal psychology_](/doc/psychology/animal/1935-maier-principlesofanimalpsychology.pdf), by a [Norman R. F. Maier](!W) & [Theodore C. Schneirla](!W) at the [University of Michigan](!W). The description is so like the Feynman anecdote that it is hard to believe that it is not closely connected or indeed, the very anecdote itself! ### Obscurity Neither Maier, nor Schneirla, nor UMich had come up at all in prior discussions or papers, nor did they have any apparent connection to a 'Young', Cornell, or Feynman. Maier, for example, appears to have worked under a 'John F. Shepard' (no Wikipedia entry), among a list of prominent psychologists who otherwise all have Wikipedia articles like "[Karl Lashley](!W) & [Heinrich Kluver](!W) at Chicago". Ominously, the Wikipedia article on Maier notes that "Although rarely discussed today, Maier's research received extensive publicity in its day...In the 1950s, Maier changed his area of research to industrial psychology, he claimed in response to prejudicial treatment of him in the profession.^[2](/doc/psychology/animal/1993-dewsbury.pdf "‘On publishing controversy: Norman R. F. Maier and the genesis of seizures’, Dewsbury 1993")^" (He was most famous for the 'two cord problem', which I had vaguely heard of but [the fulltext](/doc/psychology/1931-maier.pdf "‘Reasoning in humans II: The solution of a problem and its appearance in consciousness [the two-cords problem]’, Maier 1931") was not in Google Scholar---another bad sign.) _Principles_ was a textbook which was well-regarded, but otherwise unremarkable, and aside from a reprint of [an expanded edition 2 decades later](/doc/psychology/animal/maze/1964-maier-principlesofanimalpsychologyenlarged2ndedition.pdf "‘Principles Of Animal Psychology (Enlarged Edition)’, Maier & Schneirla 1964") in 1964 (which discussion of mazes is unchanged), long since out of print & use. Nor is it easy to find fulltext of it without either being at a university library or purchasing a used copy. Used copies of it are inexpensive, but it was not in Libgen, you cannot read it in [Google Books](https://books.google.com/books/about/Principles_of_Animal_Psychology.html?id=y59OAAAAMAAJ)^[The critical bit about sand is [*just* barely visible](/doc/technology/google/2023-05-01-googlebooks-1934maier-searchsnippetsforsandkeyword.png) in the search snippets for "sand" and there is the critical mention of putting sand on floor and it having to do with mazes, but there is no visible material about rats or 'floor cues', and to even be searching the snippets in the first place, one would have to already know to check this book---there are just too many books which have hits for "sand" and "maze". (The HathiTrust search is even more useless, as it only lists pages which have keyword hits, with no context at all, and _Principles_ has enough irrelevant mentions of 'sand' that it gets punted to the second page of hits anyway.)] (digitized 2007-11-01) nor [HathiTrust](https://catalog.hathitrust.org/Record/001503310); a reasonable number of university libraries hold it (unsurprisingly), but typically relegated to long-term storage, and there are no signs of digital fulltext for non-HathiTrust universities. So, until IA scanned it 2019, _Principles_ was part of the dark matter of 20^th^ century [orphan works](!W): it's there, it's not usually that hard to get a physical copy of, but whoever is the copyright holder hasn't bothered to free it so no one dares provide a digital copy ("copyright is why we can't have nice things"), and it's hard to know that you want it in the first place---and this appears to be such case of an unknown unknown. #### Post-Mortem *Should* we have been able to find Maier & Schneirla's _Principles_ before April 2023, and back ~2013? Looking at available results: I don't think so. Sometimes you just get unlucky. In this case, given that finding Mr Young defied all efforts for so long, we should not be surprised if this extreme difficulty was due to extreme bad luck, because [a lot of things went wrong](!W "Swiss cheese model")---Feynman getting some critical bibliographic details wrong and omitting the key term 'floor cue', Shepard for not publishing & Pillsbury for letting him not publish, psychologists for forgetting & losing the work, copyright for suppressing orphan works, bad luck that _Principles_ was scanned in 2019 rather than the ~15 years of scanning before that, the expense of cutting-edge DL systems that could have feasibly read all hits for 'sand' in a corpus and intelligently assessed if the context was similar to the Feynman anecdote... ### Contents The book scan cannot be downloaded. I have purchased a used book scan for easier consultation but it will not be available for some time; one can use an Internet Archive to 'check it out' for 1 hour and read it online. **Ch17** of _Principles_ is ["The Analysis of Maze Learning"](/doc/psychology/animal/maze/1935-maier-principlesofanimalpsychology-ch17-theanalysisofmazelearning.pdf "‘Principles of Animal Psychology: Chapter 17: The Analysis of Maze Learning’, Maier & Schneirla 1935"), focusing first on "The Sensory Control of the Maze Habit", which is about "determining what differential sensory experiences an animal uses when it runs a maze". This chapter recounts what turns out to be a bit of a long-running war in early 20^th^ century maze-running research, about the varying capabilities of sight, smell, whisker and kinesthetic, sound, and smell, producing a complex set of findings suggesting that all of the senses could be relevant & rats would rely on combinations as available---in other words, that senses like smell could help them run the maze. Illustrating the difficulty in nailing down *how* rat navigation is [Hunter 1929](/doc/psychology/animal/maze/1929-hunter.pdf "The Sensory Control of the Maze Habit in the White Rat")/[Hunter 1930](/doc/psychology/animal/maze/1930-hunter.pdf "A Further Consideration of the Sensory Control of the Maze Habit in the White Rat"), who created a carefully-machined & daily-washed maze to remove kinesthetic & olfactory cues, but found that his blinded & de-whiskered mice still managed to learn the maze. He hypothesized it was due to the rats *hearing* something indicating direction like "a constant noise from one side of the room"; so [he created a *3D maze*](/doc/psychology/animal/maze/1929-hunter.pdf#page=13 "‘The Sensory Control of the Maze Habit in the White Rat § pg13’, Hunter 1929 (page 13)") to superimpose the different choices vertically and make each choice the same horizontal distance from any distant noise. _Principles_ passes to [describing work by the afore-mentioned John F. Shepard](/doc/psychology/animal/maze/1935-maier-principlesofanimalpsychology-ch17-theanalysisofmazelearning.pdf#page=8 "‘Principles of Animal Psychology: Chapter 17: The Analysis of Maze Learning § pg8’, Maier & Schneirla 1935 (page 8)"), who has done "many years of work in the analysis of learning of various types of mazes" and trying to nail down rat senses. In a new maze of his, rats were able to learn mazes with units up to 29, suggesting some serious misunderstanding, which Shepard also ascribed to "auditory" perception---specifically, "the patter of the rats' feet seemingly resonated differently on different parts of the floor." How to eliminate it? _Principles_ passes on to the section "The Elimination of the Floor Cue" on [page 384](/doc/psychology/animal/maze/1935-maier-principlesofanimalpsychology-ch17-theanalysisofmazelearning.pdf#page=10 "‘Principles of Animal Psychology: Chapter 17: The Analysis of Maze Learning § pg10’, Maier & Schneirla 1935 (page 10)"), where he describes the work of one Curtis 1931, who "covered the concrete floor of the building with **sand** to a depth of 1.5 inches." This change substantially reduced the performance of the rats. Even more strikingly, Curtis's maze came in 5-units, with the rat positioned differently, "Thus, at the junctions the rat had to turn left at the first two units and right at the third." This is also true of "a later study (unpublished)", where "the starting point varied, so that the rat was required to run 2, 3, or 4 units before reaching the food." This chapter breaks through the blank wall: we have a technical term now, 'floor cues', and we have at least two names, 'Curtis' and 'Shepard'. Curtis may not have been Mr Young, but he fits remarkably well. We now know where to start. # Curtis 1931 The citation turns out to be a master's thesis: - "The Control of Floor-cues in the Maze-learning of the White Rat", Curtis 1931 ([requested](https://www.reddit.com/r/Scholar/comments/132vhfw/thesis_the_control_of_floorcues_in_the/)) Unfortunately, it is inaccessible, although described in [Curtis 1936](/doc/psychology/animal/maze/1936-curtis.pdf#page=28 "‘The Effect of Floor Cues Upon the Mastery of the Unit-Alike Maze § Curtis 1931’, Curtis 1936 (page 28)"). However, we can search for more information about Curtis. Quin Fischer Curtis turns out to have had a successful [subsequent career](https://psychology.wvu.edu/about/history/west-virginia-university-s-first-psychologists)[^Curtis-career], albeit one that had little to do with maze-running or pure psychological research, so he appears to've done little related followup research (aside from his subsequent [PhD thesis](#curtis-1936-section)). We can look at citations of Curtis 1931. ## Solomon 1948 This doesn't take long because the *only* citation in Google Scholar is [Solomon 1948](/doc/psychology/animal/maze/1948-solomon.pdf#page=32 "‘The influence of work on behavior § pg32’, Solomon 1948 (page 32)") which briefly says: > The literature concerning the kinesthetic control of maze performance in animals has been voluminous...In experiments involving manipulation of intra-maze and extra-maze stimulation, the results are also conflicting. Carr and Watson ([8](/doc/psychology/animal/maze/1908-carr.pdf "‘Orientation in the white rat’, Carr & Watson 1908")), Dashiell and Helms ([16](/doc/psychology/animal/maze/1925-dashiell.pdf "‘The learning by white rats of an inclined plane maze’, Dashiell & Helms 1925")), Shepard ([76](/doc/psychology/animal/maze/1929-shepard.pdf "‘An Unexpected Cue in Maze Learning’, Shepard 1929")), and more recently, Hunter ([54](/doc/psychology/animal/maze/1940-hunter.pdf "‘A Kinesthetically Controlled Maze Habit in the Rat’, Hunter 1940")) have all offered evidence that maze proficiency could be acquired on the basis of kinesthesis alone. > > On the other hand the experiments of Carr ([7](/doc/psychology/animal/maze/1917-carr.pdf "‘The Journal of Animal Behavior: Volume 7 § pg159’, Carr et al 1917")), Dennis ([18](/doc/psychology/animal/maze/1929-dennis.pdf "‘The Sensory Control of the White Rat in the Maze Habit’, Dennis 1929"), [19](/doc/psychology/animal/maze/1931-dennis.pdf "‘The proprioceptive ability of the white rat’, Dennis 1931")), Macfarlane ([65](https://books.google.com/books/about/The_Role_of_Kinesthesis_in_Maze_Learning.html?id=X0R0QwAACAAJ)), Curtis 1931 (15, original not seen, as cited in [Maier & Schneirla 1935](#maier-schneirla-1935-section)), Honzik ([41](/doc/psychology/animal/maze/1933-honzik.pdf "‘Maze learning in rats in the absence of specific intra-maze and extra-maze stimuli’, Honzik 1933")) and others have shown rats to be incapable of learning mazes when all extra-maze and intra-maze differential cues were well controlled. This turns out to be quite helpful, as going through the references & the difficulty in fulltexting them tells us several things: - there was a long-standing debate over how rats ran mazes and how to control sensory leakage, but that this literature is little-cited & now highly obscure - eg. searching for the cited, I came across as well: ["The effects of olfactory cues on the maze learning of white rats"](/doc/psychology/animal/maze/1940-demand.pdf), De Mand 1940; ["Visual Cues in Maze Running by the Albino Rat"](/doc/psychology/animal/1930-walton.pdf), Walton 1930 - _Principles_ was the only way a reviewer in 1948 had access to Curtis's results - and Shepard published [an extremely relevant but short (~1pg) paper](#shepard-1929) on the 'floor cues' in 1929, describing how using rubber flooring and swapping out floor sections confused rats running the maze, stating that it rendered unspecified earlier research uninterpretable, and stating their intention to study it more [^Curtis-career]: This might seem a little surprising to a modern reader, especially given the UMich context, that Curtis could conduct this immediately-obscure research which gets ~0 citations and yet go on to postdoc work at Cornell (hmm...?) and then a successful [clinical psychology](!W) career at the burgeoning [West Virginia University](!W), rising to department chair 1949--1968 and even getting a clinical psychology clinic named after him (the ["Quin Curtis Center"](https://psychology.wvu.edu/about/quin-curtis-center)). But note the dates: this is around WWII. As I have pointed out many times, the explosion of American high ed post-WWII has few parallels---an era of such growth that obscure academics might discover universities phoning them up with unsolicited offers of tenure positions. And as the Curtis profile mentions, clinical psychology was in huge demand due to WWII itself, and only grew more afterwards, and this drove the [WVU growth](https://psychology.wvu.edu/about/history/expansion-of-the-psychology-faculty-at-wvu-in-the-1950s) as well. So Curtis got lucky in switching from the dying field of maze-based learning research to one that had a brilliant future (at least in terms of positions & funding). ## Curtis 1936 {#curtis-1936-section} Maier & Schneirla 1935's reference to an unpublished study makes sense when we continue searching for information on Curtis and discover he moved universities from Illinois to University of Michigan (ie. where Maier, Schneirla, and Shepard all worked), and published his PhD thesis in 1936 (after _Principles_ in 1935): - ["The Effect Of Floor Cues Upon The Mastery Of The Unit-Alike Maze"](/doc/psychology/animal/maze/1936-curtis.pdf), Curtis 1936 The experiments are both extensive & intensive, and border on monomania. There can be little doubt that if these experiments described in either thesis are not the exact experiments that Feynman is referring to, then they must have been done in this context, by Curtis, Shepard, or another Illinois/Michigan person. (After all, it would beggar belief for there to be a *completely separate & equally unfindable* research programme involving floor cues, sand, rats, mazes, and American grad students in the mid-1930s---especially given how small American research & higher education was back then pre-WWII!) This is as good an answer as one can expect, and if we had only Curtis's theses to go on, we could stop there. But the answer to one question often brings us another question. Reading the thesis, the role of Shepard jumps out much more than it did from Maier & Schneirla 1935: this is Shepard's thesis from start to finish^[No thesis advisor is listed. [Raphelson explains this](/doc/psychology/animal/maze/1980-raphelson.pdf#page=9 "‘Psychology at Michigan: The Pillsbury years, 1897–1947 § pg9’, Raphelson 1980 (page 9)") as a UMich peculiarity stemming from chairman Pillsbury's insistence on being named chairman of every doctoral thesis, regardless of who was the 'real' advisor. In Curtis's case, the role of Shepard is clear from the complete absence of all other UMich professors.], and Curtis is constantly referring to unpublished work by Shepard. ## University of Michigan So, what is up with all that? What is Shepard's deal? If we start googling Shepard more, we will stumble across the amazing work of the recently-deceased UMich professor [Alfred C. Raphelson](https://obits.mlive.com/us/obituaries/annarbor/name/alfred-raphelson-obituary?id=8005875), who has written an extensive history of the UMich psychology department. A condensed version was published in [1980](/doc/psychology/animal/maze/1980-raphelson.pdf "‘Psychology at Michigan: The Pillsbury years, 1897--1947’, Raphelson 1980"), which summarizes his [two](/doc/psychology/animal/maze/1968-raphelson.pdf "‘Psychology At The University Of Michigan 1852--1950: Volume I: The History Of The Department Of Psychology’, Raphelson 1968") [1968](/doc/psychology/animal/maze/1968-raphelson-2.pdf "‘Psychology At The University Of Michigan 1852--1950: Volume 2: Biographical Sketches Of Faculty Members Serving On The Staff During The Years 1897--1945’, Raphelson 1968b") histories/biographies. These histories reveal all, and explain everything. Raphelson mentions no Young, and Curtis only in passing as one of many grad students working for Shepard over the years; Shepard and the UMich psych department as a whole emerge as the key. It turns out that Maier & Schneirla 1935's description is just the tip of an iceberg. The UMich psych department had decades of intensive research into maze-running, setting aside much of its building for it, and constructing elaborate mazes [for](/doc/psychology/animal/maze/1968-raphelson-shepard-smallhumanmazelearningapparatususingadrum.png) [humans](/doc/psychology/animal/maze/1968-raphelson-2-shepard-1940-humanwalkthroughmazeinbasementofumichhillauditorium-toplevelview.png) as well. Raphelson particularly [profiles John F. Shepard](/doc/psychology/animal/maze/1980-raphelson.pdf#page=5 "‘Psychology at Michigan: The Pillsbury years, 1897–1947 § John F. Shepard’, Raphelson 1980 (page 5)"), who helped build the department from scratch starting in 1906 (literally, spending many years overseeing the building program, from which he was ousted) to his de facto ouster in the late 1940s, and oversaw generations of grad students, who were in turn overawed by his extensive knowledge, rigorous experimentation, devastating critiques^[Raphelson accuses him further of sabotaging his students by being too overwhelming and too critical, destroying their independence, and ensuring there could be no 'Shepardism', only Shepard and acolytes.], outspoken communism, and advanced courses where he systematically laid out his (never published) paradigm. The brief 1929 paper was but a hint of the mounds of data that they had compiled in their quest to understand and eliminate "floor cues". But almost none of it was published. The UMichers did not interact much with the rest of American psychology, published little (Maier is described by Raphelson as an exception), promoted mostly from within (becoming 'inbred'), and describes how admiring students would be shocked to invoke the authority of Shepard elsewhere & get no reaction. Under the benevolent but distant leadership of department chair [Walter B. Pillsbury](!W), tenured faculty were not held to 'publish or perish' ([like Claude Shannon](/review/book#shannon-late-career) when MIT lured him from Bell Labs with tenure), and the department gradually drifted ever more distant from outsiders. Eventually, when Pillsbury began retiring in 1941, the university dean decided the department needed to be 'rebooted', and passed over the senior Shepard in favor of an outsider, [Donald G. Marquis](https://en.wikipedia.org/wiki/Donald_Marquis_(psychologist)) at Yale, who would rectify the problems. Raphelson describes this 1945 reboot as a success, and done as kindly as possible; but it largely ended Shepard's career, who quietly retired, saying he would finally, at long last, properly write up & publish all his perfected results. Shepard wrote to Boring at this point that his "only real source of anxiety now is the realization that much of my life would be lost if I don’t get my maze results published." Raphelson notes Shepard failed, and was able to finish only one thing: an article confusingly titled the same thing as the 1929 abstract. He tried to get it published in _Psychological Monograph_, acknowledging in 1959 his tardiness: >> ...frankly, I do not expect many readers to be concerned about this study until after the next monograph is out. However, I think there would be a few: some oldsters who still remember a little about the maze; and a few others curious enough to start reading. Lastly, I do not want you to bother anyone else with the matter. If, after you have read the manuscript, you have any doubts as to the advisability of publishing it, for any reason, please return the manuscript to me. I shall lithoprint it and let the Michigan department distribute it. It would be of use to the department in more than one way. > > [journal editor] Norman Munn turned the monograph down since he believed that Shepard’s contribution on the floor cues had been known for a long time and all the details included in the monograph appeared to be anticlimactic. One notes the irony of quoting on *this* page an editor rejecting a floor-cues paper for publication because it "had been known for a long time". He did indeed lithoprint it. It is not available online as far as I can tell^[WorldCat's entry mentions an 'eBook', but I cannot find any trace of this supposed digitization, even in the UMich Library catalogue.], and a copy is held in [just 7 libraries worldwide](https://worldcat.org/title/1085940557); Google Scholar erroneously conflates the 1929 & 1959 publications, and excluding citations to the earlier one, it appears to be cited by <5 publications, all on the different topic of ['spontaneous alternation'](!W), and most of them due to a Robert J. Douglas.^[Who actually did explicitly test for floor-cues in [Douglas 1966](/doc/psychology/animal/maze/1966-douglas.pdf "Cues for spontaneous alternation")! He may be the only one to ever do so post-Curtis.] The monograph: - "An unexpected cue in maze learning", Shepard 1959 Another attempted article failed to be published when Shepard withdraw it for unclear reasons. And that was it. Raphelson concludes his Shepard biography thus: > On November 2, 1965, at the age of 84, John F. Shepard died. His last draft of research writings were gathered up and examined by some former students. It was regretfully decided that there was nothing which might be salvaged for publication. RIP. ## Michigan Nexus {.collapse}
Feynman's description is consistent with having read it in Maier & Schneirla 1935. But he also took a trip at the right time to UMich itself to hear about it, and that is a possible source for his knowledge.
Maier's textbook was popular enough that Feynman could have read it, or someone in the Cornell psychology department could have read it & described it to him; it is also noteworthy that Curtis apparently worked briefly at Cornell after UMich ("completion of post-doctoral work at Cornell University under the direction of [Howard Liddell](!W "Howard Liddell (psychologist)"), an influential comparative psychologist").^[This stay at Cornell appears to have involved Maier's work on frustration and [Liddell's "Behavior Farm"](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5803279/ "‘Working across species down on the farm: Howard S. Liddell and the development of comparative psychopathology, c. 1923–1962’, Kirk & Ramsden 2018"), as [Liddell's Cornell papers](https://rmc.library.cornell.edu/EAD/htmldocs/RMA01466.html) mention "[Experimental Neurosis in the Pig](/doc/psychology/animal/1937-curtis-2.pdf)---Quin F. Curtis, 1937" (eg. [Curtis 1937](/doc/psychology/animal/1937-curtis.pdf "Diurnal Variation in the Free Activity of Sheep and Pig") & [Curtis 1938](/doc/psychology/animal/1938-curtis.pdf "Frustration As An Experimental Problem: 4. Some Physiological Consequences Of Frustration")), which resulted in a series of 6 papers on frustration ("a symposium on Frustration as an Experimental Problem held at the annual meeting of the American Psychological Association...8 September 1938", as [the prefatory first paper](/doc/psychology/animal/1938-rosenzweig.pdf "‘Frustration As An Experimental Problem: 1. The Significance Of Frustration As A Problem Of Research’, Rosenzweig et al 1938") notes) and is likely related to Maier's research programme which culminated in [_Frustration_](https://archive.org/details/in.ernet.dli.2015.90433). (The frustration research is of some interest to *Maier's* backstory and why Maier was not involed in maze-running research after his textbook---apparently some rat research [led to an unfair backlash](#dewsbury-1993) and he ultimately drifted away to greener pastures in [industrial/organizational psychology](!W), where he would coin [Maier's law](/doc/statistics/bias/publication/1960-maier.pdf)). The frustration research, however, hasn't done much better with posterity than the floor-cues research did.] If Feynman had checked for citations of Curtis 1931 while he was at Cornell, then for most of that period (ie. 1945--1950), there would indeed have been zero citations if including Solomon 1948 (and we can safely assume that indexing services back then would be long delayed). So there may be no additional need to explain how Feynman would've known: the Maier & Schneirla textbook is plausible enough. However, there is another possible source. In his [oral histories](https://www.aip.org/history-programs/niels-bohr-library/oral-histories/5020-2), Feynman mentions taking a long trip to the [University of Michigan in 1947](https://cdn.calisphere.org/data/13030/k0/kt5n39p6k0/files/kt5n39p6k0.pdf#page=10) for a 'summer symposium': > ...It [his senior thesis] was rather [a useless theorem](https://en.wikipedia.org/wiki/Hellmann%E2%80%93Feynman_theorem) that nobody referred to. And I realized that it was not only useless but it was obvious. I could have written it in half a line. It was nothing. > > Therefore I was quite surprised when, in *1948 or 1949, I went to Michigan*---(I mean, a change in the timing, just to tell you, for my own amusement, because I always felt a little bit ashamed of it, on the grounds that it was too trivial and useless, not very interesting.) > > At Michigan I was giving some lectures on electrodynamics and Ted Berlin said to me, “What do you think about the controversy about your paper?” I said, “What paper?” He was talking about “Forces and Molecules”, and he told me that there was a big debate as to whether the theorem is true---it’s called the Feynman-Hellman theorem or something like that, because somebody else has published a similar thing---and there’s big excitement about it, they’re arguing whether it’s true, and somebody in Germany said it’s not true because I only included the electrostatic forces, there are also the kinetic forces, and so on---I said, “No, that’s not right.” He said, “I know. We’ve been arguing back and forth. They claim you made two mistakes that cancel, and you were just lucky, and we claim that it’s not that you were lucky but you knew what you were doing.” I said, “Well, I knew what I was doing. It’s correct.” You know? Apparently it had caused interest, and it is interesting to physical chemists. 1947 is before Shepard completed his gradual retirement, while the Marquis era had just begun, so knowledge of the floor cue work would still be reasonably current. 1947 is also still consistent with zero citations to Curtis 1931/Curtis 1936. Further, Feynman's description is low-key, and it sounds like he was bored. So he could have spent a long time talking to random UMich people, including the psych department, especially if he had been talking to Cornell psych people and so had points of comparison & a working knowledge, and might be interested in observing the still-rather-unique UMich department. # Summing Up The Shepard research programme, as apparently conducted in part by Curtis, is the source of the Feynman anecdote. The 'Mr Young' was probably Curtis as a grad student, and the work reported in most depth in Curtis 1936, and Feynman is slightly off by a year. Feynman is either wrong about the name, confusing it with a much more famous animal psychologist, or there is another Young in a minor role somewhere---perhaps the person who told him about the UMich maze-running. And he is correct that it was not cited much, if at all, by subsequent psychology researchers (regardless of when one checks). ## Is This Cargo Cult Science? Does the Shepard-Curtis work on floor cues bear the weight of Feynman's use of it as a morality play exemplifying bad "cargo cult science"? Maybe. Maze-running may not be a huge area in 1974 (or today), but it's also not like it went away *completely*, and mazes continue to be used occasionally---are they vulnerable to floor cues? Presumably yes, because I am not aware of any standard maze construction methods which would inherently obviate floor cues, and I don't recall any mention of mazes switching to the suspended mesh that Curtis found least vulnerable to clever rats. In this regard, the story works: Shepard-Curtis found something important about constructing mazes, and it simply got ignored, being somehow in a quantum superposition of so trivial & well-known it's not worth even publishing but also done right by no one & this is not considered a problem or even known to the researchers ostensibly producing valuable knowledge. On the other hand, it may be an embarrassing story about scientific flaws, but it's unclear that it would make any difference to what maze-running gets done. The Shepard-Curtis work wasn't all that obscure, when it was primarily done in the 1920s--1930s. It sat solidly in a tradition of maze-running work going back to [Carr & Watson 1908](#carr-watson-1908) on how rats are surprisingly flexible about how they learn to run a maze, exploiting any side-channel allowed them; this may not have been the most exciting or famous kind of maze-running research, exactly, but you certainly couldn't call it 'hidden' or 'suppressed' by a cabal of psychologists pushing fatally-flawed results from their paper-mills manufacturing tenure & grants. Floor-cues certainly are no problem for the major uses of mazes today like the swimming test, and I'm not sure it would affect the various rat studies in neuroscience & reinforcement-learning I generally read. ## Publish Or Perish The story may highlight a different, and more fundamental, flaws, which cannot be fixed by a sprinkling of sand or elevated mazes with metal mesh floors. The Raphelson histories explain why it *fell* into such obscurity even by the 1940s: Shepard was a stubborn perfectionist who was enabled to never published or make any real effort to propagate his findings beyond a narrow circle of maze-running psychologists centered on UMich and to a lesser extent, the University of Illinois, and a few old psychology acquaintances like Boring. Further, not long afterwards, the UMich department was 'rebooted', and much of the institutional memory of the maze work would have begun to disintegrate. Since hardly any of it had been published beyond difficult-to-obtain gray literature (eg. a PhD thesis not even available on microfiche, described/cited only in a textbook which fell into disuse by a researcher who soon left animal psychology entirely), this was a death knell. Even worse, the reboot was necessary in part because maze-running research had ceased to be the most exciting & [fashionable](/doc/statistics/bias/publication/1966-dunnette.pdf "‘Fads, fashions, and folderol in psychology’, Dunnette 1966") psychology research, and the remarkable achievements like [Tryon's maze-running breeding experiments](/tryon "‘Tryon’s Rat Experiment’, Branwen 2020") become history. This is not necessarily because any of it was *wrong*, or even fully explored (the simple question of 'what senses does a rat use to run a maze' still doesn't appear to have a definitive answer); as [Meehl notes](/doc/psychology/1978-meehl.pdf "‘Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology’, Meehl 1978"), theories in psychology are never refuted---they just fade away. (The reason for Shepard 1959 not being published is sadly telling in this regard.) If it is so hard to recover this research now, then perhaps this historical amnesia is the real problem: that psychology is not cumulative, and that someone can discover "*all* the things you have to do to discover something about rats" and it doesn't matter, and just becomes a matter of antiquarian curiosity for someone like me tracking down obscure anecdotes in some physicist's speech half a century later. There would seem to be plenty of blame to go around. # External Links - **Discussion**: [HN](https://news.ycombinator.com/item?id=35798239); Twitter: [1](https://twitter.com/Meaningness/status/1653599867330248705), [2](https://twitter.com/zooko/status/1653582952075259906)