Simple Randomization vs Blocking

When put this way, one might wonder how to improve the design of simple randomized experiments: if you have a group with too many women and too few men, why not change your coin flip to instead pairs of women and men? Instead of flipping a fair coin for each patient to pick whether that patient goes into the control or experimental group, why not instead take a pair of women and flip a coin to decide whether the one on the left goes into the experimental group, and then the woman on the right goes into the other group? And likewise for the men. Since so many things cluster within families, it would be better to try to balance families too, and ensure that if there are multiple siblings available, one sibling goes into the experiment and the other into the control, rather than risking lopsided allocation. (This turns out to be a serious issue in some animal experiments, if experimenters accidentally randomly allocate whole litters into one arm, which can happen often given the small n that animal research typically uses.) In fact, why not try to extend this ‘matching up’ to as many things as you can measure? If we use identical twins (like in Gustav III of Sweden’s coffee experiment), we match them on not just nationality, gender, location, parents, age, food consumption or whatnot, we even match them on genetics too, which is part of why identical twins are so eerily identical even when raised apart. (Note that this point isn’t the same as the variance component estimating going on in twin study or other family designs; we care that they are similar and thus comparisons are less noisy, not the question of how much of that eerie similarity is due to genetics or other sources.) Indeed, since a person is matched with themselves on just about everything and are even better than identical twins, why not test the treatment on the same person over multiple time periods? But people do change over time and there might be lingering effects, so perhaps it would be even better if you can test it on the same person simultaneously: for example, test an acne cream A on one side of the face and acne cream B on the other half of each subject’s face, flipping a coin to decide whether A/B or B/A.

This is the concept of blocking, one of whose advocates was Student (William Sealy Gosset), the great early statistician. While Student is best known for his work on exact tests and experiment design, he had an interest in genetics (perhaps due to his friendship with R. A. Fisher & professional work on evaluating plant breeding results), making an important contribution in Student1933; in his comments on the Lanarkshire Milk Experiment, he would unite both interests, pointing out the usefulness of genetic considerations for optimal experimental design in humans as well as agriculture.

Efficiency of Blocking for the Lanarkshire Milk Experiment

The history of blocking and its advantages compared to simple randomized experiments are discussed at length in “W.S. Gosset and Some Neglected Concepts in Experimental Statistics: Guinnessometrics II”, Ziliak2011/“Balanced versus Randomized Field Experiments in Economics: Why W. S. Gosset aka ‘Student’ Matters”, Ziliak2014 (see also Pearson1939 & Ziliak2008). An early example of blocking for demonstrating benefits comes from the farm of noted systematic breeder Robert Bakewell (1725^–₇₀1795_229ya), who experimented with many methods of improvement, particularly the benefits of thorough irrigation of his fields—proving it to his many domestic & international visitors with a quasi-random blocking: Observations on live stock: containing hints for choosing and improving the best breeds of the most useful kinds of domestic animals, George Culley & Robert Heaton 1804_220ya, pg75:

At Dishley, Mr Bakewell has improved a considerable tract of poor cold land, beyond anything I ever saw, or could have conceived, by this same mode of improvement;—and, ever ready to communicate his knowledge to the public, he has left proof pieces in different parts of his meadows, in order to convince people of the great importance and utility of this kind of improvement:—particularly, in one part he has been at pains to divide a rood of ground into twenty equal divisions, viz. two perches in each piece. It is so contrived that they can water the first, and leave the second unwatered; or miss the first, and water the second; and so on through all the 20 divisions; by which contrivance, you have the fairest and most unequivocal proofs of the good effects of improving ground by watering.

Housman 1894_130ya, pg7–9 mentions the blocking was used for fertilization as well:

Young says that his irrigation is “among the rarest instances of spirited husbandry”, much exceeding anything of the kind he had seen before, even in the hands of landlords…He did not hastily either adopt or extend his system of irrigation, but felt his way as he advanced, trying various experiments to satisfy himself of the efficacy and economy of the system before incurring further expense. Side by side he had plots of land: two plots, one watered, the other not watered; two again, one watered, the other manured; and again two, one watered from a spring, the other from the stream; so that he could form his estimates of the comparative value of irrigation as against other fertilising agencies, and of different modes of irrigation…We have seen in the instance of his irrigated land how Mr. Bakewell tested the worth of his notions by frequent and varied experiment. He did the same in every department of the farm. This was the grand source of his power. He did not try to make facts square with his opinions, but his opinions with facts.

Another early historical example is the previously-mentioned Gustav III of Sweden’s coffee experiment sometime in the late 1700s, which is possibly the first twin experiment.

Ziliak writes (pg22) about an interesting example where Student calculates that a well-blocked sample of 100 could be better than 20,000 with simple randomization:

The intuition behind the higher power of ABBA and other balanced designs to detect a large and real treatment difference was given by Student in 1911_113ya.⁶⁸ “Now if we are comparing two varieties it is clearly of advantage to arrange the plots in such a way that the yields of both varieties shall be affected as far as possible by the same causes to as nearly as possible an equal extent”.⁶⁹ He used this “principle of maximum contiguity” often, for example when he for example when he illustrated the higher precision and lower costs that would be associated with a small-sample study of biological twins, to determine the growth trajectory of children fed with pasteurized milk, unpasteurized milk, and no milk at all, in “The Lanarkshire Milk Experiment” (Student 1931_93yaa).⁶⁹ Student 1931_93yaa (p. 405) estimated that “50 pairs of [identical twins] would give more reliable results than the 20,000” child sample, neither balanced nor random, actually studied in the experiment funded by the Scotland Department of Health. “[I]t would be possible to obtain much greater certainty” in the measured difference of growth in height and weight of children drinking raw versus pasteurized milk “at an expenditure of perhaps 1–2% of the money and less than 5% of the trouble.” Likewise, Karlan & List 2007_17ya (p. 1,777) could have revealed more about the economics of charitable giving—for less—using a variant of Student’s method. Instead the AER article studied n = 50,083 primarily white, male, pro-Al Gore donors to public radio, neither random nor balanced.

I thought this was a great example and topical inasmuch as raw milk is still an issue because gourmands prize it over pasteurized milk (leading to occasional illnesses or deaths and subsequent legal/political maneuvering to ban/preserve access to raw milk), and I was curious about the details of how Student computed that the surprisingly low number of 100 twins (50 pairs) would suffice. So I looked for Student’s original paper.

Student begins by summarizing the Lanarkshire experiment in a little more detail:

…In the spring of 1930_94ya, a nutritional experiment on a very large scale was carried out in the schools of Lanarkshire. For four months 10,000 school children received 0.75 pint of milk per day, 5000 of these got raw milk and 5,000 pasteurized milk, in both cases Grade A (Tuberculin tested¹); another 10,000 children were selected as controls and the whole 20,000 children were weighed and their height was measured at the beginning and end of the experiment…The 20,000 children were chosen in 67 schools, not more than 400 nor less than 200 being chosen in any one school, and of these half were assigned as “feeders” and half as “controls”, some schools were provided with raw milk and the others with pasteurized milk, no school getting both…Secondly, the selection of the children was left to the Head Teacher of the school and was made on the principle that both “controls” and “feeders” should be representative of the average children between 5 and 12 years of age: the actual method of selection being important I quote from Drs Leighton and McKinlay’s [1930_94ya] Report: “The teachers selected the two classes of pupils, those getting milk and those acting as ‘controls’, in two different ways. In certain cases they selected them by ballot and in others on an alphabetical system.”

Unfortunately, while ambitious, the randomization was heavily compromised (discussed further in Kadane & Seidenfeld1996):

…after invoking the goddess of chance they unfortunately wavered in their adherence to her for we read: “In any particular school where there was any group to which these methods had given an undue proportion of well fed or ill nourished children, others were substituted in order to obtain a more level selection.” This is just the sort of afterthought that most of us have now and again and which is apt to spoil the best laid plans. In this case it was a fatal mistake, for in consequence the controls were, as pointed out in the Report, definitely superior both in weight and height to the “feeders” by an amount equivalent to about 3 months’ growth in weight and 4 months’ growth in height. Presumably this discrimination in height and weight was not made deliberately, but it would seem probable that the teachers, swayed by the very human feeling that the poorer children needed the milk more than the comparatively well to do, must have unconsciously made too large a substitution of the ill-nourished among the “feeders” and too few among the “controls” and that this unconscious selection affected, secondarily, both measurements.

Student then observes that besides producing a baseline imbalance (which is clearly visible in the graphs on pg400 & pg402, where the controls are taller in every time period, although this advantage lessens with time), this favoring of the poor could produce a systematic bias in the recorded weights during winter, which were made with the childrens’ clothes on, as it is entirely possible that poorer children will have lighter or fewer heavy winter clothes. Another issue is allocating entire schools to using either pasteurized or raw milk as their comparison to the no-milk controls (leading to problems in accounting for the hierarchical nature of the data due to confounding of school level effects with the pasteurized/raw effect). These three issues are reflected in anomalies in the data, reducing our confidence in the results despite it having been a randomized experiment (rather than something lamer like a survey noting that children who could afford milk were taller). R. A. Fisher as well noted that (Fisher & Barlett1931) taken at face-value, the experiment produced the opposite result of expected: raw milk being superior to pasteurized milk, despite the raw nutrient value of fat/protein/etc presumably being identical between milks and the greater safety of pasteurization, and that if anything, it implied that children should not be fed pasteurized milk.

Having performed the post-mortem on the Lanarkshire Milk Experiment, Student then proposes improvements to the design, culminating in the most radical (pg405):

(2) If it be agreed that milk is an advantageous addition to children’s diet—and I doubt whether any one will combat that view—and that the difference between raw and pasteurized milk is the matter to be investigated, it would be possible to obtain much greater certainty at an expenditure of perhaps 1–2% of the money [This is a serious consideration: the Lanarkshire experiment cost about £7,500. [Which is ~£374,800 in 2016 pounds sterling or ~$676,717.62^$541,100₂₀₁₆.]] and less than 5% of the trouble.

For among 20,000 children there will be numerous pairs of twins; exactly how many it is not easy to say owing to the differential death rate, but, since there is about one pair of twins in 90 births, one might hope to get at least 160 pairs in 20,000 children. But as a matter of fact the 20,000 children were not all the Lanarkshire schools population, and I feel pretty certain that some 200–300 pairs of twins would be available for the purpose of the experiment. Of 200 pairs some 50 would be “identicals” and of course of the same sex, while half the remainder would be non-identical twins of the same sex.

Now identical twins are probably better experimental material than is available for feeding experiments carried out on any other mammals, and the error of the comparison between them may be relied upon to be so small that 50 pairs of these would give more reliable results than the 20,000 with which we have been dealing. The proposal is then to experiment on all pairs of twins of the same sex available, noting whether each pair is so similar that they are probably “identicals” or whether they are dissimilar.

“Feed” one of each pair on raw and the other on pasteurised milk, deciding in each case which is to take raw milk by the toss of a coin. Take weekly measurements and weigh without clothes.

Some way of distinguishing the children from each other is necessary or the mischievous ones will play tricks. The obvious method is to take finger-prints, but as this is identified with crime in some people’s minds, it may be necessary to make a different indelible mark on a fingernail of each, which will grow off after the experiment is over. With such comparatively small numbers further information about the dietetic habits and social position of the children could be collected and would doubtless prove invaluable.

The comparative variation in the effect in “identical” twins and in “unlike” twins should furnish useful information on the relative importance of “Nature and Nurture”. …[The twin experiment] is likely to provide a much more accurate determination of the point at issue, owing to the possibility of balancing both nature and nurture in the material of the experiment.

This is a reasonable suggestion, but I was disappointed to see that Student gives no calculation or reference to another work with a similar calculation so it’s unclear if Student calculated out an exact answer and is not giving the details for reasons of flow or space, or was giving an off-the-cuff guess based on long experience with power analysis from his past statistical research & his brewery job. (If Ziliak was going to cite it as an example, it would’ve been better if he had verified that Student’s speculation was right at least to within an order of magnitude rather than simply quoting him as an authority.)

Power Estimate of Twins vs General Population

The calculation doesn’t seem hard. For this example, using height, it’d go something like:

1. take the estimated height gain in inches

2. find the distribution of height differences for twins and find the distribution of height for the general population of Scottish kids those ages

3. convert the inch gain into standard deviations for twins and for general population

4. plug those two into an effect size calculator asking for, say, 80% power

5. compare how big n₁ you need of twin pairs and how many n₂ pairs of general population, and report the fraction n₁⁄n₂ and how close it is to 5%.

generalD <- 0.083 / 1.56249311
twinD    <- 0.083 / 0.25
generalP <- power.t.test(d=generalD, power=0.8); generalP
#      Two-sample t test power calculation
#
#               n = 5564.07129
#           delta = 0.0531202342
#              sd = 1
#       sig.level = 0.05
#           power = 0.8
#     alternative = two.sided
#
# NOTE: n is number in *each* group
twinP <- power.t.test(d=twinD, power=0.8); twinP
#      Two-sample t test power calculation
#
#               n = 143.383601
#           delta = 0.332
#              sd = 1
#       sig.level = 0.05
#           power = 0.8
#     alternative = two.sided
#
# NOTE: n is number in *each* group
twinP$n / generalP$n
# [1] 0.0257695478

twinAll    <- 0.083 / (0.338*1.56249311)
generalAll <- 0.083 / 1.56249311

generalAll <- power.t.test(d=generalD, power=0.8)
twinAll    <- power.t.test(d=twinAll,  power=0.8)

twinAll$n / generalAll$n
# [1] 0.114397136

Additional Links

• Bingley et al 2015, “Signaling and Productivity in the Private Financial Returns to Schooling”,

• Briley et al 2018, “Behaviour Genetic Frameworks of Causal Reasoning for Personality Psychology”

• Eney et al 2017, “Cross-sectional association between soda consumption and BMI in a community-based sample of twins”

• Hjelmborg et al 2016, “Lung cancer, genetic predisposition and smoking: the Nordic Twin Study of Cancer”

• Lee 2012_12ya, “Correlation and Causation in the Study of Personality”

• Rottensteiner et al 2015, “Physical Activity, Fitness, Glucose Homeostasis, and Brain Morphology in Twins”

• van Dongen et al 2012_12ya, “The continuing value of twin studies in the -omics era”

Backlinks