“Refiltering Hypothesis Tests to Control Sign Error”, Art B. Owen2016-10-31 ()⁠:

A common, though not recommended statistical practice is to report confidence intervals if and only if they exclude a null value of 0. The resulting filtered confidence intervals generally do not have their nominal confidence level. More worryingly, in low power settings their center points will be much farther from zero than the true parameter is and they will frequently lie on the wrong side of zero.

Many confidence intervals are constructed using an asymptotically Gaussian parameter estimate accompanied by a weakly consistent estimate of its variance. In these cases, we can subject the given confidence interval(s) to a second filtering step such that the probability of a sign error is controlled. This refiltering step retains only those confidence intervals that are sufficiently well separated from the origin.

It requires no assumptions on the dependencies among the test statistics.

Figure 3: The figure shows 100 randomly selected confidence intervals out of 105 that were generated to have 6% power at level α = 0.05 [and true effect = 1]. The ones shown are the first 100 to exclude 0.