83.4% confidence intervals
I received a few questions and comments about my use of 83.4% confidence intervals on the plot in my prior post, so I thought I would post an explanation that I can refer to later.
---
Often, political scientists use a p-value of p=0.05 as a threshold for sufficient evidence of an association, such that only p-values under p=0.05 indicate sufficient evidence. Plotting 95% confidence intervals can help readers assess whether the evidence indicates that a given estimate differs from a given value.
For example, in unweighted data from the ANES 2020 Time Series Study, the 95% confidence interval for Black respondents' mean rating about Whites is [63.0, 67.0]. The number 62 falls outside the 95% confidence interval, so that indicates that there is sufficient evidence at p=0.05 that Black respondents' mean rating about Whites is not 62. However, the number 64 falls inside the 95% confidence interval, so that indicates that there is not sufficient evidence at p=0.05 that the mean rating about Whites among Black respondents is not 64.
---
But suppose that we wanted to assess whether two estimates differ *from each other*. Below is a plot of 95% confidence intervals for Black respondents' mean rating about Whites and about Asians, in unweighted data. For a test of the null hypothesis that the estimates differ from each other, the p-value is p=0.04, indicating sufficient evidence of a difference. However, the 95% confidence intervals overlap quite a bit.
The 95% confidence intervals in this case don't do a good job of permitting readers to assess differences between estimates at the p=0.05 level.
But below is a plot that instead uses 83.4% confidence intervals. The ends of the 83.4% confidence intervals come close to each other but do not overlap. If using confidence interval touching as an approximation to p=0.05 evidence of a difference, that closeness without overlapping is what we would expect from a p-value of p=0.04.
Based on whether 83.4% confidence intervals overlap, readers can often get a good sense whether estimates differ at p=0.05. So my current practice is to plot 95% confidence intervals when the comparison of interest is of an estimate to a given number and to plot 83.4% confidence intervals when the comparison of interest is of one estimate to another estimate.
---
NOTES
1. Data source: American National Election Studies. 2021. ANES 2020 Time Series Study Preliminary Release: Combined Pre-Election and Post-Election Data [dataset and documentation]. March 24, 2021 version. www.electionstudies.org.
Leave a Reply