2015 APSA paper
Here is the manuscript that I plan to present at the 2015 American Political Science Association conference in September: revised version here. The manuscript contains links to locations of the data; a file of the reproduction code for the revised manuscript is here.
Comments are welcome!
Abstract and the key figure are below:
Racial bias is a persistent concern in the United States, but polls have indicated that whites and blacks on average report very different perceptions of the extent and aggregate direction of this bias. Meta-analyses of results from a population of sixteen federally-funded survey experiments, many of which have never been reported on in a journal or academic book, indicate the presence of a moderate aggregate black bias against whites but no aggregate white bias against blacks.
NOTE:
I made a few changes since submitting the manuscript: [1] removing all cases in which the target was not black or white (e.g., Hispanics, Asians, control conditions in which the target did not have a race); [2] estimating meta-analyses without removing cases based on a racial manipulation check; and [3] estimating meta-analyses without the Cottrell and Neuberg 2004 survey experiment, given that that survey experiment was more about perceptions of racial groups instead of a test for racial bias against particular targets.
Numeric values in the figure are for a meta-analysis that reflects [1] above:
* For white respondents: the effect size point estimate was 0.039 (p=0.375), with a 95% confidence interval of [-0.047, 0.124].
* For black respondents: the effect size point estimate was 0.281 (p=0.016), with a 95% confidence interval of [0.053, 0.509].
---
The meta-analysis graph includes five studies for which a racial manipulation check was used to restrict the sample: Pager 2006, Rattan 2010, Stephens 2011, Pedulla 2011, and Powroznik 2014. Inferences from the meta-analysis were the same when these five studies included respondents who failed the racial manipulation checks:
* For white respondents: the effect size point estimate was 0.027 (p=0.499), with a 95% confidence interval of [-0.051, 0.105].
* For black respondents: the effect size point estimate was 0.268 (p=0.017), with a 95% confidence interval of [0.047, 0.488].
---
Inferences from the meta-analysis were the same when the Cottrell and Neuberg 2004 survey experiment was removed from the meta-analysis. For the residual 15 studies using the racial manipulation check restriction:
* For white respondents: the effect size point estimate was 0.063 (p=0.114), with a 95% confidence interval of [-0.015, 0.142].
* For black respondents: the effect size point estimate was 0.210 (p=0.010), with a 95% confidence interval of [0.050, 0.369].
---
For the residual 15 studies not using the racial manipulation check restriction:
* For white respondents: the effect size point estimate was 0.049 (p=0.174), with a 95% confidence interval of [-0.022, 0.121].
* For black respondents: the effect size point estimate was 0.194 (p=0.012), with a 95% confidence interval of [0.044, 0.345].
You should test whether the black and white estimates are significantly different from each other.
That's a good idea, JK. I'm not sure whether I can back out a p-value using the random-effect meta-analysis confidence intervals. However, based on estimates that do not use the manipulation check restriction, if I subtract the estimate for the white sample from the estimate for the black sample for each of the sixteen survey experiments, the resulting two-tailed p-value for the test of whether the black-white difference is non-zero is 0.1118. I'm not sure whether I should weight the cases based on the precision of the estimates, though.
Following Cumming's rule, they are. When CIs don't touch, the CI for the difference doesn't contain 0. Not sure how that works with random-effects MA estimates.
Thanks, Emil. So, based on Cumming's rule, the same-race bias for blacks is statistically larger than the same-race bias for whites.
I'm not sure how to test whether the problack bias among blacks is statistically larger than the problack bias among whites, but maybe Rule of Eye 4 from here is applicable: http://www.apastyle.org/manual/related/cumming-and-finch.pdf.
The midpoint of the left side of the [0.053, 0.509] 95% CI for black respondents is 0.167, and the right end of the 95% CI for white respondents is 0.124, so, by Rule of Eye 4, the corresponding p-value is less than 0.05.
The overlap is larger for the final set of CIs listed in the post. The midpoint of the left side of the [0.044, 0.345] 95% CI for black respondents is 0.119, and the right end of the 95% CI for white respondents is 0.121.