Comments on Morning et al. 2019: "Socially desirable reporting and the expression of biological concepts of race"

The Morning et al. 2019 DuBois Review article "Socially Desirable Reporting and the Expression of Biological Concepts of Race" reports on an experiment from the Time-sharing Experiments for the Social Sciences. Documentation at the TESS link indicates that the survey was fielded between Oct 8 and Oct 14 of 2004, and the article was published online Oct 14 of 2019, so the data were about 15 years old, but I did not see anything in the article that indicated the year of data collection.

Here is a key result, discussed on page 11 of the article:

When respondents in the comparison group were asked directly whether they agreed with the statement on genetics and race, only 13% said they did. This figure is significantly lower than the 22% we estimated previously as "truly" supporting the race statement. As a result, we conclude that the social desirability effect for this item equals 9 percentage points (22 – 13).

That 22% estimate of support is for non-Black responses that are not weighted to reflect population characteristics, but my analysis indicated that the estimate of support falls to 14% when the weight variable in the TESS dataset is applied to the non-Black responses. The social desirability effect in the analysis with these weights is thus not statistically different than zero in the data. Nonetheless, the Morning et al. 2019 abstract generalizes the results to the population of non-Black Americans:

We show that one in five non-Black Americans attribute income inequality between Black and White people to unspecified genetic differences between the two groups. We also find that this number is substantially underestimated when using a direct question.

---

I would like for peer review to require [1] an indication of the year(s) of data collection and [2] a discussion of weighted results for an experiment when the data should be known or suspected to have included a third-party weight variable (such as data from TESS or a CCES module).

---

NOTES

1. This post is a follow-up of this tweet that tagged two of the Morning et al. 2019 co-authors.

2. In this tweet, I expressed doubt that a peer reviewer or editor would check these data to see if inferences are robust to weighting. Morning et al. 2019 indicates that a peer reviewer suggested that a weight be applied to account for an inequality between experimental groups (p. 8):

...the baseline group has a disproportionately large middle-income share and small lower-income share relative to the test and comparison groups. As suggested by one anonymous reviewer, we reran the analyses using a weight calculated such that the income distribution in the baseline group corresponds to that found in the treatment and comparison groups.

3. I am co-author in an article that discusses, among other things, variation in the use of weights for survey experiments in a political science literature.