Comments on Bonilla and Tillery Jr 2020

The American Political Science Review published Bonilla and Tillery Jr 2020 "Which identity frames boost support for and mobilization in the #BlackLivesMatter movement? An experimental test".

---

The Bonilla and Tillery Jr 2020 Discussion and Conclusion indicates that:

Further studies should also focus on determining why African American women are mobilizing more than men in response to every frame that we exposed them to in our survey experiment.

But I don't think that the Bonilla and Tillery Jr 2020 data support the claim that every frame caused more mobilization among African American women than among African American men.

Bonilla and Tillery Jr 2020 has figures that measure support for Black Lives Matter and figures with outcomes about writing to Nancy Pelosi, but Bonilla and Tillery Jr 2020 also combines support and mobilization with the phrasing of "mobilizing positive attitudes" (p. 959), so I wanted to check what outcome the above passage was referring to. The response that I received suggested that the outcome was writing to Nancy Pelosi. But I don't see any basis for a claim about gender differences for each frame for that outcome, in Bonilla and Tillery Jr 2020 Figure 4B, the text of Bonilla and Tillery Jr 2020, or my analysis.

---

Another passage from Bonilla and Tillery Jr 2020 (p. 958):

For those not identifying as LGBTQ+, we saw a stronger negative effect in asking for support as a result of the Feminist treatment than LGBTQ+ treatment (βFeminist = -0.08, p = 0.08; βLGBTQ+ = -0.04, p = 0.34).

The p-value was p=0.45 for my test of the null hypothesis that the -0.08 coefficient for the feminist treatment equals the -0.04 coefficient for the LGBTQ+ treatment. There is thus not sufficient evidence in these data that these coefficients differ from each other, so it's not a good idea to claim that one treatment had a stronger effect than the other.

---

The lead author of Bonilla and Tillery Jr 2020 presented these data to Harvard's Women and Public Policy Program, noting at about 31:40 the evidence that the nationalist frame had a significant effect among women on the "mention police" outcome and noting at about 32:48 that "Black men in general...were much less likely than Black women to talk about the police in general". But my analysis indicated that p=0.35 for a test of the null hypothesis that the effect of the nationalist frame does not differ by gender for the "mention police" outcome.

---

Similar problem in this passage with the suggestion about results being consistent with "a differential response to the Black feminist treatment by gender" (p. 954, footnote omitted and emphasis added):

For female respondents, we see nonsignificant (positive) effects of the Black nationalist (β = 0.03, p = 0.39) and Black LGBTQ+ treatments (β = 0.03, p = 0.30), and nonsignificant (negative) effects of the Black feminist treatment (β = -0.02, p = 0.46). In contrast, we found that male respondents were much more affected by the intersectional treatments...but both the Black feminist and Black LGBTQ+ treatments decreased Black male approval of BLM (βFeminist = -0.06., p = 0.07; βLGBTQ+ = -0.09, p = 0.008).

---

NOTES

1. Bonilla and Tillery Jr 2020 had a preregistration. Here is hypothesis 3 from the preregistration...

H3: LGBTQ and Intersectional frames of the BLM movement will have no effect (or a demobilizing effect) on the perceived effectiveness of BLM African American subjects.

...and from the article (emphasis added)...

H3: Black LGBTQ+ frames of the BLM movement will have a positive effect on Black LGBTQ+ members, but they will have no effect or a demobilizing effect on Black subjects who do not identify as LGBTQ+.

I don't think that this deviation was super important, but the difference makes me wonder whether the APSR peer reviewers and/or editors bothered to check the preregistration against the article. Even if this check was made, it would be nice if the journal signaled to readers that this check was made.

2. Bonilla and Tillery Jr 2020 thanks the prior APSR editors:

Finally, we thank the three anonymous reviewers and the previous APSR editors, Professor Thomas Koenig and Professor Ken Benoit, for pushing us to significantly improve this paper through the review process.

Something else nice would be for journal articles to indicate the editorial teams responsible for the decision to publish the article and responsible for checking the manuscript for errors and flaws.

3. I was curious to see what subsequent research has discussed about Bonilla and Tillery Jr 2020. Let's start with Anoll et al 2022 from Perspectives on Politics:

This could be of great political consequence considering the importance of parents as socializing agents (Jennings and Niemi 1974; Niemi and Jennings 1991; Oxley 2017) and the necessity of building multiracial support for BLM (Bonilla and Tillery 2020; Corral 2020; Holt and Sweitzer 2020; Merseth 2018).

I'm not sure what about Bonilla and Tillery Jr 2020 supports that citation about "the necessity of building multiracial support for BLM". Let's try Hswen et al 2021 from JAMA Network Open, which cited Bonilla and Tillery Jr 2020 as footnote 14:

Although often used following fatal encounters with law enforcement, #BlackLivesMatter also became an important tool to raise awareness around health inequities in Black communities, such as HIV, adequate access to analgesia, and cancer screening.^13,14

I'm not sure what about Bonilla and Tillery Jr 2020 supports that citation about BLM being "an important tool to raise awareness around health inequities in Black communities".

From Jasny and Fisher 2022 from Social Networks (sic for "complimentary"):

Research has also shown that when multiple issues or beliefs are seen as complimentary, a process called "frame alignment," the connection can boost support for social movements and motivate participation (Bonilla and Tillery, 2020, Heaney, 2021; for an overview of "frame alignment," see Snow et al., 1986).

I'm not sure what in Bonilla and Tillery Jr 2020 that support-boosting and/or participation-motivating multiple frame alignment refers to. At least Heaney 2022 from Perspectives on Politics is on target with what Bonilla and Tillery Jr 2020 is about:

If BLM is able to convey an intersectional message effectively to its supporters, then this idea is likely to be widely discussed in movement circles and internalized by participants (Bonilla and Tillery 2020).

But Heaney 2022 doesn't tell readers what information Bonilla and Tillery Jr 2020 provided about intersectional messages. Next are the "charity" citations from Boudreau et al 2022 from Political Research Quarterly, in which Bonilla and Tillery Jr 2020 is absolutely unnecessary to support the claims that the citation is used for:

From the protests against police brutality in the 1960s and 70s to the emergence of the Black Lives Matter movement in recent years, there is a long history of police-inspired political mobilization (Laniyonu 2019; Bonilla and Tillery 2020)...

The symbolic appeal of a movement that served as a focal point and mobilizer of Americans' outrage was manifested in the Black Lives Matter signs posted in windows and scrawled on sidewalks and buildings across the country (Bonilla and Tillery 2020).

I'm not even sure that Bonilla and Tillery 2020 is a good citation for those passages.

And from Krewson et al 2022 from Political Research Quarterly (footnote text omitted):

In May of 2021, we obtained a sample of 2170 high quality respondents from Qualtrics, a widely respected survey firm (Bonilla and Tiller 2020; Friedman 2019; Kane et al. 2021).⁶

Ah, yes, Bonilla and Tiller [sic] 2020 which, what, provided evidence that Qualitrics is widely respected? That Qualtrics provides high quality respondents? Bonilla and Tillery Jr 2020 used Qualtrics, I guess. The omitted footnote text didn't seem relevant and seems to be incorrect, based on comparing the footnotes to the working paper and based on the content of the footnotes, with, for example, footnote 6 being about the ACBC design but the main text mention of the ACBC design linking to footnote 7.

Here is a prior post about mis-citations. Caveats from that post apply to the above discussion of citations to Bonilla and Tillery 2020, with the discussion not being systematic or representative, which prevents any inference stronger than Bonilla and Tillery 2020 being miscited more often than it should be.