Home › Posts tagged inequality › Page 4

Tag: inequality

By Posted on Posted in Sex No Comments

The Peterson et al. 2019 PLOS ONE article "Mitigating gender bias in student evaluations of teaching" reported on an experiment conducted with students across four Spring 2018 courses: an introduction to biology course taught by a female instructor, an introduction to biology course taught by a male instructor, an introduction to American politics course taught by a female instructor, and an introduction to American politics course taught by a male instructor. Students completing evaluations of these teachers were randomly assigned to receive or to not receive a statement about how student evaluations of teachers are often biased against women and instructors of color.

The results clearly indicated that "this intervention improved the SET scores for the female faculty" (p. 8). But that doesn't address the mitigation of bias in the title of the article because, as the article indicates, "It is also possible that the students with female instructors who received the anti-bias language overcompensated their evaluations for the cues they are given" (p. 8).

---

For the sake of illustration, let's assume that the two American politics teachers were equal to each other and that the two biology teachers were equal to each other; if so, data from the Peterson et al. 2019 experiment for the v19 overall evaluation of teaching item illustrate how the treatment can both mitigate and exacerbate gender bias in student evaluations.

Here are the mean student ratings on v19 for the American politics instructors:

4.65     Male American politics teacher CONTROL

4.17     Female American politics teacher CONTROL

4.58     Male American politics teacher TREATMENT

4.53     Female American politics teacher TREATMENT

So, for the American politics teachers, the control had a 0.49 disadvantage for the female teacher (p=0.02), but the treatment had only a 0.05 disadvantage for the female teacher (p=0.79). But here are the means for the biology teachers:

3.72     Male biology teacher CONTROL

4.02     Female biology teacher CONTROL

3.73     Male biology teacher TREATMENT

4.44     Female biology teacher TREATMENT

So, for the biology teachers, the control had a 0.29 disadvantage for the male teacher (p=0.25), and the treatment had a 0.71 disadvantage for the male teacher (p<0.01).

---

I did not see any data reported on in the PLOS ONE article that can resolve whether the treatment mitigated or exacerbated or did not affect gender bias in the student evaluations of the biology teachers or the American politics teachers. The article's claim about addressing the mitigation of bias is, by my read of the article, rooted in the "decidedly mixed" (p. 2) literature and, in particular, on their reference 5, to MacNell et al. 2015. For example, from Peterson et al. 2019:

These effects [from the PLOS ONE experiment] were substantial in magnitude; as much as half a point on a five-point scale. This effect is comparable with the effect size due to gender bias found in the literature [5].

The MacNell et al. 2015 sample was students evaluating assistant instructors for an online course, with sample sizes for the four cells (actual instructor gender X perceived instructor gender) of 8, 12, 12, and 11. That's the basis for "the effect size due to gender bias found in the literature": a non-trivially underpowered experiment with 43 students across four cells evaluating *assistant* instructors in an *online* course.

It seems reasonable that, before college or university departments use the Peterson et al. 2019 treatment, there should be more research to assess whether the treatment mitigates, exacerbates, or does not change gender bias in student evaluations in situations in which the treatment is used. For what it's worth, the gender difference has been reported to be about 0.13 on a five-point scale based on a million or so Rate My Professors evaluations, using the illustration of 168 additional steps for a 5,117-step day. If the true gender bias in student evaluations were 0.13 units against women, the roughly 0.4-unit or 0.5-unit Peterson et al. 2019 treatment effect would have exacerbated gender bias in student evaluations of teaching.

---

NOTES:

1. Thanks to Dave Peterson for comments.

2. From what I can tell, if the treatment truly mitigated gender bias among students evaluating the biology teachers, that would mean that the male biology teacher truly did a worse job teaching than the female biology teacher did.

3. I created a index combining the v19, v20, and v23 items, which respectively are the overall evaluation of teaching, a rating of teaching effectiveness, and the overall evaluation of the course. Here are the mean student ratings on the index for the American politics instructors:

4.56     Male American politics teacher CONTROL

4.21     Female American politics teacher CONTROL

4.36     Male American politics teacher TREATMENT

4.46     Female American politics teacher TREATMENT

So, for the American politics teachers, the control had a 0.35 disadvantage for the female teacher (p=0.07), but the treatment had a 0.10 advantage for the female teacher (p=0.59). But here are the means for the biology teachers:

3.67     Male biology teacher CONTROL

3.90     Female biology teacher CONTROL

3.64     Male biology teacher TREATMENT

4.39     Female biology teacher TREATMENT

So, for the biology teachers, the control had a 0.23 disadvantage for the male teacher (p=0.35), and the treatment had a 0.75 disadvantage for the male teacher (p<0.01).

4. Regarding MacNell et al. 2015 being underpowered, if we use the bottom right cell of MacNell et al. 2015 Table 2 to produce a gender bias estimate of 0.50 standard deviations, the statistical power was 36% for an experiment with 20 student evaluations of instructors who were a woman or a man pretending to be a woman and 23 student evaluations of instructors who were a man or a woman pretending to be a man. If the true effect of gender bias in student evaluations is, say, 0.25 standard deviations, then the MacNell et al. study had a 13% chance of detecting that effect.

R code:

library(pwr)

pwr.t2n.test(n1=20, n2=23, d=0.50, sig.level=0.05)

pwr.t2n.test(n1=20, n2=23, d=0.25, sig.level=0.05)

5. Stata code:

* Overall evaluation of teaching

ttest v19 if bio==0 & treatment==0, by(female)

ttest v19 if bio==0 & treatment==1, by(female)

ttest v19 if bio==1 & treatment==0, by(female)

ttest v19 if bio==1 & treatment==1, by(female)

* Teaching effectiveness:

ttest v20 if bio==0 & treatment==0, by(female)

ttest v20 if bio==0 & treatment==1, by(female)

ttest v20 if bio==1 & treatment==0, by(female)

ttest v20 if bio==1 & treatment==1, by(female)

* Overall evaluation of the course

ttest v23 if bio==0 & treatment==0, by(female)

ttest v23 if bio==0 & treatment==1, by(female)

ttest v23 if bio==1 & treatment==0, by(female)

ttest v23 if bio==1 & treatment==1, by(female)

 

sum v19 v20 v23

pwcorr v19 v20 v23

factor v19 v20 v23, pcf

gen index = (v19 + v20 + v23)/3

sum index v19 v20 v23

 

ttest index if bio==0 & treatment==0, by(female)

ttest index if bio==0 & treatment==1, by(female)

ttest index if bio==1 & treatment==0, by(female)

ttest index if bio==1 & treatment==1, by(female)

Tagged with: , ,

By Posted on Posted in Sex No Comments

In the 2019 PS: Political Science & Politics article "How Many Citations to Women Is 'Enough'? Estimates of Gender Representation in Political Science", Michelle L. Dion and Sara McLaughlin Mitchell address a question about "the normative standard for the amount women should be cited" (p. 1).

The first proposed Dion and Mitchell 2019 measure is the proportion of female members of the American Political Science Association (APSA) by section and primary field, using data from 2018. According to Dion and Mitchell 2019: "When political scientists compose course syllabi, graduate reading lists, and research bibliographies, these membership data provide guidance about the minimum representation of scholarship by women that should be included to be representative by gender" (p. 3).

But is APSA section membership in 2018 a reasonable benchmark for gender representation in course syllabi that include readings from throughout history?

Hardt et al. 2019 reported on data for readings assigned in the training of political science graduate students. Below are percentages of graduate student readings in these data that had a female first author:

Time PeriodFemale First Author %
Before 19703.5%
1970 to 19796.7%
1980 to 198911.3%
1990 to 199915.7%
2000 to 2009 21.0%
2010 to 201824.6%

So the pattern is increasing representation of women over time. If this pattern reflects increasing representation of women over time in APSA section membership or increasing representation of women among the set of researchers whose research interests include the topic of a particular section, then APSA section membership data from 2018 will overstate the percentage of women needed to ensure fair gender representation on syllabi or research bibliographies. For illustrative purposes, if a section had 20% women across the 1990s, 30% women across the 2000s, and 40% women across the 2010s, a fair "section membership" benchmark for gender representation on syllabi would not be 40%; rather, a fair "section membership" benchmark for gender representation on syllabi would be something like 20% women for syllabi readings across the 1990s, 30% women for syllabi readings across the 2000s, and 40% women for syllabi readings across the 2010s.

---

Dion and Mitchell 2019 propose another measure that is biased in the same direction and for the same reason: gender distribution of authors by journal from 2007 to 2016 inclusive for available years.

About 68% of readings in the Hardt et al. 2019 graduate training readings data were published prior to 2007: 15% of these pre-2007 readings had a first female author, but 24% of the 2007-2016 readings in the data had a first female author.

Older readings are included on Hardt et al. 2019 readings with decent frequency: 42% of readings that had the gender of the first author coded were published before 2000. However, the Dion and Mitchell 2019 measure of journal representation from 2007 to 2016 ignores these older readings, which produces a biased measure favoring women if fair representation means representation that matches the representation in the relevant pool of syllabi-worthy journal articles.

---

In a sense, this bias in the Dion and Mitchell 2019 measures might not matter much if the measures are used in the biased manner that Dion and Mitchell 2019 proposed (p. 6):

We remedy this gap by explicitly providing conservative estimates of gender diversity based on organization membership and journal article authorship for evaluating gender representation. Instructors, researchers, and editors who want to ensure that references are representative can reference these as floors (rather than ceilings) for minimally representative citations.

The Dion and Mitchell 2019 suggestion above is that instructors, researchers, and editors who want to ensure that references are representative use a conservative estimate as a floor. Both the conservative nature of the estimate and its use as a floor would produce a bias favoring women, so I'm not sure how that is helpful for instructors, researchers, and editors who want to ensure that references are representative.

---

NOTE:

1. Stata code for the analysis of the Hardt et al. 2019 data:

tab female1 if year<1970

tab female1 if year>=1970 & year<1980

tab female1 if year>=1980 & year<1990

tab female1 if year>=1990 & year<2000

tab female1 if year>=2000 & year<2010

tab female1 if year>=2010 & year<2019

 

tab female1

tab female1 if year<2000

di 36791/87398

Tagged with: ,

By Posted on Posted in Sex No Comments

"The Gender Readings Gap in Political Science Graduate Training" by Heidi Hardt, Amy Erica Smith, Hannah June Kim, and Philippe Meister was recently published in the Journal of Politics and featured in a Monkey Cage blog post. The Kim Yi Dionne header for the Monkey Cage post indicated that:

Throughout academia, including in political science, women haven't achieved parity with men. As this series explores, implicit bias holds women back at every stage, from the readings professors assign to the student evaluations that influence promotions and pay, from journal publications to book awards.

The abstract to the JOP article indicates that "Introducing a unique data set of 88,673 citations from 905 PhD syllabi and reading lists, we find that only 19% of assigned readings have female first authors". This 19% for assigned readings is lower than the 21.5% of publications in the top three political science journals between 2000 and 2015 (bottom of page 2 of the JOP article). However, the 19% is based on assigned readings published at any time in history, including authors such as Plato and Sun Tzu. My analysis of the data for the article indicated that 22% of assigned readings have female first authors when the assigned readings are limited to assigned readings published between 2000 and 2015 inclusive. The top three publications benchmark therefore produces an estimate of the gender readings gap in political science graduate training for 2000 to 2015 publications that is less than one percent and trivially advantages women.

Figure 1 in the Hardt et al. JOP article reports percentages by subfield, with benchmarks for published top works, which I think are articles in top 10 journals; the first and third numeric columns in the table below are data reported in Figure 1. Using the benchmark for published top works, my analysis limiting the assigned readings to assigned readings published between 2000 to 2015 inclusive (the middle numeric column) produced a difference greater than 1% that disadvantaged female first authors for only one of the five subfields with benchmark data (comparative politics):

Topic% Female
1st Author
Readings
(All Time)		% Female
1st Author
Readings
(2000-2015)		% Female
1st Author
Top Pubs
(2000-2015)
Methodology 11.5713.6411.36
Political Economy 16.7518.03 NA
American 15.6618.46 19.07
Comparative 20.5523.26 28.76
IR 19.9623.41 22.42
Theory 25.0531.58 29.39

For an example topic most relevant to my work, the Hardt et al. Figure 1 gender gap for American politics is 3.41 percentage points (15.66 compared to 19.07), but falls to 0.61 percentage points (18.46 compared to 19.07) when the time frame of the assigned readings is set to the 2000-2015 time frame of the top publications benchmark. Invocation of an implicit bias that holds back women might be premature if the data indicate a gap of less than 1 percentage point in an analysis that does not include relevant control variables such as any gender gap in how "syllabus-worthy" publications are within the set of top publications. The 5.50 percentage point gender gap for comparative politics might be large enough to consider implicit bias in that subfield, but that's a localized concern.

---

NOTES

1. [*] The post title alludes to this tweet.

2. The only first authors coded female before 1776 are Titus Livy and Sun Tzu (tab surname1 if female1==1 & year<1776).

3. Code below:

* Insert this command into the Hardt et al. do file after Line 11 ("use 'Hardt et al. JOP_Replication data.dta', clear"):
keep if year>=2000 & year<=2015

* Insert these commands into the Hardt et al. do file after new Line 124 ("tab1 gender1 if gender1 < 3 [aweight=wt] // THE TOPLINE RESULTS WE REPORT EXCLUDE THOSE 304 OBSERVATIONS"):
tab1 gender1 if gender1 < 3 [aweight=wt] // This should report 21.86%
tab1 gender1 if gender1 < 3 // This should report 22.20%

* Insert this command into the Hardt et al. do file before new Line 184 ("restore"):
tab topic mn

* Run the Hardt+et+al.+JOP_Replication+code-1.do file until and including new Line 126 ("tab1 gender1 if gender1 < 3 // This should report 22.20%"). These data indicate that, of first authors coded male or female, about 22% were female.

* Run new Line 127 to new Line 184 ("tab topic mn"). Line 184 should output data for the middle column in the table in this post. See the "benchmark_teelethelen" lines for data for the right column in the table.

Tagged with: ,

By Posted on Posted in Race 4 Comments

Ethnic and Racial Studies recently published "Revisiting the Asian Second-Generation Advantage", by Van C. Tran, Jennifer Lee, and Tiffany J. Huang, which I will refer to below as Tran et al. 2019. Ethnic and Racial Studies has also published my comment, and a Tran et al. response. I'll reply to their response below...

---

Here are three findings from Tran et al. 2019 important for the discussion below:

1. Table 2 indicates that U.S. second-generation Chinese, Indians, Filipinos, Vietnamese, and Koreans are more likely than native Whites to hold a college degree.

2. Table 2 indicates that U.S. second-generation Chinese, Indians, Filipinos, Vietnamese, and Koreans are more likely than native Whites to report being in a managerial or professional position.

3. Table 4 Model 1 does not provide evidence at p<.05 that U.S. second-generation Chinese, Indians, Filipinos, Vietnamese, or Koreans are less likely than native Whites to report being in a managerial or professional position, controlling for age, age squared, gender, region, survey year, and educational attainment.

---

Below, I'll respond to what I think are the two key errors in the Tran et al. reply.

1.

From the first paragraph of the Tran et al. reply:

Given this Asian educational advantage, we hypothesized that second-generation Asians would also report an occupational advantage over whites, measured by their likelihood to be in a professional or managerial occupation.

It makes sense to expect the second-generation Asian educational advantage to translate to a second-generation Asian occupational advantage. And that is what Tran et al. 2019 Table 2 reported: 45% of native Whites reported being in a professional or managerial position, compared to 73% of second-generation Chinese, 79% of second-generation Indians, 52% of second-generation Filipinos, 53% of second-generation Vietnamese, and 60% of second-generation Koreans. Tran et al. 2019 even commented on this occupational advantage: "Yet despite variation among the second-generation Asian groups, each exhibits higher rates of professional attainment than native-born whites and blacks" (p. 2260). But here is the Tran et al. reply following immediately from the prior block quote:

Contrary to our expectation, however, we found that, with the exception of second-generation Chinese, the other four Asian ethnic groups in our study – Indians, Filipinos, Vietnamese and Koreans – report no such advantage in professional or managerial attainment over whites (Tran, Lee, and Huang 2019: Table 4, Model 1). More precisely, the four Asian ethnic groups are only as likely as whites to be in a managerial or professional occupation, controlling for age, the quadratic term of age, gender, education, and region of the country.

The finding contrary to the Tran et al. expectation (from Tran et al. 2019 Table 4 Model 1) was not from what the other four Asian ethnic groups reported but was from a model predicting what was reported controlling for educational attainment and other factors. Tran et al. therefore expected an educational advantage to cause an occupational advantage that remained after controlling for the educational advantage. The Tran et al. reply states this expressly (p. 2274, emphasis in the original):

Because second-generation Asians hold such a significant educational advantage over whites, we had expected that second-generation Asians would also report an occupational advantage over whites, even after controlling for respondents' education.

Properly controlling for a factor means to eliminate the factor as an explanation. For instance, men having a higher average annual salary than women have might be due to men working more hours on average per year than women work. Comparing the average hourly salary for men to the average hourly salary for women controls for hours worked and eliminates the explanation that the any residual gender difference in average annual salary is due to a gender difference in hours worked per year. The logic of the Tran et al. expectation applied to the gender salary gap would produce expectations such as: Because men work more hours on average than women work, we expected that men would have a higher average annual salary than women have, even after controlling for the fact that men work more hours on average than women work.

---

2.

From the Tran et al. reply (p. 2274, emphasis added):

Given that second-generation Asians are more likely to have graduated from college than whites, we hypothesized that they would evince a greater likelihood of attaining a professional or managerial position than whites, as is the case for the Chinese. Instead, we found that second-generation Chinese are the exception, rather than the norm, among second-generation Asians. Hence, we concluded that second-generation Asians are over-credentialed in education in order to achieve parity with whites in the labor market.

I think that there are two ways that labor market parity can be properly conceptualized in the context of this analysis. The first is for labor market outcomes for second-generation Asians to equal labor market outcomes for native Whites, without controlling for any factors; the second is for labor market outcomes for second-generation Asians to equal to labor market outcomes for native Whites, controlling for particular factors. Tran et al. appear to be using the "controlling for" conceptualization of parity. Now to the bolded statement...

Ignoring the advantage for second-generation Chinese, and interpreting as parity insufficient evidence of a difference in the presence of statistical control, Tran et al. 2019 provided evidence that second-generation Asians are over-credentialed in education relative to native Whites *and* that second-generation Asians have achieved labor market parity with native Whites. But I do not see anything in the Tran et al. 2019 analysis or reply that indicates that second-generation Asians need to be over-credentialed in education "in order to achieve" this labor market parity with native Whites.

Returning to the gender salary gap example, imagine that men have a higher average annual salary than women have, but that this salary advantage disappears when controlling for hours worked, so that men have salary parity with women; nothing in that analysis indicates that men need to overwork in order to achieve salary parity with women.

---

So I think that the two key errors in the Tran et al. reply are:

1. The expectation that the effect of education will remain after controlling for education.

2. The inference from their reported results that second-generation Asians need to be over-credentialed in order to achieve labor market parity with natives Whites.

Tagged with: , ,

By Posted on Posted in Race No Comments

The 2018 CCES (Cooperative Congressional Election Study, Schaffner et al. 2019) has two items to measure respondent sexism and, in the same grid, two items to measure respondent racism, with responses measured on a five-point scale from strongly agree to strongly disagree:

  • White people in the U.S. have certain advantages because of the color of their skin.
  • Racial problems in the U.S. are rare, isolated situations.
  • When women lose to men in a fair competition, they typically complain about being discriminated against.
  • Feminists are making entirely reasonable demands of men.

The figure below reports the predicted probability of selecting the more liberal policy preference (support or oppose) on the CCES's four environmental policy items, weighted, limited to White respondents, and controlling for respondents' reported sex, age, education, partisan identification, ideological identification, and family income. Blue columns indicate predicted probabilities when controls are set to their means and respondent sexism and racism are set to their minimum values, and black columns indicate predicted probabilities when controls are set to their means and respondent sexism and racism are set to their maximum values.

Rplot01

Below are results replacing the two-item racism measure with the traditional four-item racial resentment measure:

rresent

One possibility is that these strong associations are flukes; but similar patterns appear for the racism items on the 2016 CCES (the 2016 CCES did not have sexism items).

If the strong associations above are not flukes, then I think three possibilities remain: [1] sexism and racism combine to be a powerful *cause* of environmental policy preferences among Whites, [2] this type of associational research design with these items cannot be used to infer causality generally speaking, and [3] this type of associational research design with these items cannot be used to infer causality about environmental policy preferences but could be used to infer causality about other outcome variables, such as approval of the way that Donald Trump is handling his job as president.

If you believe [1], please post in a comment below a theory about how sexism and racism cause substantial changes in these environmental policy preferences. If you believe [3], please post in a comment an explanation why this type of associational research design with these items can be used to make causal inferences for only certain outcome variables and, if possible, a way to determine for which outcome variables a causal inference could be made. If I have omitted a possibility, please also post a comment with that omitted possibility.

NOTES

Stata code.

Tagged with: ,

By Posted on Posted in Uncategorized No Comments

The Kearns et al. study "Why Do Some Terrorist Attacks Receive More Media Attention Than Others?" has been published in Justice Quarterly; the abstract indicates that "Controlling for target type, fatalities, and being arrested, attacks by Muslim perpetrators received, on average, 357% more coverage than other attacks". A prior Kearns et al. analysis was reported on in a 2017 Monkey Cage post and a paper posted at SSRN with a "last edited" date of 3/5/17 limited to "media coverage for terrorist attacks in the United States between 2011 and 2015" (p. 7 of the paper).

Data for the Kearns et al. study published in Justice Quarterly has been expanded to cover terrorist attacks from 2006 to 2015 (instead of 2011 to 2015) and now reports a model with a predictor for "Perpetrator and group unknown", with a p-value under 0.05 for the Muslim perpetrator predictor. Footnote 9 of Kearns et al. 2019 discusses selection of 2006 as the starting point:

Starting in 2006, an increasing percentage of Americans used the Internet as their main source of news [URL provided, but omitted in this quote]. Since the news sources used for this study include both print and online newspaper articles, we started our analysis in 2006. In years prior to 2006, we may see fewer articles overall since print was more common and is subject to space constraints (p. 8).

That reason to start the analysis in 2006 does not explain why the analysis in the Monkey Cage post and the 3/5/17 paper started in 2011, given that the news sources in these earlier reports of the study also included both print and online articles.

In this 3/28/17 post, I reported that the Muslim perpetrator predictor had a 0.622 p-value in my analysis predicting the number of articles of media coverage using the Kearns et al. 2011-2015 outcome variable coding, controlling for the number of persons killed in the attack and for whether the perpetrator was unknown.

Using the 2006-2015 dataset and code that Dr. Kearns sent me upon request, I ran my three-predictor model, limiting the analysis to events from 2011 to 2015:

Kearns1The above p-value for the Muslim perpetrator predictor differs from my 0.622 p-value from the prior post, although inferences are the same. There might be multiple reasons for the difference, but the 3/5/17 Kearns et al. paper reports a different number of articles for some events; for example, the Robert Dear event was coded as 204 articles in the paper and as 178 articles in the 2019 article, and the number of articles for the Syed Rizwan Farook / Tashfeen Malik event dropped from 179 to 152.

---

The inference about the Muslim perpetrator predictor is more convincing using the 2006-2015 data from Kearns et al. 2019 than from the 2011-2015 data: the 2006-2015 data produce a 2.82 Muslim perpetrator predictor t-score using my three-predictor model above and a 4.20 t-score with a three-predictor model replacing the number killed in the event with a predictor for whether someone was killed in the event.

For what it's worth, along with higher-than-residual news coverage for events with Muslim perpetrators, the Kearns et al. data indicate that, compared to other events with a known perpetrator, events with Muslim perpetrators also have higher-than-residual numbers of deaths, numbers of logged wounded, and (at least at p=0.0766) likelihood of a death:

Kearns2Kearns3Kearns4---

NOTES

1. I could not find the 3/5/17 Kearns et al. paper online now, but I have a PDF copy from SSRN (SSRN-id2928138.pdf) that the above post references.

2. Stata code for my analyses:

gen PerpUnknown=0
replace PerpUnknown=1 if eventid==200601170007
replace PerpUnknown=1 if eventid==200606300004
replace PerpUnknown=1 if eventid==200607120007
replace PerpUnknown=1 if eventid==200705090002
replace PerpUnknown=1 if eventid==200706240004
replace PerpUnknown=1 if eventid==200710200003
replace PerpUnknown=1 if eventid==200710260003
replace PerpUnknown=1 if eventid==200802170007
replace PerpUnknown=1 if eventid==200803020012
replace PerpUnknown=1 if eventid==200803060004
replace PerpUnknown=1 if eventid==200804070005
replace PerpUnknown=1 if eventid==200804220011
replace PerpUnknown=1 if eventid==200806140008
replace PerpUnknown=1 if eventid==200807250030
replace PerpUnknown=1 if eventid==200903070010
replace PerpUnknown=1 if eventid==200909040003
replace PerpUnknown=1 if eventid==201007270013
replace PerpUnknown=1 if eventid==201011160004
replace PerpUnknown=1 if eventid==201101060018
replace PerpUnknown=1 if eventid==201102220009
replace PerpUnknown=1 if eventid==201104230010
replace PerpUnknown=1 if eventid==201105060004
replace PerpUnknown=1 if eventid==201109260012
replace PerpUnknown=1 if eventid==201110120003
replace PerpUnknown=1 if eventid==201205200024
replace PerpUnknown=1 if eventid==201205230034
replace PerpUnknown=1 if eventid==201208120012
replace PerpUnknown=1 if eventid==201301170006
replace PerpUnknown=1 if eventid==201302260036
replace PerpUnknown=1 if eventid==201304160051
replace PerpUnknown=1 if eventid==201304170041
replace PerpUnknown=1 if eventid==201304180010
replace PerpUnknown=1 if eventid==201307250065
replace PerpUnknown=1 if eventid==201308220053
replace PerpUnknown=1 if eventid==201403180089
replace PerpUnknown=1 if eventid==201403250090
replace PerpUnknown=1 if eventid==201406110089
replace PerpUnknown=1 if eventid==201410030065
replace PerpUnknown=1 if eventid==201410240071
replace PerpUnknown=1 if eventid==201411040087
replace PerpUnknown=1 if eventid==201502170127
replace PerpUnknown=1 if eventid==201502230104
replace PerpUnknown=1 if eventid==201503100045
replace PerpUnknown=1 if eventid==201506220069
replace PerpUnknown=1 if eventid==201506230056
replace PerpUnknown=1 if eventid==201506240051
replace PerpUnknown=1 if eventid==201506260046
replace PerpUnknown=1 if eventid==201507150077
replace PerpUnknown=1 if eventid==201507190097
replace PerpUnknown=1 if eventid==201508010105
replace PerpUnknown=1 if eventid==201508020114
replace PerpUnknown=1 if eventid==201508190040
replace PerpUnknown=1 if eventid==201509040048
replace PerpUnknown=1 if eventid==201509300082
replace PerpUnknown=1 if eventid==201512260016
tab PerpUnknown, mi
tab PerpUnknown PerpMuslim, mi
tab PerpUnknown PerpNonMuslim, mi
tab PerpUnknown PerpGroupUnknown, mi
nbreg TOTALARTICLES PerpMuslim numkilled PerpUnknown if eventid>=201101060018
nbreg TOTALARTICLES PerpMuslim numkilled PerpUnknown
gen kill0=0
replace kill0=1 if numkilled==0
tab numkilled kill0
nbreg TOTALARTICLES PerpMuslim kill0     PerpUnknown
ttest numkilled if PerpUnknown==0, by(PerpMuslim)
ttest numkilled                  , by(PerpMuslim)
ttest logwound  if PerpUnknown==0, by(PerpMuslim)
ttest logwound                   , by(PerpMuslim)
prtest kill0    if PerpUnknown==0, by(PerpMuslim)
prtest kill0                     , by(PerpMuslim)

3. Kearns et al. 2019 used a different "unknown" perpetrator measure than I did. My PerpUnknown predictor (in the above analysis and the prior post) coded in a dichotomous variable as 1 any perpetrator listed as "Unknown" in the Kearns et al. list. Kearns et al. 2019 has a dichotomous PerpGroupUnknown variable that differentiated between perpetrators in which the group of the perpetrator was known (such as for this case with an ID of 200807250030 in the Global Terrorism Database, in which the perpetrators were identified as Neo-Nazis) and perpetrators in which the group of the perpetrator was unknown (such as for this case with an ID of 200806140008 in the Global Terrorism Database, in which the perpetrator group was not identified). Kearns et al. 2019 footnote 17 indicates that "Even when the individual perpetrator is unknown, we often know the group responsible so 'perpetrator unknown' is not a theoretically sound category on its own, though we account for these incidents in robustness checks"; however, I'm not sure why "perpetrator unknown" is not a theoretically sound category on its own for the purpose of a control when predicting media coverage: if a perpetrator's name is not known, then there might be fewer news articles because there will be no follow-up articles that delve into the background of the perpetrator in a way that could be done if the perpetrator's name were known.

Tagged with: ,

By Posted on Posted in Academia 6 Comments

[Please see the March 13, 2019 update below]

Studies have indicated that there are more liberals than conservatives in the social sciences (e.g., Rothman et al. 2005, Gross and Simmons 2007). If social scientists on average are more likely to cite publications that support rather than undercut their assumptions about the world and/or are more likely to cite publications that support rather than undercut their policy preferences, then it is reasonable to expect that, all else equal, publications reporting findings that support liberal assumptions or policy preferences will receive a higher number of citations than publications reporting findings that undercut liberal assumptions or policy preferences.

---

Here is a sort-of natural experiment to assess this potential ideological citation bias. From an April 2015 Scott Alexander post at Slate Star Codex (paragraph breaks omitted):

Williams and Ceci just released National Hiring Experiments Reveal 2:1 Faculty Preference For Women On STEM Tenure Track, showing a strong bias in favor of women in STEM hiring...Two years ago Moss-Racusin et al released Science Faculty's Subtle Gender Biases Favor Male Students, showing a strong bias in favor of men in STEM hiring. The methodology was almost identical to this current study, but it returned the opposite result. Now everyone gets to cite whichever study accords with their pre-existing beliefs.

It has been more than three years since that Slate Star Codex post, so let's compare the number of citations received by the article with the finding that supports liberal assumptions or policy preferences (Moss-Racusin et al. 2012) to the number of citations received by the article with the finding that undercuts liberal assumptions or policy preferences (Williams and Ceci 2015). Both articles were published in the same journal, and both articles have a mixed-sex authorship team with a woman as the first author, and both of these factors help eliminate a few alternate explanations for any difference in citation counts to the articles.

Based on Web of Science data collected August 24, 2018, Moss-Racusin et al. 2012 has been cited these numbers of times in the given year, with the number of years from the article's publication year in square brackets:

  • 5 in 2012 [0]
  • 39 in 2013 [1]
  • 74 in 2014 [2]
  • 109 in 2015 [3]
  • 111 in 2016 [4]
  • 131 in 2017 [5]
  • 105 in 2018 to date [6]

Based on Web of Science data collected August 24, 2018, Williams and Ceci 2015 has been cited these numbers of times in the given year, with the number of years from the article's publication year in square brackets:

  • 4 in 2015 [0]
  • 21 in 2016 [1]
  • 27 in 2017 [2]
  • 15 in 2018 to date [3]

So, in the second year from the article's publication year, Williams and Ceci 2015 was cited 27 times, and Moss-Racusin et al. 2012 was cited 74 times. Over the first three years, Williams and Ceci 2015 was cited 52 times, and Moss-Racusin et al. 2012 was cited 118 times.

---

The potential citation bias against research findings that undercut liberal assumptions or policy preferences might be something that tenure-and-promotion committees should be aware of. Such a citation bias would also be relevant for assessing the status of the journal that research is published in and whether research is even published. Suppose that a journal editor were given a choice of publishing either Moss-Racusin et al. 2012 or Williams and Ceci 2015. Based on the above data, an editor publishing Williams and Ceci 2015 instead of Moss-Racusin et al. 2012 would, three years in, be forfeiting roughly 66 citations to an article in their journal (118 minus 52). Editors who prefer higher impact factors for their journal might therefore prefer to publish a manuscript with research findings that support liberal assumptions or policy preferences, compared to an equivalent manuscript with research findings that undercut liberal assumptions or policy preferences.

---

NOTES

1. Williams and Ceci 2015 was first published online or in print earlier in the year (April 8, 2015) than Moss-Racusin et al. 2012 (Sept 17, 2012), so this earlier publication date in the publication year for Williams and Ceci 2015 should bias upward citations in the publication year or in a given year from the publication year for Williams and Ceci 2015 relative to Moss-Racusin et al. 2012, given that Williams and Ceci 2015 had more time in the publication year to be cited.

2. There might be non-ideological reasons for Moss-Racusin et al. 2012 to be enjoying a 2:1 citation advantage over Williams and Ceci 2015, so comments are open for ideas about any such reasons and for other ideas on this topic. The articles have variation in the number of authors—2 for Williams and Ceci 2015, and 5 for Moss-Racusin et al. 2012—but that seems unlikely to me to be responsible for the entire citation difference.

3. Some of my publications might be considered to fall into the category of research findings that undercut liberal assumptions or policy preferences.

---

UPDATE (Nov 30, 2018)

Here is another potential article pair:

The 1996 study about items measuring sexism against women was published earlier and in a higher-ranked journal than the 1999 study about items measuring sexism against men, but there is to date an excess of 1,238 citations for the 1996 study, which I suspect cannot be completely assigned to the extra three years in circulation and the journal ranking.

---

UPDATE (Mar 13, 2019)

Lee Jussim noted that Moss-Racusin et al. (2012) has been cited much more often than Williams and Ceci (2015) has been (and note the differences in inferences between articles), before I did. Lee's tweet below is from May 28, 2018:

https://twitter.com/PsychRabble/status/1001250104676929542

Tagged with: