Background on the corrections for "Gender Bias in Student Evaluations" and "Comic-Con: Can Comics of the Constitution Enable Meaningful Learning in Political Science?"

In May 2020, PS published a correction to Mitchell and Martin 2018 "Gender Bias in Student Evaluations", which reflected concerns that I raised in a March 2019 blog post. That correction didn't mention me, and in May 2020 PS published another correction that didn't mention me but was due to my work, so I'll note below evidence that the corrections were due to my work, which might be useful in documenting my scholarly contributions for, say, an end-of-the-year review or promotion application.

---

In August 2018, I alerted the authors of Mitchell and Martin 2018 (hereafter MM) to concerns about potential errors in MM. I'll post one of my messages below. My sense at the time was that the MM authors were not going to correct MM (and the lead author of MM was defending MM as late as June 2019), so I published a March 2019 blog post about my concerns and in April 2019 I emailed PS a link to my blog post and a suggestion that MM "might have important errors in inferential statistics that warrant a correction".

In May 2019, a PS editor indicated to me that the MM authors have chosen to not issue a correction and that PS invited me to submit a comment on MM that would pass through the normal peer review process. I transformed my blog post into a manuscript comment, which involved, among other things, coding all open-ended student evaluation comments and calculating what I thought the correct results should be in the main three MM tables. Moreover, for completeness, I contacted Texas Tech University and eventually filed a Public Information Act request, because no one I communicated with at Texas Tech about this knew for certain why student evaluation data were not available online for certain sections of the course that MM Table 4 reported student evaluation results for.

I submitted a comment manuscript to PS in August 2019 and submitted a revision based on editor feedback in September 2019. Here is the revised submitted manuscript. In January 2020, I received an email from PS indicating that my manuscript was rejected after peer review and that PS would request a corrigendum from the authors of MM.

In May 2020, PS published a correction to MM, but I don't think that the correction is complete: for example, as I discussed in my blog post and manuscript comment, I think that the inferential statistics in MM Table 4 were incorrectly based on a calculation in which multiple ratings from the same student were treated as independent ratings.

---

For the Comic-Con correction that PS issued in May 2020, I'll quote from my manuscript documenting the error of inference in the article:

I communicated concerns about the Owens et al. 2020 "Comic-Con" article to the first two authors in November 2019. I did not hear of an attempt to publish a correction, and I did not receive a response to my most recent message, so I submitted this manuscript to PS: Political Science & Politics on Feb 4, 2020. PS published a correction to "Comic-Con" on May 11, 2020. PS then rejected my manuscript on May 18, 2020 "after an internal review".

Here is an archive of a tweet thread, documenting that in September 2019 I alerted the lead "Comic-Con" author to the error of inference, and the lead author did not appear to understand my point.

---

NOTES:

1. My PS symposium entry "Left Unchecked" (published online in June 2019) discussed elements of MM that ended up being addressed in the MM correction.

2. Here is an email that I sent the MM authors in August 2018:

Thanks for the data, Dr. Mitchell. I had a few questions, if you don't mind:

[1] The appendix indicates for the online course analysis that: "For this reason, we examined sections in the mid- to high- numerical order: sections 6, 7, 8, 9, and 10". But I think that Dr. Martin taught a section 11 course (D11) that was included in the data.

[2] I am not certain about how to reproduce the statistical significance levels for Tables 1 and 2. For example, for Table 1, I count 23 comments for Dr. Martin and 45 comments for Dr. Mitchell, for the N=68 in the table. But a proportion test in Stata for the "Referred to as 'Teacher'" proportions (prtesti 23 0.152 45 0.244) produces a z-score of -0.8768, which does not seem to match the table asterisks indicating a p-value of p<0.05.

[3] Dr. Martin's CV indicates that he was a visiting professor at Texas Tech in 2015 and 2016. For the student comments for POLS 3371 and POLS 3373, did Dr. Martin's official title include "professor"? If so, than that might influence inferences about any difference in the frequency of student use of the label "professor" between Dr. Martin and Dr. Mitchell. I didn't see "professor" as a title in Dr. Mitchell's CV, but the inferences could also be influenced if Dr. Mitchell had "professor" in her title for any of the courses in the student comments analysis, or for the Rate My Professors comments analysis.

[4] I was able to reproduce the results for the Technology analysis in Table 4, but, if I am correct, the statistical analysis seems to assume that the N=153 for Dr. Martin and the N=501 for Dr. Mitchell are for 153 and 501 independent observations. I do not think that this is correct, because my understanding of the data is that the 153 observations for Dr. Martin are 3 observations for 51 students and that the 501 observations for Dr. Mitchell are 3 observations for 167 students. I think that the analysis would need to adjust for the non-independence of some of the observations.

Sorry if any of my questions are due to a misunderstanding. Thank you for your time.

Best,

L.J