Month: December 2019

Let's continue our discussion of studies in Holman et al. 2019 "Evidence of Bias in Standard Evaluations of Teaching" listed as "finding bias". See here for the first entry in the series and here for other entries.

---

34.

Martin 1984 "Power and Authority in the Classroom: Sexist Stereotypes in Teaching Evaluations" reported on data from 240 female students and 154 male students across nine course at a large Midwestern university. The instructors were three male social scientists, three female social scientists, and three female women's studies faculty. From the article (p. 488):

I made no attempt to match up male and female faculty on the basis of equal teaching skills (an impossible task in any case). I assumed that, if there were statistically significant differences between the ratings of an instructor by male and female students, then I could attribute these differences to sex bias.

Here is a sample from the results section (p. 490):

Results showed that, when male students were evaluating female social science instructors, high ratings on teaching effectiveness were strongly associated with high ratings on friendliness (r2 = 51 percent), smiles (r2 = 61 percent), eye contact (r2 = 61.6 percent), confidence (r2 = 62.9 percent), and decisiveness (r2 = 72.6 percent). I found no similarly strong associations in male students' ratings of male instructors or of women's studies instructors. No strong associations between teaching effectiveness and personal traits appeared when I examined female students' ratings.

This is interpreted as a bias (p. 492):

...more than their male colleagues, female instructors are likely to have their competence judged by male students on the basis of personal characteristics associated with feminine behavior, such as friendliness, frequent eye contact, and regular smiles. The message to women seems clear: if your institution bases personnel decisions on student evaluations, make sure your colleagues are aware of the possibility of sex bias.

I'm not sure that this sort of bias favors men over women, if the results mean that women social science instructors can improve their student evaluations by smiling and being friendlier but men social science instructors can't improve their student evaluations by smiling and being friendlier.

---

35.

Schuster and Van Dyne 1985 "The Changing Classroom" is an essay that reports no novel data. I'm not sure that it is properly placed in a list of studies "finding bias".

---

36.

Basow and Silberg 1987 "Student Evaluations of College Professors: Are Female and Male Professors Rated Differently?" reported responses from 553 male students and 527 female students at a small private college. Of the 22 female professors at the college who had taught at the college full-time at least one year, 16 were matched to a male professor by rank, division, and years of experience at the college.

Results indicated, among other things, that "...male students gave female professors significantly (p < .05) less positive ratings than they gave male professors on all dependent measures" and that "female students rated female professors significantly more negatively than they rated male professors on Instructor-Individual Student Interaction (p < .01), Dynamism/Enthusiasm (p < .05), and overall teaching ability (p < .01)" (p. 310).

However, Basow and Silberg 1987 notes that "...male professors may in fact be better overall teachers than female professors" (p. 313). Moreover (p. 313):

...the magnitude of the effect sizes is quite small, indicating that sex of instructor and sex of student account for only a small percentage of the variance in student ratings. For example, the combination of knowledge of teacher sex and student sex can predict only about 4% of the variance in scores on overall teaching ability.

---

Comments are open if you disagree, but I don't think that data from the 1980s or earlier are relevant for discussions of whether student evaluations of teaching should be used in employment decisions made in 2019 or beyond.

Let's continue our discussion of studies in Holman et al. 2019 "Evidence of Bias in Standard Evaluations of Teaching" listed as "finding bias". See here for the first entry in the series and here for other entries.

---

31.

Bray and Howard 1980 "Interaction of Teacher and Student Sex and Sex Role Orientations and Student Evaluations of College Instruction" selected 6 instructional faculty at the College of Social Sciences at the University of Houston from each of the following categories: feminine man, androgynous man, masculine man, feminine woman, androgynous woman, and masculine woman. For each of these 36 faculty members, one of their undergraduate courses was given during the final week of classes the IDEA questionnaire for measuring student evaluation of teaching and the Bem Sex Role Inventory (BSRI).

Bray and Howard 1980 indicate that the IDEA had 40 items across four parts: 20 instructor items, 10 student-rated progress items, 3 course evaluation items, and 7 items for the student self-rating of the course (p. 243). The article indicates that "During the last full week of classes (prior to final examination week) the appropriate students in each target class completed BSRI and IDEA questionnaires" (p. 244) and that "Two sections of IDEA were employed as outcome criteria: (1) the average of the responses on Part 2: Student Rated Progress (progress), and (2) student satisfaction with the instructor (satisfaction; IDEA question 37)" (p. 243). From what I can tell, the article thus uses only 11 of the 40 measured IDEA items and does not report results for any of the 20 instructor items from the first part of the IDEA questionnaire.

Results indicate that "Androgynous teachers received somewhat higher student evaluations than did masculine and feminine teachers" (p. 246), but there is no statistical control to assess whether any differences are due to differences in teaching styles or other factors.

---

32.

Bennett 1982 "Student Perceptions of and Expectations for Male and Female Instructors: Evidence Relating to the Question of Gender Bias in Teaching Evaluation" reported on data from 253 liberal arts college students in nonscience introductory courses. Students were instructed to respond to a self-returned questionnaire about a specific course.

Results in Table 2 indicate that, of the four factors in questionnaire responses, female instructors had higher ratings than did male instructors for non-authoritarian interpersonal style and charisma (potency) and did not differ at p<0.05 from male instructors on the self-assurance factor or the instructional approach factor.

Students reported more scheduled office visits for female instructors than for male instructors and were more likely to report feeling free to contact a female instructor at home than to contact a male instructor at home.

Results also suggested that the explanatory power of certain predictors of performance ratings differed for female instructors and for male instructors. From page 176:

Additionally, a highly structured instructional approach—described by students as communicating greater professionalism—was consistently more important for women's performance ratings than for men's. This was especially true for students' ratings of instructor's organization, clarity, and coherence in classroom presentation (rs = .76 and .53, for instructional approach for women and men, respectively), command of material for classroom presentation (rs = .49 and .27), and overall evaluation (rs = .57 and .30). Although in the collective student mind men and women do not differ in instructional approach, students are clearly more tolerant of what they perceive as a lack of formal professionalism in the conduct of teaching from their male professors, demanding of women a higher standard of formal preparation and organization.

Moreover, from pages 177 and 178:

...male instructors are judged independently of students' personal experiences of contact and access, whereas female instructors are judged far more closely in this regard. In this sense women are negatively evaluated when they fail to meet this gender appropriate expectation (and rewarded when they do so), although this study cannot provide evidence of the obverse—whether women are devalued for achievement in stereotypically masculine domains.

The "(and rewarded when they do so)" reflects how evidence that that explanatory power of the predictors of performance ratings differs for female instructors and for male instructors is mixed evidence of unfair bias in these performance ratings. If it is true that, for example, organization matters more for the performance ratings of female instructors than for male instructors, that might merely mean that female instructors have more ability to influence their performance ratings through their level of organization. That's not obviously worse to me than, say, male instructors not being rewarded for higher levels of organization. For a recent discussion of a related finding, here is a passage from Hesli et al. 2012 (p. 486, emphasis in the original):

Another critical finding that we note with some consternation is that among women, the probability of being an associate professor over an assistant professor is unrelated (given other controls) to the total number of publications (model 4D). This confirms that the promotion process at this level involves different dynamics for men as compared with women.

I'm not sure that the Hesli et al. 2012 results indicate that there is a p<0.05 difference in the explanatory power of publications. The coefficient for publications is statistically significant for men and not for women, but that does not necessarily indicate that a difference between the coefficients can be inferred. For what it's worth, the "publications" logit coefficient for men is 0.713 with a 0.230 standard error and for women is 0.303 with a 0.453 standard error.

---

33.

Brooks 1982 "Sex Differences in Student Dominance Behavior in Female and Male Professors' Classrooms" isn't a study about student evaluations of teaching. Rather, the study involved analyzing behaviors of first-year students in a master's of social work program, such as student frequency of speaking in class, student frequency interrupting a fellow student, and student frequency interrupting the professor.

Results indicated that "...male students interrupted both male and female professors significantly more often than female students (p < .01 and p < .001, respectively), and interrupted female professors significantly more often than male professors (p < .001)" (p. 687). Interruptions by female students were more evenly distributed than interruptions by male students: of all female interruptions of professors, 45 percent were of female professors; of all male interruptions of professors, 74 percent were of female professors.

---

Comments are open if you disagree, but I don't think that data from the early 1980s is relevant for discussions of whether student evaluations of teaching should be used in employment decisions made in 2019 or beyond.

From what I can tell, the main evidence of bias in student evaluations of teaching to be concerned with in these three studies is from Bennett 1982. But, from what I can tell, the bias in performance evaluations being better predicted by certain criteria for female instructors than for male instructors doesn't necessarily cause a gender-wide bias. For example, imagine these performance evaluation scores:

2 for a disorganized male instructor

2 for a organized male instructor

1 for a disorganized female instructor

3 for organized female instructor

Organization matters more for the female instructors than for the male instructors, but that particular difference in evaluation criteria does not cause a gender-wide bias. Following Hesli et al. 2012, it could be noted with some consternation that performance evaluation scores for men are unrelated to the instructor's organization.

Let's continue our discussion of studies in Holman et al. 2019 "Evidence of Bias in Standard Evaluations of Teaching" listed as "finding bias". See here for the first entry in the series and here for other entries.

---

28.

I did not locate the text of Hogan 1978 "Review of the Literature: The Evaluation of Teaching in Higher Education", but I'm guessing from the "Review of the Literature" title that Hogan 1978 doesn't report novel data.

---

29.

Kaschak 1978 "Sex Bias in Student Evaluations of College Professors" had 100 seniors or first-year graduate students at San Jose State University (50 male and 50 female) rate fictional professors in business administration, chemistry, home economics, elementary education, psychology, and history, based on descriptions of the professors' teaching methods and practices; each professor had a male name or a female name, depending on the form that a student received. Ratings were reported for 1-to-10 scales for: effective/ineffective, concerned/unconcerned, likeable/not at all likeable, poor/excellent, powerless/powerful, and definitely would/would not take the course.

Male students had mean ratings on each item that were more positive for the male professor than for the female professor. Female students had statistically different mean ratings by professor sex only for indicating that the male professor is more powerful and for indicating a preference for taking the female professor's course.

---

30.

Lombardo and Tocci 1979 "Attribution of Positive and Negative Characteristics of Instructors as a Function of Attractiveness and Sex of Instructor and Sex of Subject" had 120 introductory psychology students (60 male and 60 female) rate a person in a photograph, with the experimental manipulation that the person in the photograph was male or female and was attractive or unattractive. Students were told that the photograph was of Mary Dickson or Andrew Dickson and were told that the person had earned a Ph.D. and had just finished a second year of teaching. Ratings included nine scales (such as from intelligent to not intelligent) and the items "Compared with the faculty members at this college, how would you rate the over-all teaching performance of this instructor?" and "How much would you like to take a course from this faculty member?".

Results indicated that "Each of the dependent measures was analyzed by a 2 (attractive vs unattractive) X 2 (sex of pictured person) X 2 (sex of subject) analysis of variance...A significant main effect was found...for attractive-unattractiveness. The absence of other main effects or interactions indicated that the attractive pictures were rated significantly more attractive" (p. 493) and that "An interaction between the attractiveness of the picture and the sex of the instructor...on the question of how much they would like to take a course from this instructor indicated that all subjects preferred to take a course from a male" (p. 494).

---

Comments are open if you disagree, but I don't think that data from the 1970s is relevant for discussions of whether student evaluations of teaching should be used in employment decisions made in 2019 or beyond. For example, there has been a substantial increase since the 1970s in female representation among students and faculty, which can be plausibly expected to have reduced biases against female college faculty present during the Nixon administration.