Today, I’m following up on a short post about the use of the word abrasive in performance reviews for women. Similar discussions of word choice in student evaluations of college professors have been a hot topic in the past week. See Is the Professor Bossy or Brilliant in the New York Times. Or in Inside Higher Ed yesterday:
A law dean last month urged students to stop commenting on female professors’ attire in reviews, noting that they don’t do so in the same way for men.
Professor Benjamin Schmidt provides an interactive chart for viewing the distribution of words used to describe male and female instructors in a range of disciplines based on reviews from RateMyProfessor.com. Use of abrasive, although relatively rare (e.g., appearing twice per million words for accounting instructors), still appears to be linked to gender. The chart plots its use in negative reviews. What’s up with students of criminal justice and political science?
Here are the results for rude. Note that this word is far more commonly used (e.g., between 250 to 850 appearances per million in negative reviews). And it is clearly attributed more often to female than male instructors across nearly every discipline.
I encourage you to do your own searches. And, for those who want to investigate Schmidt’s methodology, he explains details of the sample, etc. on his website.
If the tool itself interests you, it’s called Bookworm. You can use it to explore lexical trends in texts collected by the developers or even in your own.
I saw this (the original post, not your post) yesterday, and my immediate reactions were: How did the author know what gender the students were? And how did the author know what gender the professors really were? (For example, how did the author deal with non-European names? Even European names can have a gender different from what you assume. Jean is one very good example given that I’m from Canada. And given the university I’m from Jan would be another good example.) I am not an expert but I know natural language processing is rive with errors; I just don’t trust the results unless I know the author’s exact methodology.
I went to the actual site and found that both of my points have been addressed (by questions -3 and 3) but not resolved. The author, in fact, agreed that these issues are not resolved and said this was not a “study.”
So call me skeptical, but I am not convinced any of this is valid until I see all the details.
There are also negative expressions, which has been raised in the comments and the author acknowledged that he did not handle. Even without negative modifiers, “genius” alone can be have negative connotations instead of positive.
Hi, Ambrose. You make good points–as usual. To be clear, I did not mean to endorse the results as if this were a peer-reviewed research study. But, in light of my recent post about gender bias in workplace performance reviews, I thought it would be interesting to investigate the student evaluations of professors based on gender.
There is considerable evidence of gender bias in the workplace. For example, in three studies researchers found evidence that people had difficulty seeing women’s behaviors as evidence of leadership–even when they had no such difficulty with the same behaviors performed by men:
Scott, K. A., & Brown, D. J. (2006). Female first, leader second? Gender bias in the encoding of leadership behavior. Organizational behavior and human decision processes, 101(2), 230-242.
As another example, the study below showed that, although job performance ratings (not verbal reviews) favored females, ratings of promotion potential were higher for males:
Roth, P. L., Purvis, K. L., & Bobko, P. (2012). A meta-analysis of gender group differences for measures of job performance in field studies. Journal of Management, 38(2), 719-739.
The difficulty of accurately identifying gender is one of the reasons I included the link to Schmidt’s website (where he discusses methodological issues). The fact that a word like “genius” might be used with either positive or negative meaning, does not negate the bias shown by the fact that the term is applied more for men than women.
It’s interesting stuff. And well worth more thought!
It’s an interesting subject and bears a need for greater scrutiny, but I agree with Ambrose that this “study” hardly shows anything. The tiny sample size for the first word makes it comical to even discuss anything statistical, especially when presented on a chart with exactly the same dimensions for the word with a more appropriate sample size.
If the topic interests you, take a look at Slate’s piece based on research at North Carolina State University, which confirmed previous findings that while “female professors are judged somewhat less harshly if they conform more to female stereotypes, men still get bonus points for showing up male.”
http://www.slate.com/blogs/xx_factor/2014/12/09/gender_bias_in_student_evaluations_professors_of_online_courses_who_present.html
Thanks!