Sexism is a major problem in the sciences, technology, engineering and medicine, and the first step to fix this problem is awareness.

Yet a new study suggests awareness is exactly what many men lack.

Moss-Racusin et al write in Psychology of Women Quarterly that among 423 respondents in an online forum about articles on sexism in STEM fields, men were more likely to suggest sexism isn't a problem even when confronted with clear evidence to the contrary.

Let that sink in a little.

Men sure are clueless, aren't they. Let's get out the cattle prods.

BZZT!

Or... maybe not.

A closer reading of the study reveals the way it was conducted could lead to bias. Not bias in the sense of being sexist, but bias in the statistical sense of the word, i.e. that the authors' conclusions may be invalid.

Here's why.

It turns out the authors collected 831 responses from an online forum regarding articles on sexism in STEM fields. We don't know who responded, and how they might differ from those who didn't. Were non-sexist men less likely to opine, leaving the sexist trolls to put the dark side of their humanity on full display? Probably - but we just can't really know because the information is unavailable.

That's not all. From out of those 831 responses, the authors analyzed data only from the 423 in which the respondent's sex could be inferred from their response. If the respondent said, "As a man, I am aggrieved that women are scientists but there is no evidence of sexism," then their rabidly sexist and clueless response was included in the study. If a man wrote, "Yay women in science - but they face sexism all the time" without indicating his sex, his more enlightened response was thrown out.

That's right: the authors did not systematically evaluate the major variable of interest - respondent sex - in an unbiased way. As a result their analyses were potentially biased to include people who mention their sex when writing about sexism. If, for instance, male sexists are more likely to mention that they are men, then the study conclusions are completely invalid.

The authors do cop to some of the study limitations, but not this one. They write, regarding study limitations, that the study approach,

necessitated the usage of a non-random, self- selected sample of commenters. ... Commenters were limited to those who had access to the Internet, were sufficiently interested in bias in STEM to read an article on the topic, read one of our three selected articles, and decided to leave a comment. As such, although our sample may be representative of the focal underlying population of individuals who are interested in such issues, as well as read and comment on such articles, these commenters may not be representative of other groups.

True enough. As a man in STEM, I hope they aren't representative of me!

But more centrally, can the authors really conclude they know how aware men (or even men online) are of sexism in STEM if they do not even know how many of the original 831 respondents are men? (Not to mention the full group of men who read the articles in question in the first place.) Similarly, how can they conclude that men are more likely to be skeptical of sexism in STEM fields if they cannot compare all men's responses to all women's responses.

Short answer: they can't.

What is sad is that the authors are able to cite a bevy of other articles that DO suggest some men in STEM fields are sexist, and that some men deny that sexism exists. I don't doubt it: I've seen both, and wouldn't argue otherwise.

Here's a random sampling of anonymized Twitter commentary about the findings:

Reading this I realize that stepping into this issue could leave me with egg on my face or something stinky on my shoe.

I would guess that the lived experience of these and other commentators - like mine, let me emphasize, wait, here are some emphases: like mine - supports the idea that some men in STEM disregard sexism even when it's staring them in the face.

With such preexisting beliefs - realities! - appear to be confirmed by a study like this one, it's easy to turn off our critical thinking skills and trot it out for the world to see. The risk of course is that the clear problems with study structure, platinum-plated as the authors' intentions surely are, come to light, the ensuing discussion can undermine the true message we were trying to convey.

We should call out and fight sexism in STEM fields. It is pernicious, archaic and it must go. While fighting that noble fight - which we are winning by the way - we should not undermine the effort by basing conclusions on flawed data.

This blog was cross-posted at KevinMD.

Was a study about bias - biased?