At the risk of beating a dead horse, I wanted to venture once more into the contentious subject of the recent survey that has been distributed across the internet, ostensibly to learn more about the beliefs of members of the LDS church with regards to gender roles within the church. Earlier this week, we posted some initial concerns we had about the survey. Yesterday, Dr. Andrew Auman shared his thoughts as well.

The concerns expressed so far have ranged from the biases of the researchers to the construct validity of the questions on the survey. The accusation is that the survey does not objectively measure what it claims to be measuring, that the results will not be analyzed objectively, and that the research in this and other ways fails to meet the standards of rigorous research that good scholarship generally ought to have. All of these concerns are legitimate. I just wanted to contribute a critique of the project from a slightly different point of view.

On Biases and Assumptions

The social sciences have what many have termed “physics envy,” in which it tries to model its methods and assumption off of the physical sciences. The goal of many in the discipline is to conduct the kind of strict, rigorous, empirical research that has given the natural sciences its credibility. Whether it has succeeded in this or not, there is wide disagreement; regardless, it is widely agreed that psychologists are not nearly as reflective on their methods and assumptions as they ought to be.

However, there are other strands of thought within the social sciences — particularly social scientists who attempt to engage in more qualitative research. I am one of those — I have an M.S. in Psychology and I’m currently working towards a PhD in Instructional Technology. My work has focused on the theoretical and philosophical assumptions of social science research, and my research has been purely qualitative in nature. Within the school of thought I generally adhere to, the quest for objective, unbiased, non-perspectival research revealing the raw “facts of the matter” is fiction and wholly unachievable.

This is not merely for practical reasons; rather, I am convinced by the arguments of hermeneutic scholars such as Gadamer, who argue that we cannot escape our biases and assumptions. Such biases and assumption do not keep us from learning about the world, however — they are the launching point for inquiry in the first place. Without prejudices, biases, and assumptions, I wouldn’t even know what to look at, much less how to start going about investigating it. We must start any empirical investigation with pre-empirical assumptions and biases, and there’s simply no escaping it. All research, in this way, is perspectival, and should be perspectival.

As such, perfectly permissible to have biases when conducting research. Even the research of those who take the greatest pains to be objective and unbiased is grounded in philosophical predispositions and assumptions (such as, for example, the assumption that it’s possible to be objective, and that objectivity is the highest measure of sound methodology — an assumption, true or not, that has been disputed by many scholars). In the most mundane of examples, even classroom research that tests the effectiveness of an classroom intervention on test scores of learners assumes from the outset that learning can be measured by classroom tests. Psychological research on the influence of religious institutions on happiness assumes that happiness is a product of external factors, that whatever measure is used in the study adequately operationalizes happiness, and that the self-reported happiness of adherents is a measure by which we can evaluate the value of a religious belief system, etc. See? Assumptions everywhere — true or not, they underly every inquiry in the social sciences.

On Doing Good Research

All research, therefore, is grounded in the researcher’s assumptions about the world, their operationalizations, their methods, the political and social norms of the academic community they associate most with, etc. There’s no way around it. And when doing research about politically and ideologically charged topics such as gender and religion, there’s doubly no way around it. The problem is not so much a lack of objectivity (a largely fictional construct anyways), but a pretense of objectivity. Good researchers recognize their biases, and openly admit them in the course of their research. Good research can be conducted from a biased point of view, so long as said research does not try to claim the legitimacy of objectivity. Instead, it can claim the legitimacy of having been thorough, thoughtful, and honest.

And that’s the problem with this study. They haven’t done anything of that sort. Just saying, “We’re trained academicians and we know what we are doing,” is not enough.  All three of those traits of good research on on trial in this study: Their sampling methods are sloppy (non-thorough); their questions are unthoughtful (otherwise they wouldn’t alienate the population of study); and their perspectives and objectives are hidden from the population of study (presumably to make the survey more palatable to them), and the research is thus dishonest.

I just wanted a perspective on record acknowledging the inevitability of bias in social science research, and the fact that all research is ultimately perspectival, but which still finds this survey, and its sponsoring group, to be severely lacking. Sounding the alarm of “bias” is all well and good — but falls flat for those who see bias as unavoidable. Instead, I want to raise the alarm of bias masquerading as objectivity.

Ethics of Care in Social Science Research

Further, a term that I guarantee that almost all the researchers on the team know is postcolonialism, which is — roughly speaking — a movement in sociology research that critiques (and condemns) the study of indigenous or vulnerable populations from the perspective of Western categories of thought. For example, if a Western research goes to an indigenous African tribe to report on their barbaric practices, the researcher is bound to misunderstand and misrepresent the peoples and their customs (and in ways that ultimately harm the target population, by minimizing and demeaning them in their reports to those with political sway and power).

That’s because you can’t understand a population’s customs, practices, and worldviews unless you view those customs through their eyes. A Western researcher may see backwardness and savagery, but in the eyes of the indigenous population, there is only reason and sense. This is because the researcher has not taken the time to understand the shared norms, practices, and understandings of the community. Many have gone so far to say that only members of the target population should be allowed to conduct such research (I disagree); in either case, post-colonialism is a broad paradigm of research that attempts to treat with extreme care the study of a target population (particularly vulnerable populations), to ensure that researchers do not evaluate the population’s practices, customs, and worldviews from an outsider’s perspective, but instead simply attempts to understand them on their own terms. They try and figure out how the world makes sense to them (the population being researched).

I can almost guarantee that most of the researchers on the project are familiar with and ascribe to these ideals. And, fortunately, the researchers claim membership in the LDS Church; but it’s clear to us, at least, that many of them have adopted foreign metrics by which they evaluate the customs and practices of Church. Many of their participation in Ordain Women makes this clear. And it’s clear that their questions are heavily informed by these evaluative metrics. And so in this sense, they likely violate (without even thinking about it) the cherished ideals of post-colonial sociological research. And in their reports to the media, I believe they will use the research to demean and diminish the practices, customs, and worldviews of the average member — and those reports will be conveyed to those with social and political power and be used, ultimately, in ways that harm the institutional Church and its members. The organizations they affiliate with (such as Ordain Women) already have done this — presented “facts” to the media without any clarifying context, and in ways that seem aimed to misrepresent the average member.

For an example of the subtleties of this, one question implicitly assumes the disputed distinction between “iron rod” Mormons and “Liahona” Mormons — a distinction most members don’t make, but which some dissenters from key Church practices and doctrines frequently make. I guess if we are talking about construct validity, this question might actually measure what it purports to measure: whether someone considers themselves an iron rod or liahona Mormon. But only for respondents who view themselves as either. For those who view the distinction as a false dichotomy, it doesn’t represent their views at all — and the vast majority of members of the Church probably fall in this latter category (after all, the prophet himself rejected the dichotomy as false and spiritually dangerous).

This is an example of misrepresenting the target population by crafting research questions and measures from perspectives foreign to the target population, rather than trying to see the world through their eyes. And this is just one example of many on the survey. Other questions are much less subtle. For example, one question asks if participants have ever seen a woman give up career aspirations to meet cultural expectations. Many members have seen a woman give up career aspirations, but both they and the woman believe it was to meet God’s expectations to start a family and focus on the family (a decision based on prayer study). They don’t see it as a merely cultural expectation at all. So how are they supposed to answer this question? The authors of the study seem to treat it as a merely cultural expectation. No question (except for some long-form answers at the end) tries to get into the minds of the average member and figure out how they see the world (and the clarifying, long-form answers at the end will not stop them from using the multiple choice questions in their results anyways). Ultimately, it seems that the researchers aren’t trying to understand their target population — they are subtly evaluating their target population.

Conclusion

The fact that several of the people on the team have already responded to criticism and feedback in a dismissive way — implying that those who are critical are mainly “anti-intellectuals” whose concerns are grounded entirely in fear — confirms for me my concerns about their use of the survey data. The fact of the matter is that they are already misrepresenting the average member, long before the survey results have been released — they are representing us as anti-intellectual and ignorant people who resist any and all academic attempts to study the LDS population. This is far from the truth — it is because of my love of a good research study that I am critical of theirs. It is because of my love of well-done academics that I am skeptical of theirs.