This is a guest post by Dr. Andrew Auman, who holds B.Sc. and M.Sc. degrees in physics from Utah State University, and a Ph.D. in astrodynamics from the University of Surrey. His research interests include geometric integration, geometric estimation, and attitude and orbital mechanics. He is also a semi-regular contributor to the blog Just An Average Mormon
I.
Recently, a survey by The Mormon Gender Issues Survey Group (TMGISG) has been floating around social media, and I have accepted an invitation to write this guest post as I wanted to weigh in on the discussion surrounding this survey and TMGISG’s approach to their research.
To those unfamiliar with the online dialogue surrounding the research being performed by TMGISG, many individuals are calling into question TMGISG’s research methodologies. The concern is that the wording used in the TMGISG’s survey shows a bias in support of the ordination of women to the priesthood in The Church of Jesus Christ of Latter-Day Saints—whose members are commonly referred to as Mormons. Individuals have also expressed concern in the ambiguity of some questions; e.g., what constitutes a “good Mormon”? And there are further concerns that not only do the answers provided on the survey not reflect the most commonly held views on the topics in question, but that at times the only answers provided contain views with which respondents cannot fully agree mixed in with those views to which they do ascribe. That this is the case is acknowledged in the survey. But as these biases, ambiguities, and false dichotomies could easily be removed by the inclusion of additional choices and/or the rewording of current answers, why was the effort not made?
The purpose of this post is to discuss research ethics, and apophasis is not my intent in the above expression of concerns being brought up in the dialogue elsewhere regarding TMGISG’s research practices. May the interested reader peruse the survey and what has been said on the matter for themselves, thoughtfully reflect, and then draw their own conclusions about its phraseology. I simply mention these issues as they are pertinent to the matter of ethics in research, and will be referred to herein without rehashing them for the sake of brevity—brevity being an admittedly relative term.
Before going any further, in the spirit of open, honest communication let me clear the air about who I am and am not. I am not, for instance, a social scientist. Nor do I claim to be fully apprised of the minutia of research methodologies in the social sciences. However, as a physicist I do perform research and am certainly acquainted with the scientific method. For those who resort to ad hominem in debate as in an ill-founded attempt to bolster their own arguments or attack the arguments of those of differing views, I preemptively state that I do have a Ph.D. I am also a Mormon, and I fully ascribe to the tenets of The Church of Jesus Christ of Latter-Day Saints. Among these tenets is the belief that God calls prophets in our day, that He reveals His will and eternal truths to His prophets, and that when His prophets speak in the official capacity of their calling what they speak represents His will and teachings (if you struggle to understand how a scientist can believe in such things, I suggest reading the post I wrote here). Among the truths revealed by prophets is that “The Lord has directed that only men will be ordained to offices in the priesthood.” Full disclosure: this view stands in contrast to the expressed views of members of TMGISG, who appear to support ordination of women to the priesthood.
II.
This post on research ethics is motivated by a comment made by Dr. Brent D. Beal, a TMGISG member, to J. Max Wilson’s post “What You Should Know About That Mormon Gender Issues Survey”. Pew’s 2011 research Mormons in America found that 90% of self-identifying Mormon women in America do not feel that dedicated women members of the LDS Church should be ordained to the priesthood, and in his comment Dr. Beal claims that Pew’s research “isn’t reliable for all sorts of reasons”. Dr. Beal states:
For example, Pew reports that 90% (or so) of women in the LDS church don’t see gender issues as a problem. At the same time, on any given Sunday, 2/3rds of LDS women (2/3rds of the women that are part of the 15+ million membership number) are absent, and we know that the majority of these women are inactive and/or no longer identify as LDS. We also know that gender issues are one of the primary reasons why women disaffiliate from the church. Putting all this together leads pretty quickly to a contradiction (and it’s a contradiction that can’t be easily resolved without more–and better–data).
Before addressing Dr. Beal’s comment and the reliability of TMGISG’s research methodologies, I first turn to the reliability of Pew’s research. On their website Pew states their code of ethics:
Independence, impartiality, open-mindedness and professional integrity are indispensable to the mission and success of the Pew Research Center. To promote and preserve these values, the center’s Code of Ethics includes the following policies:
Conflicts of Interest
Pew Research employees must avoid conflicts of interest or the appearance of conflicts of interest. They should never engage in any activity that might compromise or appear to compromise the center’s credibility or its reputation for independence or impartiality. All employees are required to seek prior approval from a supervisor before engaging in any activity that may be deemed a potential conflict of interest, including membership in groups, boards and associations that may call into question the center’s credibility or its reputation for impartiality.
Prohibitions on Electioneering
As a nonprofit 501(c)3 organization, Pew Research and all of its employees, when acting in their professional capacity, are prohibited from participating, directly or indirectly, in any political campaign activities on behalf of, or in opposition to, any candidate for public office. In addition, the center has a strict prohibition against partisan political activity by senior staff, even when they are acting in their individual capacity and on their personal time.
Integrity of Research
To ensure that the information we generate is of the greatest value to citizens and policymakers, the center is committed to conducting research in a manner that is impartial, open-minded and meets the highest standards of methodological integrity. We employ only those tools and methods of analysis that, in our professional judgment, are well suited to the research question at hand. We describe our findings and methods accurately and in sufficient detail to permit outsiders to evaluate the credibility of our results. We encourage inquiries about our research methods and practices, and attempt to answer requests for information promptly.
In harmony with the claims that Pew makes under the section “Integrity of Research” quoted above, Pew makes all of its research datasets available to interested researchers. These datasets can be found here generally, and here in regards to the Mormons in America study. Those who wish to download datasets are required to agree to a “Dataset Use Agreement”, but aside from the legal talk the purpose of this agreement is mainly to protect the privacy of research participants and assure that those using the data will never present the Pew Research Center as supporting any side of any debate. See here for more information.
Indeed, Pew is so concerned about maintaining the integrity of the research it performs that it doesn’t even refer to itself as a “think tank”, but rather as a “fact tank”.
To summarize, Pew demonstrates the epitome of ethical research by:
- Having no conflicts of interest. In fact, all Pew employees must even avoid the appearance of a conflict of interest, and seek approval from their supervisor before establishing personal affiliations which may taint Pew’s perceived impartiality.
- Having no participation in partisan politics or political campaigns. For senior staff, this even includes when they are acting in their individual capacity and on their personal time.
- Assuring that all research meets the highest standards of methodological integrity. It must be impartial.
- Assuring that only the most well suited tools and methods of analysis will be used. These tools and methods are evaluated for each and every study performed.
- Assuring that their findings and methods are presented in sufficient detail to permit outsiders to evaluate the credibility of their results. In fact, they encourage outside evaluation of their research.
- Making all datasets available to outsiders for analysis. These datasets are typically available either immediately after or six months after all reporting is completed, depending on the Pew center performing the research.
III.
In regards to their Mormons in America study called into question by Dr. Beal, Pew’s specific research methodologies for this study are described in “Section 5: Survey Methodology” of their report. In part, they state on pg. 70 of their report:
For both freshly sampled households and those in the recontact sample, the survey began with a screening interview. Respondents reached by landline were selected by randomly asking for the youngest adult male or female currently at home. Interviews on cell phones were conducted with the person who answered the phone, if that person was an adult 18 years of age or older. The screening interview consisted of a few short warm-up questions (about the respondent’s level of satisfaction with their community and their life), followed by a question about the respondent’s religious affiliation: “What is your present religion, if any? Are you Protestant, Roman Catholic, Mormon, Orthodox such as Greek or Russian Orthodox, Jewish, Muslim, Buddhist, Hindu, atheist, agnostic, something else, or nothing in particular?&rldquo; Those who described themselves as Mormons in response to this question were then administered the main survey, while the interview was discontinued for non-Mormons.
After identifying themselves as Mormons, qualified respondents were asked a separate question, “And is that The Church of Jesus Christ of Latter-day Saints, the Community of Christ, or some other Mormon church?” All self-identified Mormons were eligible for the survey, regardless of whether they identified themselves as part of the LDS Church. In practice, nearly all self-identified Mormons (99% in the current sample) describe themselves as part of the LDS Church.
To clarify the terms “freshly sampled” and “recontact sample”, understand that when Pew finds an individual who is willing to take a survey they keep that person’s contact information in a pool to draw from for future surveys. Having been contacted before for other surveys this group is known as the “recontact sample”, whereas those who have not previously participated in a Pew research survey make up the “freshly sampled” group. Respondents from both groups were included in the Mormons in America survey.
In these two paragraphs from Pew’s report there are three key elements to their research methodologies which ought to be explicitly highlighted. First, they began with a “screening interview”. If in the screening interview it was discovered that the respondent qualified for the “main” survey—in this case, qualification being self-identifying as Mormon—they were then given the actual survey used for this study. The interview was discontinued for all others. This step is critical; it acts as a filter which only permits involvement of those whose views are pertinent to the study at hand. Note that during this screening interview no indication of the study being performed was given. They even used an open-ended question as the main interview qualifier; they did not, for example, ask “Do you self-identify as being Mormon?” Hence, although a respondent may have been able ascertain that the study in question had to do with religion, they would not have been able to know how they needed to respond in order to be involved. Such an approach prevents those whose strong opinions may lead them to knowingly participate in a study which doesn’t pertain to them—thereby tainting the results. Consider the caution and foresight demonstrated by Pew in this approach.
The second key element I want to highlight is that the qualification for participation was specifically whether or not the individual described themselves as being Mormon. Why is this important? The answer is twofold: i. it doesn’t put any weight on the respondent’s faithfulness to their religion or its tenets, e.g., a respondent’s church attendance was not considered relevant; and ii. it prevents the views of those who do not self-identify as being Mormon from being presented as the views of those who do self-identify as being Mormon—whether or not their names are on the records of any given church.
Note that there may be some individuals who do self-identify as being Mormon, but who may not be members of The Church of Jesus Christ of Latter-Day Saints. This brings me to the third key element: qualifying respondents were asked to which denomination they belonged, but they still had no way of knowing whether this study pertained to any single church or group within the set of those who self-identify as Mormon. Those who did self-identify as Mormon but who were not members of The Church of Jesus Christ of Latter-Day Saints were still included in the survey for the report—recall that the title of the study is Mormons in America not Members of The Church of Jesus Christ of Latter-Day Saints in America—but Pew reports that these respondents only comprised 1% of those given the main survey.
IV.
Having discussed, in sections II and III respectively, Pew’s code of ethics and research methodologies generally and in regards to their Mormons in America study, I would like to now address Dr. Beal’s claim that Pew’s research is unreliable. As quoted in section II, Dr. Beal attempts to substantiate his claim by arguing:
- Pew reports that 90% of self-identifying Mormon women in America do not feel that dedicated women members of the LDS Church should be ordained to the priesthood .
- Two-thirds of women whose names are on the records of the Church do not regularly attend Sunday meetings of the Church.
- Women whose names are on the records of the Church but who do not regularly attend the Church’s Sunday meetings: are inactive, do not identify as being LDS, or are inactive and do not identify as being LDS.
- Gender issues are one of the primary reasons women disaffiliated from the Church.
- These statements constitute a contradiction.
- Hence, Pew’s research is unreliable.
I acknowledge that I have taken some poetic license in this expression of Dr. Beal’s argument in order to clarify it, but did so without changing or misrepresenting that argument. Please see the exact excerpt of his quote in section II, or his full quote in the comments section of the aforementioned J. Max Wilson post.
Now, if Pew’s screening interview had only allowed for those who both self-identified as being Mormon and who attended The Church of Jesus Christ of Latter-Day Saints regularly to be eligible to take the main survey; and if one assumed the following statements as premises: “two-thirds of women whose names are on the records of the Church do not regularly attend church”, “women whose names are on the records of the Church and who do not regularly attend church have disaffiliated from the Church”, and “women who have disaffiliated from the Church have done so because they feel that dedicated women members of the Church should be ordained to the priesthood”; then one could logically draw the conclusion “two-thirds of women whose names are on the records of the Church feel that dedicated women members of the Church should be ordained to the priesthood“. If this statement were true, it would contradict the results found in Pew’s study if Pew’s study had also been worldwide—one would have to define a premise giving percentages of women whose names are on the records of the Church and who live in America in order to correlate this conclusion to Pew’s actual survey results.
But that’s not Dr. Beal’s argument, nor is that Pew’s study.
Regarding Dr. Beal’s actual argument, a simple question: with the qualifier for the Pew study only being self-identification as Mormon—having nothing whatsoever to do with Church attendance—then how do statements concerning Church attendance statistics have any logical bearing on the reliability of the findings of Pew’s research?
Answer: they don’t. At all.
And that, in and of itself, completely refutes Dr. Beal’s argument against the reliability of Pew’s research.
But while I’m at it, I would like to address Dr. Beal’s premises. Setting the validity of his original premises aside for a moment, these premises are still too ambiguous to allow for the development of any pertinent logically sound conclusion. To shore up his premises, Dr. Beal would have to give percentages for his classifications of the three groups of women whose names are on the records of the Church and who do not regularly attend, as well as the percentage of women who disaffiliate from the Church because of gender issues. At this point one could logically make conclusions about the percentage of women whose names are on the records of the Church and who have disaffiliated from the Church due to gender issues. However, this logical framework could still not be used to quantify the validity of Pew’s research. Furthermore, even if such a framework is logically sound, it is not guaranteed to be accurate. To be accurate, the validity of Dr. Beal’s premises would need to be established.
So, even though this has no bearing on the reliability of Pew’s research, are Dr. Beal’s stated premises even accurate? Well, since he doesn’t substantiate the claims he made in his premises how can we assume they are valid? For the statement that two-thirds of women whose names are on the records of the Church do not regularly attend Sunday meetings of the Church, one could look at the Church’s statistical report, take the number of congregations, make some guesses as to the average number in a congregation attending weekly, multiply the two, and then see what percentage the result is of the total listed membership of the Church. But, that involves some guessing. And although one could try to make an educated guess, it would still be a guess and it would be very hard to measure the error associated with it. The Church Handbook of Instructions, Book 2, Section 21.1.35 “Research Studies in the Church” states, “The only authorized research agency of the Church is the Research Information Division of the Correlation Department. ” So, if one is going to make claims about worldwide Church attendance, they are going to have to get that information from Research Information Division. To the best of my knowledge they don’t give that information out. Yet, whether or not one agrees with the Research Information Division policies, their non-disclosure of those numbers doesn’t grant an individual or group the right to guess what those numbers are and then claim that guess as being accurate.
What about Dr. Beal’s other premises? I would not doubt that the majority of the women: i. with whom Dr. Beal associates, ii. and whose names are on the records of the Church, iii. and who have disaffiliated from the Church, iv. have done so primarily because of gender issues. However, if this is true I cannot see how Dr. Beal could make the extrapolation to the general set of women whose names are on the records of the Church and who have disaffiliated from the Church—at least, not without substantiating such an extrapolation with uncompromised data. I further posit that, world-wide, gender issues are a relatively small contribution to the disaffiliation of women from the Church. I have not performed a study to verify this hypothesis, but the statement from Pew’s Mormons in America study referenced earlier leads me to place more confidence in my hypothesis than in unsubstantiated claims to the contrary.
V.
Moving on to TMGISG as a whole, I question the integrity of this group’s research generally.
Consider the lack of reliability in the web survey floating around. First off, in simply making survey respondents feel that there is a bias—even if that was unintentional—TMGISG has tainted their results. Why? Well, because it’s hard to prevent a respondent from reacting one way or another to a perceived bias, especially on such an emotionally driven topic. For example, many of my Mormon acquaintances feel put on the defensive by the phraseology used in the survey. This has led to individuals choosing to stop midway through the survey, or not even take it at all, because they feel that TMGISG’s perceived bias will lead them to misrepresent these respondents’ opinions on the matter. How can a survey that alienates—to the point of non-participation—a non-negligible portion of the population being studied ever accurately reflect the views of that population? Furthermore, with those Mormons who do feel that women should be ordained to the priesthood not likely taking offence to the phraseology used in the survey, how can the survey results not be biased towards their view?
One may try to argue that some respondents are reading too much into the phraseology of the survey because of the socio-political views of those behind the survey. However, I don’t think such a claim is founded. Furthermore, even if that claim was justified, the aforementioned consequences on the reliability of the results of a survey seeking to sample the entire body of the Church would remain. This also speaks to the importance of a social science research group not giving any indication of subscribing to any socio-political agenda.
Also, how can a survey whose respondents do not feel they have a response option to a posed question which reflects their viewpoint ever accurately portray the viewpoints of the respondents?
How can a survey which employs no screening method prevent those with strong opinions from intentionally misleadingly presenting their views as those of a group to which they don’t belong?
How can a survey which allows anyone to take it as many times as they want not produce biased results? Don’t tell me responses from the same individual can be identified and filtered out. They can’t.
If TMGISG feels that Pew’s restriction of their study to just Mormons in America is insufficient, what is TMGISG doing to reach the world-wide membership? Has TMGISG translated their survey into the languages of all those who do not speak English?
How does this web survey reflect the viewpoints of those without internet access?
How does this web survey, which is being promoted through social media outlets, account for the discrepancies between age and social media usage? I.e., how do they prevent the results from being biased towards the viewpoints of those groups who tend to be more predominantly active on social media?
If, judging from Mr. Beal’s comment, TMGISG is seeking the viewpoints of Mormons on gender issues, why would they seek to include the viewpoints of those who do not self-identify as being Mormon and then present their views as being those of Mormons? If TMGISG wants to study the viewpoints on gender issues of those who do not identify as being Mormon but whose names are on the records of the Church, by all means they should go ahead. But if, in order to get the conclusion they want from their current survey, they have to present the views of those who do not self-identify as being Mormon as the views of Mormons—irrespective of whether or not their names are on the records of the Church—then they have gone very far afield of what we call science. (As a side note: if one were to correlate individuals whose names are on the records of the Church but who no longer self-identify as being Mormon with opposing views to the Church on gender issues, one would still be a long, long way from showing that those differing views were what caused these individuals to no longer self-identify as being Mormon.)
Taking these things into consideration, what will be the magnitude of error in the results produced from TMGISG’s web survey? Is it even possible for this error to be quantified?
Yes, I know that TMGISG’s web survey has been approved by the Institutional Review Boards (IRBs) at The University of Tampa and Georgia Southern University. But please understand, dear internet, that the only thing IRBs concern themselves with is the ethical treatment of human subjects in a research project (even though I’m a physicist, I’ve gone through training on how to meet IRB requirements because—as a scientist—I was curious about the research ethics surrounding the use of human subjects). That this survey has been approved by these IRBs gives us confidence that the privacy of participants will be maintained, but not that the research methodologies are scientifically sound.
VI.
Given all of the issues with their web survey, I’m going to give TMGISG the benefit of the doubt and assume that they recognize the utter lack of its scientific value. So why then, did they create it? The only hypothesis I can come up with is that they intended this survey to become a social media campaign as a means of drawing attention to their cause. That would explain why they are pushing the survey through social media outlets. It is also in harmony with the general tactics of those who seek for women’s ordination: to encourage any form of dialogue in order to draw attention to their cause, inflate its perceived importance to the average Mormon, and pressure the Church to change divinely decreed doctrines.
Now, TMGISG states that aside from the web survey, a second survey will be conducted using a “random, nationally representative sample of members of The Church of Jesus Christ of Latter-day Saints. ” Unfortunately, no more details concerning this survey are provided. Given the lack of merit of the web survey, I imagine that TMGISG will be spending its Kickstarter funds on a database of phone numbers of individuals to survey. If so, this sounds a lot like Pew’s research—only without the open, impartial objectivity they demonstrated. But by exhausting their research integrity capital in their purchase of a social media campaign, TMGISG have left themselves with nothing more than their Kickstarter revenue for their second survey.
And that’s too bad, because if TMGISG’s intent with their web-survey social media campaign is to help instigate and contribute to a conversation on gender issues in the Church, they are going about it the wrong way. For an excellent example on how to constructively contribute to this dialogue, see Sharon Eubank’s 2014 FairMormon address “This is a Women’s Church”.
Furthermore, regarding whether TMGISG will make the data from both of their surveys in their study available for external evaluation, they state:
Maybe… Why do you ask? More seriously, if you want access to the data you’ll need to contact one of the members of the Mormon Gender Survey Group and explain how you plan to use it. The group will then discuss it and make a decision.
This raises all sorts of red flags. One of the most important steps of the scientific method is repeatability. If one’s research cannot withstand the scrutiny of other scientists, then it is not scientifically sound. But even more alarming than whether or not TMGISG shares their data, is their statement “Maybe… Why do you ask? ” and the case-by-case need for a group discussion determining whether they will share their data. I cannot but infer from this that their willingness to share their data is dependent upon the motives of those requesting it. An unwillingness to share scientific data with those who disagree with one’s hypothesis—and who may discredit one’s analysis—speaks not of a desire to seek the truth but rather a desire to protect an agenda.
To TMGISG: can you not see why you are being called out on your research ethics, methodologies, and policies? Your motivation shouldn’t be “primarily academic” as Dr. Beal states; it should just be academic. And that’s the main issue here—that there appears to be a non-academic agenda motivating your research. Given, we are all human. We all have opinions. And sometimes our opinions can, naturally, influence our scientific research. That’s something all scientists need to both recognize and keep in mind. But I think that the concern you see being expressed here and elsewhere is that your group is letting its opinion get in the way of the science; your group is coming across as setting out to prove your hypothesis to be true, instead of setting out to test whether your hypothesis is true.
It is for these reasons that the codes of ethics, rigorous research methodologies, and dataset policies espoused by respectable research groups, such as Pew, are so important. And it is because of the well-established research integrity demonstrated by Pew, that you only do yourselves a disservice by calling it into question.
Therefore, I encourage and caution you to mark your path forward according to the following five cairns placed by Pew:
- Completely remove yourselves from any actual or perceived conflicts of interest. This includes your individual participation in and affiliation with any group which may call into question your impartiality.
- Employ only those tools and methods of analysis which are best suited for testing the validity of your hypothesis. Read: ditch the web survey; it is of no scientific worth. And be very, very, very careful about how you set up and conduct your second survey.
- Describe your findings and methods accurately and in sufficient detail to permit outsiders to evaluate the credibility of your results. Repeatability, repeatability, repeatability .
- Make your raw, unaltered datasets publicly available to any researcher desiring to evaluate the credibility of your results. You may have spent much time and money collecting your data, but unless you are willing to share it we have no way of knowing whether you are being honest and accurate in the findings you report.
- Explicitly establish a code of ethics and make it publicly available. This code of ethics should not only pertain to the treatment of those participating in your surveys, but should outline how you plan on handling conflicts of interest of researchers involved, maintaining your impartiality, and assuring only sound research tools and methodologies are employed.
Otherwise, TMGISG, you might consider yourselves to be on the right side of history, but you will be on the wrong side of science.