Utilizing Mixed Methods in Middle Grades Research

During most of the 20^th century, social and behavioral research was dominated by quantitative methods with positivism (and variants thereof, such as post-positivism) as its dominant worldview. The last two decades of the century witnessed the emergence of qualitative methodology with a worldview associated with variants of naturalism and constructivism. Each of these disparate approaches was grounded upon different epistemologies, ontologies, axiologies, and methodologies. While the proponents of each approach were to some degree assured of the “paradigm purity” of their position (Smith, 1994), it was this belief in purity that led to the paradigm wars that ensued. These paradigm wars were not reserved solely to quantitative and qualitative issues as the legitimacy of mixed approaches to research was also brought into question during these debates.

Despite the obvious merits of both quantitative and qualitative approaches, each of these has been criticized by proponents from the other orientation. These criticisms center particularly on issues related to rigor of procedures and overall quality and goodness (i.e., reliability and validity, or trustworthiness) of outcomes. One result of these criticisms has been the birth of a “third methodological movement” known as the field of mixed methodology. It is through the application of this third movement that pragmatism took hold philosophically inviting researchers to look at very pragmatic ways of using the strengths of both approaches in their research (Table 1).

Writing on the philosophical underpinnings of social and behavioral research seldom includes discussions of pragmatism (e.g., Charles Sanders Peirce, William James, George Herbert Mead, Arthur Bentley). This lack of awareness of the role of pragmatism in mixed-methods research permitted researchers to think that mixed-methods designs are simply combinations of quantitative and qualitative techniques with little attention paid to the need to coordinate the mixing of approaches, the analysis of data, and the drawing of inferences (e.g., that you can add some open-ended questions to the end of a quantitative data collection instrument and call it a mixed methodology). Tashakkori and Teddlie (2003) do not support this position and are fearful that many researchers may dismiss pragmatism as a “naive” or even “vulgar” orientation that simplifies highly complex philosophical issues into “what works” (p. x).

Since the 1930s, social science and educational researchers have combined various data collection strategies in their research. In 1979, Jick combined surveys, semi-structured interviews, observations, and archival materials to provide a “rich and comprehensive picture” (p. 609) of anxiety and job insecurity during organizational mergers. Jick’s strategy of triangulating data allowed for the integration of different types of data and the convergence of findings. Mixed-methods designs are conducted when you have or can collect qualitative and quantitative data which together will provide a better understanding of your research problem. When quantitative and qualitative data are combined, “we have a powerful mix” (Miles & Huberman, 1994, p. 42). Or as Greene and Caracelli (1997) remind us, by assessing both the outcomes of a study (i.e., quantitative) as well as the process (i.e., qualitative), we can develop a “complex” picture of a social phenomenon (p. 7).

The purpose of this article is to raise the awareness of readers in relation to the use of mixed methodologies, especially as this approach applies to middle grades research. As noted in the introduction to this volume of Middle Grades Research Journal, we desperately need to answer some of the lingering questions that have allowed indictments to be leveled against the middle school movement, policymakers to ignore our calls for assistance in the education of young adolescents, and practitioners to ignore the need to create and maintain “true” middle schools (i.e., according to the tenets of This We Believe, NMSA, 2003) in school districts across America. In addition to defining mixed methods and discussing some of the important issues related to it, two published mixed-methods studies will be reviewed.

Mixed-methods designs evolved from the notion of “triangulating” the information from different data sources, researchers, theories, and/or methods, a technique that came out of psychology (Campbell & Fiske, 1959) and sociology (Denzin, 1971) and reached some level of mature application in nursing (Morse, 1991) and evaluation research (Patton, 1990). Jick (1979) discussed triangulation in terms of the weaknesses of one method being offset by the strengths of another method. Those promoting and using mixed methods firmly believe that quantitative and qualitative approaches are compatible. They also resent the restrictions or constraints placed on researchers’ choices.

Mixed methods is formally defined as “the class of research where the researcher mixes or combines quantitative and qualitative research techniques, methods, approaches, concepts or language into a single study” (Johnson & Onwuegbuzie, 2004, p. 17). Its logic of inquiry includes the use of induction (discovery of patterns), deduction (testing of theories and hypotheses, and abduction (uncovering and relying on the best set of explanations for understanding one’s results (e.g., de Waal, 2001). In short, a major advantage of mixed-methods research is that it enables the researcher to simultaneously answer confirmatory and exploratory questions, and therefore verify and generate theory in the same study.

Another definition of mixed methods is offered by Creswell (2003). He states that a mixed-methods study involves the collection of both quantitative and qualitative data in a single study, in which the data are collected concurrently or sequentially, and involves the integration of the data at one or more stages in the process of research. He reminds his readers, by quoting Brewer and Hunter (1989), that it is a “legitimate inquiry approach” (Brewer & Hunter, p. 28, cited in Creswell, p. 510).

As noted earlier, the paradigm wars questioned the legitimacy of mixed-methods research. This debate was much more than two opposing camps arguing about issues of validity and rigor in research. Paradigm purists posited the “incompatibility thesis”—that compatibility between quantitative and qualitative methods is impossible due to the incompatibility of the paradigms underlying the methods. This thesis played itself out for several years during the late 1980s and early 1990s, especially at national conferences. Supporting this position, Guba (1987) stated that one paradigm precludes the other “just as surely as the belief in a round world precludes belief in a flat one” (p. 31).

Scholars criticized the incompatibility thesis on the grounds that mixed methods were already being employed in many fields (e.g., Brewer & Hunter, 1989; Patton, 1990). In 1989 Greene, Caracelli, and Graham conducted a search and found 57 studies that used mixed methods. Other scholars (Reichardt & Cook, 1979) argued that the thesis created a false dichotomy that did not hold up under careful scrutiny. While some methods may be more closely associated with one worldview than the other, they do not belong to one method more than the other. In opposition to the “incompatibility thesis,” the “unity thesis” was posited which asserts that the idea of worldviews (i.e., distinct epistemologies, ontologies, and axiologies) is mistaken (Creswell, Goodchild, & Turner, 1996; Walker & Evers, 1988).

Tashakkori and Teddlie (2003) identified six major unresolved issues and controversies in the use of mixed methods in social and behavioral research. These include: (1) the nomenclature and basic definitions used in mixed-methods research, (2) the utility of mixed-methods research (why do we do it?), (3) the paradigmatic foundations for mixed-methods research, (4) issues in drawing inferences in mixed-methods research, (5) design issues in mixed-methods research, and (6) the logistics of conducting mixed-methods research (p. 4). Various authors (e.g., Jick, 1979; Morse, 1991) explored the purposes of mixed-methods research, identified types of designs, and specified a notational system. This interest in the procedural aspects of mixed-methods research helped to answer some of these controversies. In addition to numerous articles and book chapters, the appearance of the Handbook of Mixed Methods in Social & Behavioral Sciences by Tashakkori and Teddlie (2003) contributed greatly to settling some of these issues.

A listing of the strengths and weakness of mixed approaches can be found in Table 2. A more complete treatment of this topic can be found in the Educational Researcher article by Johnson and Onwuegbuzie (2004).

Typologies are generally considered helpful in assisting researchers who are trying to make sense of the diversity of methods available and make decisions related to a study’s design. Many authors have written in very different ways about the types of mixed methodologies. Authors discussed here include Morse (1991), Creswell (1994), and Tashakkori and Teddlie (1998). But others, like Greene and Caracelli (1997), Morgan (1998), and McMillan and Schumacher (2001), have addressed issues related to mixed-methods design types.

Morse (1991), writing in the applied field of nursing, authored an important article on approaches to qualitative-quantitative methodological triangulation. In that article she presented a notational system that remains the current standard. This notational system includes the following four items:

1. The use of abbreviations QUAN for quantitative and QAUL for qualitative.

2. Use of the plus sign (+) to indicate that data are collected simultaneously or concurrently (e.g., QUAN + qual or QUAL + quan)

3. Use of arrows to indicate that data collection occurs sequentially (e.g., QUAL? quan)

4. Use of uppercase to denote more priority given to that orientation (e.g., QUAN) or lowercase to denote the less- dominant approach (e.g., qual)

According to Morse’s typology there are two obvious criteria: (1) the sequence in which data are collected, and (2) the priority given to one research approach over the other. Using Morse’s notational system, the following designs are possible (Table 3).

Creswell (1994) noted that there are three major types of mixed-methods designs, including the triangulation design, the explanatory design, and the exploratory design. The purpose of the triangulation design is to simultaneously collect both qualitative and quantitative data, analyze the data, compare the results to see whether the results from both forms of data are similar or dissimilar, and use the results to answer the questions that were posed by the researcher. The direct comparison of the two data analyses provides triangulation. A basic premise of this type of design is that one data collection strategy provides strength to offset the weaknesses of the other form of data collection. Generally speaking, the quantitative data allows for generalizability while the qualitative data provides context and meaning.

The explanatory design allows the researcher to collect quantitative or qualitative data sequentially or in two phases. This is currently the most popular of the mixed-methods designs. The researcher first collects the quantitative data then collects the qualitative data. In this example, the qualitative data can be used to explain the quantitative results.

Creswell’s (1994) final type, the exploratory design, is employed when the researcher first wants to collect qualitative data to explore a phenomenon, and then collect quantitative data to explain the variables discovered through the qualitative analysis. Researchers use this approach when existing instruments, variables, and measures may not be known or available for the population being studied.

Tashakkori and Teddlie (1998) developed a typology of mixed-method designs that includes sequential studies, parallel/simultaneous studies, equivalent status studies, and dominant/less-dominant studies. They note that their designs are based on “procedure” rather than on priority of orientation.

We usually expect that a typology covers all possible choices. Tashakkori and Teddlie (2003) caution us that this is not the case with mixed-methods research. Additionally, Maxwell and Loomis (2003) noted that “…the actual diversity in mixed-methods studies is far greater than any typology can adequately encompass…” (p. 244).

There are five essential issues that must be decided upon while designing a mixed-methods study. According to Creswell (2005), these six issues include: (1) providing an appropriate rationale for the design, (2) explaining what quantitative and qualitative forms of data will be collected and why, (3) deciding on the priority given to one method over the other or upon their equal status, (4) deciding about the sequence of the methods, and (5) matching the data analysis to design. The answers to these design issues will ensure that researchers do not simply label research as a mixed design because it contains both qualitative and quantitative components. We will revisit these five areas as we proceed to review two mixed-methods studies related to middle grades education.

All of this theoretical discussion of mixed methodologies warrants examples from the field of middle grades education. The ERIC database dating back to 1966 was searched for research that utilized mixed approaches and yielded only four results. These include:

1. Cifuentes, L., & Hsieh, Y. J. (2004). Visualization for middle school students’ engagement in science learning. Journal of Computers in Mathematics and Science Teaching, 23(2), 109-137.

2. Henderson, C. L., Buehler, A. E., Stein, W. L., Dalton, J. E., Robinson, T. R., & Anfara, V. A., Jr. (2005). Organizational health and student achievement in Tennessee middle schools. NASSP Bulletin, 89(644), 5475.

3. Minnema, J. E., Thurlow, M. L., & Warren, S. H. (2004). Understanding out-of-level testing in local schools: A second case study of policy implementation and effects (Out-of-Level testing Report 12). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

4. Robert, L., & Wilson, M. (1998). Evaluating the effects of an integrated assessment system on science teachers’ assessment perceptions and practice. (ERIC Document Reproduction Service No. ED422347)

The first two of these manuscripts were published in journals that are peer reviewed. It is also interesting to note that the first three have 2004 or 2005 publication dates. Because of these facts, the first two articles were chosen to be briefly reviewed according to the essential design issues that Creswell (2005) noted.

In the Cifuentes and Hsieh (2004) article, the authors used a mixed-methods design to explore the effects of student-generated visualization on middle-schoolers’ science concept learning. Quantitative methods were employed to compare students who visualized during study time with those who did not and found that visualization as a study strategy led to students’ improved test performance. Quantitative data were also collected to test if middle school students’ scores on a test of science concepts improved as a result of a computer-based visualization workshop. Qualitative data were collected to identify the elements in the school setting that interfered with instructional effectiveness of the computer-based workshops. As a result of the qualitative data analysis, Cifuentes and Hsieh found that these elements included motivational problems with students, distractions caused by computer hardware and software, and the limited amount of time (i.e., only three hours) allotted for the visualization workshop.

While there is no mention of the type of design (i.e., appropriate use of nomenclature) other than noting that mixed methods were employed, Cifuentes and Hsieh’s (2004) research questions were designed so that the first two were quantitative and the third was qualitative. Data sources included posttest scores, researchers’ journals, observation notes, students’ study notes, students’ computer graphics, and the “Study Strategies Questionnaire.” The authors present no rationale for using a mixed design. Although no indication of priority is evident, it is obvious from the research questions that the dominant method is quantitative. Finally, no mention is made about the sequencing (i.e., sequential or simultaneous) of data collection efforts.

The second study was conducted by Henderson et al. (2005) and involves an examination of the organizational health of 10 middle schools and the relationship of organizational health to student academic performance. Because of the well-constructed quantitative instrument, the Organizational Health Inventory—ML, quantitative data collection was dominant and occurred first. Qualitative data were collected from semi-structured interviews with administrators and teachers. The interview protocols were designed utilizing the factors (i.e., academic emphasis, teacher affiliation, collegial leadership, resource support, principal influence, and institutional integrity) contained in the quantitative instrument. The qualitative data were used to give meaning to standardized scores along with standard deviations which indicated the level of “health” of a given middle school. The researchers wanted to know what a school of average health (i.e., a score of 500) looked like. In other words, they wanted to give the standardized score of 500 some additional context and meaning. In another example, they wanted to know what the difference of two standard deviations between two middle schools looked like qualitatively or contextually. Henderson et al. employed appropriate nomenclature, provided the rationale for a mixed design, and explained the relationship between data types.

Creswell (2005) offers the following advice to researchers who are interested in undertaking a mixed-methods research project: (1) be specific about the type of design you are using and the relationship between the qualitative and quantitative data collection strategies; (2) specify the advantages that will accrue from collecting both qualitative and quantitative data; (3) recognize that it is easier to conduct a sequential explanatory design or exploratory design than a triangulation design; (4) recognize that you have taken on a challenging project that involves extensive data collection and data analysis; and (5) create a diagram that portrays the steps in the process to best present the procedures in your research design (excerpted from pp. 527-528).

Creswell (2005) additionally offers his readers advice to evaluate a mixed-method study? He recommends that we ask the following questions: (1) Does the author provide a rationale for mixing methods in a single study?, (2) Does the author identify the type of design utilized?, (3) Does the author mention the priority given to quantitative and qualitative data and their sequence?, (4) Has the author written research questions that are appropriate for a mixed design?, (5) Have the quantitative and qualitative data collection and analysis strategies been clearly explained?, and (6) Is the written structure of the study consistent with the type of design? These are excellent guideposts to help ensure that you are truly designing a mixed-methods study. They are also quite useful in assessing the quality and goodness of mixed-methods studies that you may encounter in professional journals and reports.

Currently there are three models for professional competency as it relates to mixed-methods research. The first model holds that the researcher should be able to fully use methods from both traditions. One of the major concerns with this model is that it might be difficult for a single researcher to have the necessary skills in both qualitative and quantitative methods. The second model proposes a team approach to conducting research. Research teams would be comprised of people who have the requisite skills allowing for one’s weaknesses to be compensated for by other members of the research team. Such a model is not uncommon in large-scale studies and medical research. A third model (Newman & Benz, 1998) calls for “minimum competency” in both qualitative and quantitative methods on the part of all researchers in the project, together with each researcher possessing a highly specialized set of skills in one of the two designs.

While the team approach, promoted in models two and three, can work in many instances, it unfortunately will not work for dissertation research and other similar research ventures. Also, inherent in the team approach is the need for researchers to be able to communicate with each other about all aspects of the research project and to understand the methods and analyses from both approaches being employed.

In the final section of Research and Resources in Support of This We Believe (Anfara, Andrews, Hough, Mertens, Mizelle, & White, 2003), it is noted that “research affirms middle level education as a distinct level of schooling worthy of its own ideas, philosophies, and strategies. Its’ practice provides opportunities for scholars and practitioners to apply this knowledge in daily practice” (p. 69). But there is also a warning offered that identifies the need for additional research—research that is different from most of what has been done and is currently being conducted. In addition to a call for more large-scale, longitudinal studies, replication studies, and experimental designs, the authors wrote, “…we need studies that combine quantitative and qualitative methodologies” (p. 70). Efforts are now underway by the Middle Level Education Research Special Interest Group (MLER SIG), an affiliate of the American Educational Research Association and the largest organization of middle grades researchers, to design a large-scale research project that will most likely follow a mixed methodology. Mindful of this context, I would like to conclude this article by sharing some personal experiences and insights regarding mixed methodologies and offering some advice to those members of the MLER SIG who will be designing this complex research project.

Recently I reviewed proposals for a conference that will occur in the fall of 2006. In two of the five proposals, authors identified that they had performed mixed-methods studies. Knowing that I was writing this article, I was anxious to review the proposals and see how the authors had written their methods sections, analyzed their data, and arrived at findings that demonstrated a convergence of qualitative and quantitative data analysis. To my disappointment, the authors of both of the proposals indicated that they were collecting data using Likert-type surveys and that two or three open-ended questions had been added to the end of the quantitative instrument. The similarity between the two methods sections was uncanny to the point that I began to wonder if the proposals had been submitted by the same person or persons. There was no mention of the type of design (beyond mentioning that it was a mixed-methods study), no rational for choosing a mixed methodology, no indication of priority or sequence of approaches, and no use of the nomenclature that is associated with mixed designs. Just as with quantitative and qualitative approaches to research, there are rules and established procedures that should (really should read “must”) be followed in the design, conduct, and reporting of mixed-methods studies. I caution readers not to label their work as “mixed” unless the essential issues (Creswell, 2005) mentioned earlier are adequately addressed.

I wrote a qualitative dissertation and started my career in higher education as a qualitative researcher. But the more qualitative work I did, the more I realized that I had to venture into the realm of quantitative research. Between 1999 and 2001, Kathleen Brown and I conducted a large qualitative study of middle school principals. That work eventually resulted in a book (Brown & Anfara, 2002) focused on what it means to be developmentally responsive as a middle school principal. But that qualitative work forced me to develop a quantitative instrument to more adequately measure developmentally responsive leadership (Anfara, Roney, Smarkola, DuCette, & Gross, 2006). After 10 years in higher education, I find that as I design studies I am driven to mixed designs and avoid the “paradigm purity” that was discussed earlier.

As noted in the quote by Morse (2003) at the start of this article, there is no doubt that combining both qualitative and quantitative approached to research can offer us a more complete picture of what we are studying. The triangulation of methods that occurs as a result of this approach strengthens the inferences we can draw from research and extends the generalizability of the findings. As we continue to expand the knowledge base related to middle grades education, researchers are challenged to employ rigorous mixed designs in answering the questions that still need to be answered.

In the introduction to this article, I mentioned that the 1980s saw the birth of a “third methodological movement” in the social and behavioral sciences. Evidence of this movement is apparent in many ways. Most importantly, we saw the publication of the Handbook of Mixed Methods in Social & Behavioral Research (Tashakkori & Teddlie, 2003). While the conditions are present for the continued growth of mixed methods as a legitimate research approach, this movement is not on completely firm ground. As more mixed-methods research is published, it will be interesting to see what reception it receives from the larger community of researchers, policymakers, and practitioners. We are all aware of the paradigm wars that occurred as qualitative methods edged its way into a world mostly dominated by researchers with a quantitative orientation.

This third methodological movement challenges us to move beyond the labels of quantitative and qualitative and to look at what methods are best suited to answer the important questions that still haunt us. In the field of middle grades education, there are multiple issues and questions that cry out to be answered. The research that is cited to support the middle school movement needs to be extended and expanded. While the findings from current research (e.g., Felner, Jackson, Kasak, Mulhall, Brand, & Flowers, 1997; Lee & Smith, 1993; Mertens, Flowers, & Mulhall, 1998) are quite promising, not enough quality research has been conducted to silence those who oppose middle grades education (e.g., Yecke, 2003) and to convince practitioners in school districts across America that investing in “true” middle schools is the right thing to do for the education of young adolescents. There are far too few schools across this nation that are characterized by the tenets of This We Believe (NMSA, 2003).As was noted in the introduction to this journal, we need more large-scale studies, more replication studies, and more mixed-methods studies as we continue to conduct research focused on middle grades education.

Greene and Caracelli (1997) pointed out that “the underlying rationale for mixed-method inquiry is to understand more fully, to generate deeper and broader insights, [and] to develop important knowledge claims that respect a wide range of interests and perspectives” (p. 7). Mixed-methods research has the potential to be very useful in answering the questions that middle grades practitioners, advocates, researchers, and policymakers need answered.

Readers who are interested in learning more about mixed methodologies are encouraged to complete additional reading from the following list of books, chapters, and journal articles. Many of these references were not used in the preparation of this article, but are important contributions to the field of mixed methods. The references provided are meant to be a starting point and are not to be considered exhaustive.

Reichardt, C. S., & Rallis S. F. (Eds.). (1994). Qualitative-quantitative debate: New perspectives (New Directions for Program Evaluation, No. 61). San Francisco, CA: Jossey-Bass.

Newman, I., & Benz, C. R. (1998). Qualitative-quantitative research methodology: Exploring the interactive continuum.

Carbondale, IL: University of Illinois Press.

Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage.

Smith, M. L. (1986). The whole is greater: Combining qualitative and quantitative approaches in evaluation studies. In D. D. Williams (Eds.), Naturalistic evaluation (New Directions for

Evaluation, No. 30, pp. 37-54). San Francisco: Jossey-Bass.

Creswell, J. W. (1999). Mixed method research: Introduction and application. In C. Ciznek (Ed.), Handbook of educational policy (pp. 455-472). San Diego: Academic Press.

Rossman, G. B., & Wilson, B. L. (1985). Numbers and words:

Combining quantitative and qualitative methods in a single large-scale evaluation study. Evaluation Review, 9, 627-643.

Laurie, H., & Sullivan, O. (1991). Combining qualitative and quantitative methods in the longitudinal study of household allocations. Sociological Review, 39(1), 113-139.

Morse, J. (1996). Is qualitative research complete? Qualitative Health Research, 6(1), 3-5.