Learning Analytics and the Shape of Things to Come Free

In 2012, Laurie Dringus laid out five “MUST” statements to “present a prospective stance on how learning analytics, as a core evaluative approach, must help instructors uncover the important trends and evidence of quality learner data in the online course” (p. 88, emphasis in original):

1. Effective learning analytics in online courses MUST develop from the stance of getting the right data and getting the data right.

2. Effective learning analytics in online courses MUST have transparency.

3. Effective learning analytics in online courses MUST yield from good algorithms.

4. Effective learning analytics in online courses MUST lead to responsible assessment and effective use of the data trail.

5. Effective learning analytics in online courses MUST inform process and practice (p. 89, emphasis in original).

As obviously important as such issues are, this essay will argue that, even if learning analytics satisfy all of Dringus’ MUST statements, they will continue to dramatically change education as it is known today. As previous stated, learning analytics will change education epistemologically, ontologically, systemically, and politically, and there is likely nothing that can be done to prevent these changes.

Learning analytics shape education epistemologically through the ways in which it privileges knowledge and learning that can be quantified and tested, and marginalizes knowledge and learning that cannot. Analytics shape education ontologically through the ways in which educational goals are changed to favor quantifiable outcomes and measurable objectives. Analytics shape education systemically in the way their use percolates through all levels of the educational establishment, and how that which is measured becomes the “currency of the realm.” Analytics shape education politically through the changing power relationships that develop around measurement and data collection, and through the mandating of not only data collection, but of specific and measurable results.

“Learning analytics” were first introduced—and continue to be most important in postsecondary online learning—simply because online learning generates significant amounts of data about learners and learning. Indeed, the New Media Consortium’s 2016 Horizon Report (Johnson et al., 2016) links learning analytics to online and adaptive learning:

Learning analytics, however, are quickly becoming mainstream, and so this essay explores how learning analytics might eventually change higher education in general simply by being used. Such effects certainly apply, and most likely will first be felt, in online education; perhaps their shapes are already becoming evident therein. In the sections that follow, those shapes and the forms such effects might take/are taking, are explored.

Epistemology is the study of knowledge and knowing. Epistemology is concerned with the nature of knowledge, with what we can know, and how we know it. Epistemology is thus also intimately connected with learning and the process by which people learn. Learning analytics is also, of course, concerned with learning, but primarily from the perspective of measuring the information students know and have previously learned. Surfacing the differences between knowing and measuring knowing highlights those differences as well as the question of what types of learning can be objectively measured. Attributes which many educators value in teaching—such as, curiosity, resilience, the consideration of multiple perspectives, the capacity to learn, awe–are ones that are not easily or obviously quantifiable, and thus not valued in an analytics culture. Moreover, one suspects that the only methods we could devise to measure such things would actually profoundly diminish our understanding of these phenomena. Learning analytics thus affects education epistemologically through the ways in which it privileges knowledge that can be quantified and tested over knowledge that cannot be.

The ways in which No Child Left Behind legislation has impacted K–12 education in the United States (Klein, 2015) intimates the types of effects educators might expect learning analytics to have at the postsecondary level. We do not deny that No Child Left Behind was bipartisan legislation implemented with all good intentions. The idea behind the legislation was that all children can learn, and that all teachers and schools should be held accountable for their learning. To these ends, the legislation required all states to develop “challenging” standards in reading and mathematics for grades three through eight, as well as annual assessments to measure whether students were meeting those standards. In other words, No Child Left Behind required all states to develop learning analytic systems. All states developed standardized tests in reading and mathematics, which accordingly narrowed elementary and middle school curricula to the quantifiable concepts appearing on these assessments. Additional complications associated with No Child Left Behind and its successor, Race to the Top (U.S. Department of Education, 2009), mostly regarding their implementation, have developed. However, most importantly, if most or all of those implementation problems could be alleviated, it would not change the fact that high-stakes standardized testing reduces and limits teaching and learning. Stated another way, highstakes standardized testing marginalizes nonquantifiable understandings.

At the postsecondary level in the United States, testing is not yet quite so high stakes (for institutions, that is, not students: failure at this level can certainly end a student’s academic career), but the trend favors movement in this direction. Federal Pell grants to students already are performance based in that students must meet certain grade point average and credit accumulation standards to continue receiving support (Schudde & Scott-Clayton, 2014). Many U.S. states are currently linking their funding of public postsecondary institutions, such as it is, to student performance, and many more are considering doing so in the future (National Council of State Legislatures, 2015). At the federal level, new regulations link access to federal loans for career training programs to potential earnings for students who complete them (U.S. Department of Education, 2014).

What makes requirements like these possible is analytics, specifically, the Integrated Postsecondary Education Data System (n.d.), which beginning in the 2015–2016 academic year, is collecting information on undergraduate retention, progression and graduation from every college, university, and vocational/technical institution that participates in the federal student financial aid programs.

Learning analytics thus linked to highstakes outcomes are leading to more, and more specific analytics to track individual student progress within courses and programs. When embedded in competency-based systems, analytics accumulate data on the specific knowledge and skills that students have acquired. Needless to say, these trends are narrowing college course content to information that can be objectively measured and learning that can be quantified, particularly at the community college level, and marginalizing complex understandings and skills that educators wish to develop in students, but cannot be directly tied to employment criteria.

Ontology is the philosophical study of being, existence, and reality. The term “ontology” can also be employed to refer to a particular theory about the nature of being, or the kinds of things that have existence. In computer science, an “ontology” is the formal naming and definition of the types, properties, and interrelationships among the entities that exist in a particular virtual domain. For example, the word card has many different meanings. An ontology about the domain of poker would model the “playing card” meaning of the word, while an ontology about the domain of computer hardware would model the “punched card” and “video card” meanings. Similarly, in the domain of education, what we perceive as reality is based on the process by which we have named and defined teaching, learning strategies and practices, as well as the interrelationships among them.

Learning analytics change how we perceive education. They shape education ontologically through the ways in which educational goals are changed to favor quantifiable outcomes and measurable objectives. They shape education ontologically through the new quantifiable meanings assigned to existing educational practices and the ways in which the resulting changed meanings affect our common understandings of those practices. Bowker and Star (1999) remind us that the data concepts and relationships data scientists choose to model matter; that what gets coded, matters; but what does not get coded—or “coded out”—matters more. This same mechanism of choice affects the educational world as much as it does the scientific world.

Consider, for example, the construct “student success.” In the analytics of postsecondary retention and progression, student success is usually defined as passing courses (achieving a “C” or better); achieving a high credit ratio (passed courses/courses taken), passing a certain number of courses a year, and most importantly, earning a credential within one and a half times the period traditionally allocated (3 years for 2-year programs and 6 years for 4-year programs). It is important to consider, all of these measures lack any direct relationship with learning, or knowledge: even the limited, measurable sort of leaning to which we are accustomed. These measures do, however, narrow the meaning of academic success to exclude the achievement of goals students have set for themselves (e.g., learning a particular skill, obtaining a degree while working and raising a family, or returning to school for intellectual and social stimulation). Such focus on course completion can be seen, therefore, to promote the passing of tests over deep, intellectual, and meaningful learning on an individual level.

In addition, a key issue in the ontology of learning analytics is that of machine thought and decision making versus human thought and human decision making. In Computer Power and Human Reason, Joseph Weizenbaum (1976) argued, “however intelligent machines may be made to be, there are some acts of thought that ought to be attempted only by humans” (p. 76). Weizenbaum’s point being, machines are only capable of instrumental reason, on finding the most efficient means to achieve a specific end; they are not capable of understanding or reflecting on the value of that end. Therefore, he maintained that certain societal functions that require human understanding and evaluation, such as psychoanalysis or judicial rulings or, one could easily argue, education, should never be assumed by algorithms.

It is quite clear, however, that learning analytics are taking us in the direction of instrumental reason. For example, John Campbell (Campus Technology, n. d.) maintains that an advantage of analytics is they allow administrators to “view the institution through the lens of empirical efficacy.” What such a statement presumes is that “empirical efficacy” represents the greatest good. This, however, is not necessarily true. The way data is transmuted and subsequently decontextualized for entry into a database is one of the ways in which this data acquires its apparent objectivity and associated authority. Nevertheless, context is often a critical component in learning. As Jeremy Knox (2015) asks, “Why has an algorithm been given the responsibility of saying ‘you’re doing OK’ or ‘you’re not doing OK’ over that of the teacher, or indeed the student?” (p. 55).

Analytics shape education systemically in the way their use percolates through all levels of the educational establishment, and what is measured becomes the “currency of the realm.” A “system” is an organized purposeful structure consisting of interrelated and interdependent elements bound together by common processes and functions and possessing clear boundaries. Educational systems exist in a hierarchy ranging from the individual classroom, to the school, to the school system, to particular types of schools (e.g., K–12 schools, community colleges, medical colleges) to political groupings (state, regional, national, international). In essence, due to issues of scale, analytics generally require implementation at higher levels, often by decree, and so have a hegemonic influence on teaching and learning within and across institutions.

Most educators acknowledge significant changes in institutional systems must be implemented before learning analytics can successfully be applied. The U.S. Department of Education (2012), for example, argues that “successful application of educational data mining and learning analytics will not come without effort, cost, and a change in educational culture” (p. 37). The Office of Educational Technology’s report on learning analytics finds that not only will large numbers of new employees be necessary for data preparation, processing, and analysis, but administrators, faculty, and staff at all levels will require retraining, and to change the way they work to incorporate big data into the day-to-day functioning of educational institutions.

Indeed, the EDUCAUSE Center for Analysis and Research’s Analytics Maturity Index for Higher Education (Reinitz, 2015) provides a sort of blueprint for just how extensive the systemic changes generated by analytics might be. The Analytics Maturity Index was designed to measure an institution’s capacity for analytics, and is part of a new EDUCAUSE Benchmarking Service. It locates institutional capacity at six levels—starting, visioning, launching, implementing, and transforming— on six dimensions—decision-making culture, policies, data efficacy, investment/resources, technical infrastructure, and institutional research involvement. Clearly, this is a blueprint for transformational change that will eventually impact every aspect of higher education institutions.

Building the capacity of educational organizations to use data mining and analytics meaningfully is a major undertaking, and this is also true of larger systems such as state departments of education or the national educational system (U.S. Department of Education, 2012). Moreover, it is becoming increasingly important for these systems to develop the ability to interact and share data between themselves. Thus, that which must be measured to meet the requirements of individual institutions, states, and the federal government takes on increasing importance, and institutions at all levels must change to accommodate the collection, processing and analysis of that data. Ben Williamson (2015) writes,

Politics is concerned with the exercise and distribution of power. Whatever else any organization may be, it is also a political structure. This is as true of educational institutions, as well as private sector, for-profit corporations. All organizations operate by distributing authority and creating a framework within which power is implemented (Zaleznik, 1970).

Analytics shape education politically through the changing power relationships that develop around measurement and data collection, and through the mandating of both data collection and specific, measurable results. As education systems at all levels are changing to accommodate learning analytics, education governance is now being enacted through new types of digital policy instruments that allow educational policy to be made operational, and its effects measured. Not only are the individuals who have the ability to understand and manipulate learning analytics gaining power within such educational institutions (consider the ways in which the humble “institutional research staff” have morphed into “chief data officers”), but so are educators at all levels whose metrics meet the desires of political demands.

Perhaps the greatest power of all resides in the algorithms employed in data analytics themselves. Lyndsay Grant (2015) writes, “In educational policymaking and regulation, it seems that ‘big data’ is able to lay claims to greater legitimacy and authority than other ways of knowing” (p. 35). Seemingly objective data are now being integrated into much of educational policymaking, with data from schools and classrooms linked to vast global data collection networks, and education itself rearticulated within numerically based paradigms employed by new software development, data companies, and data analysis instruments (Williamson, 2015).

For example, Pearson Education’s The Learning Curve (http://thelearningcurve.pearson.com) combines 60 global datasets in order to “enable researchers and policymakers to correlate education outcomes with wider social and economic outcomes.” It seems apparent that in a system such as this, existing on a scale such as it is, the algorithms that drive data collection and analysis possess enormous power. Moreover, it is becoming apparent the Pearson network is just the beginning, merely a hint of the scale and influence future learning analytics systems might eventually possess within the educational setting.

In science, the notion that the act of observation influences the phenomenon being observed is commonly accepted and known as “observer effect.” For example, voltmeters usually need to be connected to the circuit whose voltage they are measuring, and so by their very presence affect the voltage they report. In the social sciences, the impact of observation, and in particular the impact of the tools chosen to record social phenomena, are if anything, of greater significance.

Marshall McLuhan wrote, “We shape our tools and thereafter our tools shape us” (1964, p. xxi). McLuhan was concerned primarily with cognitive tools, tools we use to support and extend our thinking, in general and communication tools in particular. Cognitive tools, he pointed out, privilege particular ways of knowing and marginalize others. McLuhan’s argument was that persistent use of a particular cognitive tool therefore changes, not only our habits of mind, but also the culture in which such use is embedded.

Clearly, the set of tools that comprise learning analytics are simultaneously observational and cognitive in nature. Moreover, they are designed to affect actions within the educational domain; thus, their effects are in some real sense immediate. Simon Buckingham Shum (2015, para. 4) asks, “In the very process of trying to value certain learning qualities by tracking them, will we in fact distort or even destroy a living, organic system, through clumsy efforts to categorise and quantify?”

The use of learning analytics will surely affect education in ways that extend beyond the information they are purportedly designed to uncover. This essay has examined how the use of learning analytics might shape education epistemologically, ontologically, politically, and systemically. There are most likely other unanticipated effects they will have on education as we know it. Not only are we unable to anticipate such effects, we probably cannot prevent either them, or the effects we can anticipate, from occurring. We can, however, in some sense contain and mitigate their effects. We can be aware of the transformations entailed in the widespread adoption of analytics approaches, and, most importantly, work to preserve the human systems that such machine systems were designed to serve in the first place.

We might begin with reengaging what we believe is the mission of education, especially higher education. Do we think the purpose of education is to provide our students with a set of skills that will make them more employable; or is it to produce independent thinkers and life-long learners, citizens of the republic (see Friere, 1970)? This is not really an either/or question; clearly, specific skills are necessary building blocks for higher learning; but rather, it is a question of focus. Specifically, it is critical that any learning analytics we allow to be employed serve our education missions, and not that we redesign our educational missions to serve learning analytics.