Defining Learning Analytics…

learning with insight

Our Definition of Learning Analytics

Authors: Prof Vivekanandan Kumar, David Boulanger

Analytics is a machine-learning driven, continuously self-validating, self-learning and self-directing meta-analysis that, aimed at optimizing human learning, will autonomously inform both experts and non-experts, autonomously offer formative assistance in a domain-specific decision process, autonomously justify technically and non-technically the validity of reached/brewing conclusions, autonomously verify appropriateness of recommended/brewing actions, and autonomously continue to observe recommended/brewing actions of tapestry-oriented self-similar systems. Learning Analytics nurtures and measures learning in terms of competencies, which in turn can be used to measure learning outcomes and as a non-traditional means to credential learning.

By ‘autonomous’ we mean that LA will continue to observe various learning contexts and measure competencies without human intervention. They may use different types of measures/techniques and dynamically determine a more appropriate one for a person under observation.

By ‘recommended actions’, we mean that LA is not confined to actions that are pre-defined. There may be emerging learning scenarios that bring forth new ‘actions’ to be observed. So, LA may autonomously recommend to include these new emerging actions into its calculations of competencies.

By ‘tapestry-oriented’, we mean that the scope of LA is thoroughly global. For instance, when LA measures coding competencies in Java, it may potentially include Java programming courses offered all around the world, at any level – university or high school or MOOC or private tutorials (e.g., Khan academy). Tapestries are designed to place students in a global pool of coding talent.

By ‘self-similar systems’, we mean that as students learn to program across multiple coding scales (e.g., a module, a program, a package, a large application) and across different types of programming environments (pair-programming, agile-programming, etc.), the types of issues they face will be similar to each scale and environment. This is just a guess at this time, but we can try to prove it. The hypothesis can be that core competencies of students will remain the same irrespective of scale or environment. Mastery of programming (e.g., novice, intermediate, advanced, professional programmer) is what will remain a constant across scales and environments.

Our thinking behind our definition of LA

Prof Vivekanandan Kumar:

We need to specify LA’s unique traits that distinguishes it from other areas of research such as intelligent tutoring.

At the beginning, we felt that volume of data handling is a key feature of LA. LA handles a ‘large’ volume of data and a large ‘variety’ of data, yielding a large number of ‘models’. The areas of big data and data science took ownership of that trait. So, it cannot be the defining trait.

AI, data mining, machine learning, causal modelling, and so on are specific techniques that can be applied in any area of research.

The descriptive, prescriptive, predictive, and diagnostic features are functionalities of LA systems. They themselves do not define LA. You can also find these functionalities in other systems such as enterprise management systems, healthcare systems, and so on.

If we observe self-similarity in LA, it also becomes a trait but is not a defining trait since many computer networks systems have this trait.

We then thought self-validation to be a trait. Apparently, there are systems that self-repair themselves. So, it cannot be a defining characteristic of LA.

Being autonomous (self-sufficient and self-directedness) is not a defining trait of LA since there is an area called “Autonomous Systems”.

Tapestry is about ‘automated measurement of leading-edge of knowledge’. The leading-edge is being continually expanded with continuous arrival of information about human and machine efforts. I think this is the defining feature of LA. This can be a global endeavour. Or, it can be restricted to a particular school or class or even an individual. It has to be the ‘leading-edge’ since educational assessments would take care of the other types of ‘automated, real-time assessments’ such as automated quiz systems, automated marking systems, formative assessments, and so on. The differences between ‘leading-edge’ and ‘formative assessment’ are: a) leading-edge can handle VVV – volume, variety, veracity – of datasets, b) unlike formative assessments, information for measuring leading-edge is obtained through both scheduled assessment activities as well as unscheduled learning activities (through observations), c) the value of ‘leading-edge’ outcomes is most useful immediately after its detection (as in the case of information gathered by spies) so that just-in-time feedback and remediation can be effected based on observed outcomes and cumulative/current patterns, d) since leading-edge datasets include data on emotion, capacity, executive functions and other metacognitive elements that are typically not measured under formative assessments, highly complex models can be built to derive new pieces of information, and e) observational study mechanisms are more relevant and readily applicable for leading-edge datasets than traditional formative assessment datasets.

In summary, Learning Analytics is the next stage of evolution of assessments – summative assessments, formative assessments, and now Learning Analytics leading-edge assessments

David Boulanger:

I like this idea of identifying defining and non-defining traits of LA although all of these non-defining traits are still needed to make possible its defining traits. To start with, I would abstract the LA definition away to just analytics, which would still directly apply to LA.

Analytics is about thinking out of the box, pushing the knowledge frontier further, and the ability to improve itself (going beyond its pre-programmed state). I would tend to describe current artificial intelligences as being dumb (limited and bounded) intelligences (like animals). On the other side, analytics is concerned with smart intelligences (inspired by human intelligence).

Analytics is also not limited to structured data and structured analyses. It is capable to handle and making sense of new types of information (unstructured).

That notion of ‘leading-edge’ is interesting because it is similar to the evolutive nature of the term ‘big data’. What is big data in 2018 would not necessarily be big data in 2030. Similarly, analytics is concerned with discovering (structuring) the unknown (unstructured), that is, finding ways to generate new types of knowledge. For example, we may not know certain things but we know how to get/derive that information. This is not analytics. Analytics is about new analysis processes to discover what we do not even suspect. That is exactly what I told you the other day about that meta-analysis that analyzed the effectiveness of a drug. Many separate studies have been conducted in the past (during several decades), encapsulating all the evidences necessary to ‘know’ the causal effect of that drug. That knowledge remained hidden until a human had the insight to conduct a meta-analysis to finally discover that hidden piece of knowledge. Analytics is that insight, that smart intelligence that triggered somebody to do that meta-analysis and push the knowledge frontier further. By connecting that new piece of knowledge to the existing pool of knowledge, new triggers will occur, which will contribute to pushing the knowledge frontier further. So, this is non-stop.

Bradshaw et al. state the following (http://www.jeffreymbradshaw.net/publications/IS-28-03-HCC_1.pdf):

“However, even when the self-directedness and self-sufficiency of autonomous capabilities are balanced appropriately for the demands of the situation, humans and machines working together frequently encounter potentially debilitating problems relating to insufficient observability or understandability (upper right quadrant of Figure 1). When highly autonomous machine capabilities aren’t well understood by people or other machines working with them, work effectiveness suffers.”

Could also this trust management be a defining trait of analytics? Clearly, self-sufficiency and self-directedness do not address that. Analytics will. Analytics involves a social interaction component.

Moreover, analytics strives to make sense of any observation it collects and will take the initiative to fill in any gap that stands in the way of generalization.

With these things in mind, could we make a difference between causal modelling and causal analytics? Causal modeling is static and pre-defined, while causal analytics is automated and inquisitive, seeking to find and investigate causal relationships that it is not told to do, with the objective to measure and reach generalizability. That is why I would say that causal analytics is central to the development of domains like learning analytics, healthcare analytics, business analytics, etc.

Prof Vivekanandan Kumar:

Defining and non-defining traits is the best way forward.

Your statement on analytics is getting closer to defining this area. Let me refine it further: “Analytics is an ethics-bound, autonomous and trustable human-AI fusion that measures knowledge frontiers at an increasing rate and pushes the boundaries of knowledge frontiers further”. I think the ability of the human-AI fusion to be able to measure the knowledge frontier and push it further is the essence, a defining trait for analytics.

I also like this statement: “Analytics is the insight that triggers a meta-analysis to measure and push knowledge frontier further” – this is a more restrictive trait in that we say that ‘meta-analysis’ is the only means to measure and push knowledge frontiers. We don’t know that yet. It may be possible to get insights without historical evidence.

Non-defining traits include, handling of a variety of data, volumes of data, autonomous nature, ethical principles, meta-analysis techniques, observational study techniques, causal modelling, connecting pools of knowledge, …

Trust certainly should play a part; otherwise, there won’t be any belief in the measurement of knowledge frontier. For now, I think it can be a non-defining trait.

Causal analytics, I think, is one way of deploying analytics. There may be other ways of doing analytics.

And yes, APA’s are the ultime human-AI fusion mechanisms.


LA Definition in the Literature

Nistor, N., & García, Á. H. (2018). What Types of Data Are Used in Learning Analytics? An Overview of Six Cases. Computers in Human Behavior.

“The most accepted definition of Learning Analytics (LA) positions this discipline as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Long & Siemens, p. 1). The definition focuses on one central object—data about learners, or educational data—and implying the objectives included by Clow’s (2013) learning analytic cycle: understanding learning and learners, collecting data, defining metrics, and deriving interventions aimed to optimizing the educational processes (see also Buckingham Shum, 2018). The emergence of LA as a discipline is tightly related to advances in computer-supported learning and the large amount of data—big data—collected by virtual learning environments, including, but not limited to, Learning Management Systems (LMS) and Massive Online Open Courses (MOOCs).”

Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE review, 46(5), 30.

“According to the 1st International Conference on Learning Analytics and Knowledge, “learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs.””

1st International Conference on Learning Analytics and Knowledge, Banff, Alberta, February 27–March 1, 2011, <https://tekri.athabascau.ca/analytics/>.

Maseleno, A., Sabani, N., Huda, M., Ahmad, R., Jasmi, K. A., & Basiron, B. (2018). Demystifying learning analytics in personalised learning. International Journal of Engineering & Technology, 7(3), 1124-1129.

“Defining Learning Analytics in Personalised Learning

The term learning analytics came into use in 2009 [21]. Siemens and Gasevic [31] defined learning analytics as a specialty area whereby it focuses on students’ data, in terms of collecting, analyzing and reporting in order to understand and improve the learning experiences to an optimum level. While the use of analytics and data analytics is relatively new in education, in the past, this was typically driven by the needs of the education sector to sup-port data-driven decision-making and planning [32]. Learning analytics are defined by the Society for Learning Analytics Research (SoLAR) as the measurement, collection, analysis and re-porting of data about learners and their contexts, for understanding purposes and optimizing learning and the environments in which it occurs [33].”

[21] US Department of Education (2010), Transforming American Edu-cation: learning powered by technology. Washington DC: Author. Retrieved from https://www.ed.gov/sites/default/files/NETP-2010-final-report.pdf

[31] Siemens, G., and D. Gasevic. (2012), “Guest Editorial-Learning and Knowledge Analytics,” Educational Technology & Society, vol. 15, no. 3, pp. 1-2.

[32] Baker, R.S.J.D., and K. Yacef. (2009). The State of Educational Data Mining in 2009: A Review and Future Visions. Journal of Ed-ucational Data Mining, Vo. 1 No. 1, pp. 3–17.

[33] SoLAR (Society for Learning Analytic Research. (2011), “Open Learning Analytics: an integrated & modularized platform. Proposal to design, implement and evaluate an open platform to integrate heterogeneous learning analytics techniques, SoLAR.

Viberg, O., Hatakka, M., Bälter, O., & Mavroudi, A. (2018). The current landscape of learning analytics in higher education. Computers in human behavior.

“Definitions of LA vary. Some define it explicitly in terms of the use of student-generated data for the prediction of educational outcomes, with the purpose of tailoring education (Junco & Clem, 2015; Xing, Guo, Petakovic, & Goggins, 2015). Others define LA as a means to help educators examine, understand, and support students’ study behaviours and change their learning environments (Drachsler & Kalz, 2012; Rubel & Jones, 2016). While there is no generally accepted definition of LA, many refer to it as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Long & Siemens, 2011, p. 34).”

Arafat, S., Aljohani, N., Abbasi, R., Hussain, A., & Lytras, M. (2018). Connections between E-learning, web science, cognitive computation and Social Sensing, and their relevance to learning analytics: A preliminary study. Computers in Human Behavior.

“Analytics corresponds to the evaluation of processes and states, of student discourses, tests, student interactions pertaining to learning, social metrics, information about student’s emotional state (Bahreini, Nadolski, & Westera, 2016) and demographics, etc. (Martinez-Maldonado et al., 2016).”

Howell, J. A., Roberts, L. D., & Mancini, V. O. (2018). Learning analytics messages: Impact of grade, sender, comparative information and message style on student affect and academic resilience. Computers in Human Behavior, 89, 8-15.

“Learning analytics is the collection, analysis, and feedback of data to learners to improve their learning (Siemens, 2013).”

Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57, 1380–1400. https://doi.org/10.1177/0002764213498851.

“Defining Learning Analytics and Tracing Historical Roots
As the field of learning analytics (LA) is further refined and established, an authoritative definition will emerge. At present, the vast majority of LA literature has begun to adopt the following definition offered in the 1st International Conference on Learning Analytics:

Learning analytics is the measurement, collection, analysis, and reporting of data about learners and their contexts, for the purposes of understanding and optimizing learning and the environments in which it occurs.”

IAD Learning: https://www.iadlearning.com/learning-analytics-2018/

“Learning Analytics 2018 – An updated perspective

There are multiple definitions of learning analytics, but the industry preferred one comes from the International Conference on Learning Analytics (LAK 2011):

“Learning analytics is the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.””

Stewart, C. (2017). Learning Analytics: Shifting from theory to practice. Journal on Empowering Teaching Excellence, 1(1), 10.

“Defining Learning Analytics
Many authors have defined learning Analytics, yet the following definitions are used in framing the focus of this paper. The Society for Learning Analytics (SoLAR, n.d.) stated that LA, “is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs”. This definition emphasizes the focus on the learner and optimization of the learning process. It also highlights the potential use of techniques in modeling, generating profiles of learners, and possibility of personalized and adaptable learning as well as others (Seimens, 2012).
Johnson and colleagues in 2014 (as cited by Firat, 2016) defined LA as “an area which focuses on reaching patterns or tendencies via data sets related to student or via large sets of educational data to maintain the development of supplementary and personalized higher education systems.” (p.76) Similar to this definition, Agudo-Peregrina et al. (2014) have emphasized the focus of LA being on discovering the “unobservable patterns and the information underlying the learning process.” (as cited by Firat, 2016, p.76) These definitions provide a vision of the potential usability and application of LA in assisting educational institutions, teachers, and even learners in improving student learning.”

Gradiant: https://www.gradiant.org/noticia/learning-analytics-current-challenges/
October 13, 2017

Learning Analytics

In the literature we can find several definitions of Learning Analytics [1]. Thinking about the approaches followed in our projects, and being pragmatic, perhaps the definition of Johnson, Adams, & Haywood 2011 [2] is one of the most successful:

“Learning analytics refers to the interpretation of a wide range of data produced by and gathered on behalf of students in order to assess academic progress, predict future performance, and spot potential issues. Data are collected from explicit student actions, such as completing assignments and taking exams, and from tacit actions, including online social interactions, extracurricular activities, posts on discussion forums and other activities that are not directly assessed as part of the student’s educational progress. The goal of learning analytics is to enable teachers and schools to tailor educational opportunities to each student’s level of need and ability. Learning analytics promises to harness the power of advances in data mining, interpretation, and modelling to improve understandings of teaching and learning, and to tailor education to individual students more effectively. Still in its early stages, learning analytics responds to calls for accountability on campuses across the country and leverages the vast amount of data produced by students in day-to-day academic activities.”

We are, therefore, talking about a process of measurement, collection, analysis and interpretation of data that allows us to take advantage of advances in several disciplines (e.g. information sciences, sociology, computer science, statistics, psychology, learning sciences, educational data mining) in order to make the most of the vast amount of data that is produced in educational activities and, from them, to adapt education to students effectively.

[1] Chatti, M., Dyckhoff, U., Schroeder, U., & Thüs, H. (2012). A Reference Model for Learning Analytics. International Journal of Technology Enhanced Learning (IJTEL) – Special Issue on “State-of-the-Art in TEL”, 318-331.

[2] Johnson, L., Adams, S., & Haywood, K. (2011). The NMC Horizon Report: 2011 K-12 Edition. Austin, Texas: The New Media Consortium.

Chatti, M., Dyckhoff, U., Schroeder, U., & Thüs, H. (2012). A Reference Model for Learning Analytics. International Journal of Technology Enhanced Learning (IJTEL) – Special Issue on “State-of-the-Art in TEL”, 318-331.

“Different definitions have been provided for the term ‘learning analytics’. Learning analytics (LA) is defined on the LAK11 website (https://tekri.athabascau.ca/analytics/) as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs”. Elias (2011) describes LA as “an emerging field in which sophisticated analytic tools are used to improve learning and education”. Siemens (2010) views LA as “the use of intelligent data, learner-produced data, and analysis models to discover information and social connections, and to predict and advise on learning”. LA is defined in EDUCAUSE’s Next Generation learning initiative as “the use of data and models to predict student progress and performance, and the ability to act on that information” (as cited in Siemens, 2010). According to Johnson et al. (2011), LA “refers to the interpretation of a wide range of data produced by and gathered on behalf of students in order to assess academic progress, predict future performance, and spot potential issues”. Although different in some details, these definitions share an emphasis on converting educational data into useful actions to foster learning. Furthermore, it is noticeable that these definitions do not limit LA to automatically conducted data analysis.”

Edutech Wiki: http://edutechwiki.unige.ch/en/Learning_analytics#Introduction

Page last modified 17:05, 29 March 2017.
“One could define learning analytics as collection of methods that allow teachers and maybe the learners to understand what is going on. I.e. all sorts of data collection, transformation, analysis and visualization tools that allow to gain insight on participant’s behaviors and productions (including discussion and reflections). The learning analytics community in their call for the 1st International Conference on learning analytics provides a more ambitious definition: “Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs”. This definition includes a clearly transformative perspective.”

Sclater, N., Peasgood, A., & Mullan, J. (2016). Learning analytics in higher education. London: Jisc. Accessed February, 8, 2017.

“The Society for Learning Analytics Research (SoLAR) was formed, and adopted probably the most oft-quoted definition of learning analytics: Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs.” Siemens & Gašević, 2012

Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64-71.

“Despite the challenges of online delivery, the adoption of educational technologies has afforded a new opportunity to gain insight into student learning. As with most IT systems, the student’s interactions with their online learning activities are captured and stored. These digital traces (log data) can then be ‘mined’ and analysed to identify patterns of learning behaviour that can provide insights into education practice. This process has been described as learning analytics. The study of learning analytics has been defined as the “measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Siemens & Gašević, 2012). Learning analytics is a bricolage field drawing on research, methods, and techniques from numerous disciplines such as learning sciences, data mining, information visualization, and psychology. This paper reviews the learning analytics research to outline a few of the major topics that the learning analytics field needs to address in order to deliver its oft cited promise for transforming education practice. In so doing, we argue that learning analytics needs to build on and better connect with the existing body of research knowledge about learning and teaching. Specifically, in this paper, we suggest how learning analytics might be better integrated into existing educational research and note the implications for learning analytics research and practice.”

Gunn, C., McDonald, J., Donald, L., Milne, J., Nichols, M., & Heinrich, E. (2015). A practitioner’s guide to learning analytics. ascilite 2015.

“Learning analytics has been variously defined (e.g. Barneveld, Arnold & Campbell, 2012; Elias, 2011; Society for Learning Analytics Research, 2012). Cooper (2012a) suggests that a single definition is impossible because of the broad range of perspectives and motivations involved. For our purpose, which is working with teachers, learning designers and learners, Cooper’s description is useful:
Analytics is the process of developing actionable insights through problem definition and the application of statistical models and analysis against existing and/or simulated future data. (p.7)”

Learning Analytics Community Exchange (LACE) website: http://www.laceproject.eu/faqs/learning-analytics/

December 15, 2014

“1. What are Learning Analytics?

There is no universally agreed definition of the term ‘learning analytics‘. One popular definition states that learning analytics are “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs[1].

In a series of briefing papers on analytics [2] the term was defined as “the process of developing actionable insights through problem definition and the application of statistical models and analysis against existing and/or simulated future data”.

Erik Duval has proposed the definition “learning analytics is about collecting traces that learners leave behind and using those traces to improve learning[3].

[1] Learning and Academic Analytics, Siemens, G., 5 August 2011, http://www.learninganalytics.net/?p=131

[2] What is Analytics? Definition and Essential Characteristics, Vol. 1, No. 5. CETIS Analytics Series, Cooper, A., http://publications.cetis.ac.uk/2012/521

[3] Learning Analytics and Educational Data Mining, Erik Duval’s Weblog, 30 January 2012,  https://erikduval.wordpress.com/2012/01/30/learning-analytics-and-educational-data-mining/