AIED 2017 Tutorial on Matching Techniques

Attention AIED Tutorial attendees:

Please email vivek@athabascau.ca for a copy of the presentation.

Matching Techniques: Hands-on Approach to Measuring and Modeling Educational Data (Tutorial)

Vivekanandan Kumar, David Boulanger, Shawn N. Fraser
School of Computing and Information Systems
Athabasca University, Canada
vivek@athabascau.ca

Abstract. This tutorial will introduce three matching techniques (Coarsened Exact Matching, Mahalanobis Distance Matching, and Propensity Score Matching) and three data imbalance metrics (L1 vector norm, Average Mahalanobis Imbalance, and Difference in Means) to assess the level of data imbalance within matched sample datasets in an interactive setting. It explains key traits of observational studies that are relevant for AIED, considering comparable traits of fully randomized experiments. Using randomized and non-randomized data, participants will conduct an observational study by approximating blocked randomized experiments. The hands-on session specifically targets skills that will enable participants to run observational studies using R packages such as MatchingFrontier, CEM, and MatchIt through an interactive Shiny web application and programmatically by writing an R script. A discussion on a matching-based observational study design for a learning analytics application that uses large, fine-grained, and self-similar datasets concludes the tutorial.

Keywords: matching • propensity score matching • randomized experiment • interactive analysis • observational study • learning analytics • data imbalance • causality

1. Objectives

This tutorial introduces observational study; explains matching techniques like Propensity Score Matching [1], Coarsened Exact Matching [2], and Mahalanobis Distance Matching [3] along with their corresponding imbalance metrics, that is, L1 vector norm, Average Mahalanobis Imbalance, and Difference in Means; offers a hands-on observational study with randomized and non-randomized data [4-6] using R libraries (MatchingFrontier [7], CEM, and MatchIt) and the web application framework for R called Shiny; and discusses ways to measure impact of learning analytics applications.

2. Audience

The tutorial targets AIED researchers, data scientists, and teachers. Some background in statistics (e.g. descriptive statistics, probability, analysis of variance) and research methods (e.g. randomized designs, observational studies) is an asset.

3. Outcomes

  • Describing experimental methods and studies in education/learning analytics
  • Proposing a valid observational study design using matching
  • Comparing different matching techniques: Coarsened Exact Matching,
    Mahalanobis Distance Matching, and Propensity Score Matching
  • Demonstrating the suboptimality of Propensity Score Matching, the most popular matching technique in observational studies [8]
  • Measuring the accuracy (in terms of data imbalance) of the proposed design against a randomized experiment
  • Performing interactive observational studies using Shiny/R
  • Discussing why it is important to have valid designs of observational studies; whether machine learning deals mainly with observational data; what is the real impact of handling properly observational data on learning analytics

4. Tools & Installation Instructions

Internet connectivity will be required for participants. Participants can optionally bring a laptop (Windows Vista/7/8/10, Mac OS X, Linux) to install R and RStudio and to use a web browser for accessing instructional materials and the Shiny web application.

  1. Install RStudio Desktop Free Edition (requires R 2.11.1+): https://www.rstudio.com/products/rstudio/download2/
  2. If you do not have R installed on your computer, install the latest version of R: https://cran.rstudio.com/
  3. Download the tutorial’s materials by clicking here.
  4. Watch a demo of the MatchingFrontier software in an interactive Shiny web application by clicking here.

5. Observational Studies

Given the discriminatory nature of completely randomized experiments and the ethical issues that they raise in educational settings, observational studies are being investigated in educational research to supplement and possibly replace randomized experiments. The research community at large refers to the randomized experiment as the gold standard [9-13] and many view observational studies as “having less validity because they reportedly overestimate treatment effects” [10]. The results of observational studies are disputed since they may contain undetected confounding bias. On the other hand, one should not oversimplify the benefits of randomized experiments [11]. Silverman [12] indicates that observational studies can complement findings in randomized experiments by using a larger and more diverse population over longer follow-up periods.

This tutorial pursues a design of observational study using matching techniques as prescribed by King [8], where new sensors are increasingly available to better observe/record teaching and learning experiences at real time. It will demonstrate the embedding of observational sensors as part of learning analytics processes and will advance blocked randomized experiments as measurements of impact of analytics. It strives to empower teachers themselves to step into the roles of analytics researchers using Shiny’s interactive analyses.

6. Tutorial Interaction

The tutorial is designed to be 1/3 presentation, 1/3 hands-on, and 1/3 discussion. Participants, in small groups, will discuss key traits of matching methods, imbalance metrics, and key differences between observational studies and randomized experiments. They will have an opportunity to work, individually or in small groups, with hands-on data, tools, and models to perform an observational study [14] using Coarsened Exact Matching, Mahalanobis Distance Matching, or Propensity Score Matching. For those who desire, they will also interact in small groups to respond to different types of research questions using an interactive Shiny web application. Different levels of participation will be offered: 1) listening to the presentation (every step will be shown on slides), 2) a web application will be available for non-programmer participants to run their analyses without any coding activity, and 3) an R script will be available for those who are interested in programming directly some portions of the analyses.

References

  1. Olmos, A., & Govindasamy, P. (2015). Propensity Scores: A Practical Introduction Using R. Journal of MultiDisciplinary Evaluation, 11(25), 68–88.
  2. Iacus, S. M., King, G., Porro, G., & Katz, J. N. (2012). Causal inference without balance checking: Coarsened exact matching. Political Analysis, 1–24.
  3. King, G., Nielsen, R., Coberley, C., & Pope, J. E. (2011). Comparative Effectiveness of Matching Methods for Causal Inference. Unpublished Manuscript, 15, 1–26. http://doi.org/10.1.1.230.3451
  4. LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 604–620.
  5. Dehejia, R. H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053–1062.
  6. Dehejia, R. H., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and Statistics, 84(1), 151–161.
  7. King, G., Lucas, C., & Nielsen, R. (2014). The Balance-Sample Size Frontier in Matching Methods for Causal Inference. American Journal of Political Science.
  8. King, G., & Nielsen, R. (2016). Why propensity score should not be used for matching, (617).
  9. Hannan, E. L. (2008). Randomized Clinical Trials and Observational Studies: Guidelines for Assessing Respective Strengths and Limitations. JACC: Cardiovascular Interventions, 1(3), 211–217. http://doi.org/http://dx.doi.org/10.1016/j.jcin.2008.01.008
  10. Concato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs. The New England Journal of Medicine, 342(25), 1887–1892.
  11. Medical Publishing Internet, Kent W. The advantages and disadvantages of observational and randomised controlled trials in evaluating new interventions in medicine. Educational article [Internet]. Version 1. Clinical Sciences. 2011 Jun 9. Available from: https://clinicalsciences.wordpress.com/article/the-advantages-and-disadvantages-of-1blm6ty1i8a7z-8/.
  12. Silverman, S. L. (2009). From Randomized Controlled Trials to Observational Studies. The American Journal of Medicine, 122(2), 114–120. http://doi.org/http://dx.doi.org/10.1016/j.amjmed.2008.09.030
  13. At Work, Issue 83, Winter 2016: Institute for Work & Health, Toronto.
  14. Sullivan, G. M. (2011). Getting Off the “Gold Standard”: Randomized Controlled Trials and Education Research. Journal of Graduate Medical Education, 3(3), 285–289. http://doi.org/10.4300/JGME-D-11-00147.1