Each month, we highlight one of the new additions to the LACE Evidence Hub, which brings together evidence about learning analytics. You are welcome to add to the Hub site, which you can visit via a tab at the top of this page.
The evidence of the month for March 2016 is a paper by Kovanović, Gašević, Dawson, Joksimović, Baker & Hatala in the special edition of the Journal of Learning Analytics containing invited extended versions of papers presented at LAK15. The original paper at LAK15 won best paper, and this extended version adds an extra dataset.
Learning analytics makes extensive use of trace data from learners interacting with Learning Management Systems (LMS), and one of the most common uses is to derive an estimate of the time each learner spent on task, that is, engaging in particular learning activities. The authors of this paper note that while this approach is widely used, the details are often not given, and the consequences of choices made in generating these measures are not explored.
They present two experiments exploring different measures of time on task, one using data from a fully-online course at a Canadian university, and another using data from a blended course at an Australian university.
They found that time-on-task measures typically outperformed count measures for predicting student performance, but more importantly, identified that the precise choice of time-on-task measure “can have a significant effect on the overall fit of the model, its significance, and eventually on the interpretation of research findings”.
Given their findings, they argue that there are implications for research in learning analytics: “Above all is the need for more caution when using time-on-task measures for building learning analytics models. Given that details of time-on-task estimation can potentially impact reported research findings, appropriately addressing time-on-task estimation becomes a critical part of standard research practice in the learning analytics community. This is particularly true in cases where time-on-task measures are not accompanied by additional measures such as counts of relevant activities.”
They call for researchers to recognise the importance of time-on-task measures, for fuller detail to be given of the measures used, and for further investigation.
: Kovanović, V., Gašević, D., Dawson, S., Joksimović, S., Baker, R. S., & Hatala, M. (2015) Does Time-on-task Estimation Matter? Implications on Validity of Learning Analytics Findings.https://epress.lib.uts.edu.au/journals/index.php/JLA/article/view/4501: