Evaluation Framework for LA

The Evaluation Framework for Learning Analytics (EFLA) is one of the outcomes of the LACE project and will be further developed and improved throughout and after the duration of the project, especially by evaluating LA tools with the EFLA survey.

Although the added value of learning analytics for learners as well as educators has clearly been recognised and the research field of learning analytics has been growing steadily, research on comparability of learning analytics tools is quite sparse. We have therefore worked towards the identification of indicators for learning analytics that will help to standardise the evaluation of learning analytics tools and have developed the Evaluation Framework for Learning Analytics (EFLA).

The first version of the framework as described in D3.1 was developed based on input form a group concept making study with experts from the learning analytics community and consists of five criteria with four indicators each.

EFLA 1
EFLA1

The results of a first evaluation study, however, indicate that several issues needed to be adjusted. Based on these results as well as a revisitation of the original GCM data and a literature review, the framework has been improved and the next version of the framework was developed. The EFLA is now split in two parts: one for learners and one for teachers as different stakeholders have different points of view on learning analytics issues and thus need to be addressed differently.

The framework now consists of four sections for the two stakeholder groups: data aspects, awareness, reflection and impact. Each section is then validated with three aspects that are adapted to the stakeholder’s point of view. The figure below shows the four sections and their indicators.

EFLA 2

After an evaluation within an online course and an experts workshop, the framework has been improved again. The third version still consists of four sections but the questions have been further refined.

EFLA 3

The third version was then used to evaluate several widgets of a MOOC platform’s learning analytics dashboard. After conducting a principal component analysis as well as a reliability analysis, the framework has been improved and refined again. For both learners and teachers the fourth version now consists of three dimensions and eight items in total.

EFLA 4

If you are interested in using EFLA to evaluate your learning analytics tool, please get in touch with us!