Evaluation Framework for LA

The Evaluation Framework for Learning Analytics (EFLA) is one of the outcomes of the LACE project and will be further developed and improved throughout and after the duration of the project, especially by evaluating LA tools with the EFLA survey.

Although the added value of learning analytics for learners as well as educators has clearly been recognised and the research field of learning analytics has been growing steadily, research on comparability of learning analytics tools is quite sparse. We have therefore worked towards the identification of indicators for learning analytics that will help to standardise the evaluation of learning analytics tools and have developed the Evaluation Framework for Learning Analytics (EFLA).

The first version of the framework as described in D3.1 was developed based on input form a group concept making study with experts from the learning analytics community and consists of five criteria with four indicators each.

EFLA 1
EFLA1

The results of a first evaluation study, however, indicate that several issues needed to be adjusted. Based on these results as well as a revisitation of the original GCM data and a literature review, the framework has been improved and the next version of the framework was developed. The EFLA is now split in two parts: one for learners and one for teachers as different stakeholders have different points of view on learning analytics issues and thus need to be addressed differently.

The framework now consists of four sections for the two stakeholder groups: data aspects, awareness, reflection and impact. Each section is then validated with three aspects that are adapted to the stakeholder’s point of view. The figure below shows the four sections and their indicators. The results of another series of evaluations are currently being analysed to further improve the framework.

EFLA 2
EFLA

After an evaluation within an online course and an experts workshop, the framework has been improved again. The third version still consists of four sections but the questions have been further refined.

EFLA 3
EFLA3

Currently, the framework is being applied and evaluated within the context of MOOC dashboards.

If you are interested in using EFLA to evaluate your learning analytics tool, please get in touch with us!