Table of Contents:
- The TELT Evaluation Framework Vision
- Why the Need for a TELT Evaluation Framework?
- What is the TELT Evaluation Framework?
- What the TELT Evaluation Framework is not
- Framework Structure Overview
- Categories of TELT Applications
- Evaluation Framework – Stage 3
- Generic Usability Evaluation
- Evaluation Framework Documents
A note for the reader
The TELT Evaluation Framework, developed in 2009 through to 2011 at the University of New South Wales (UNSW), is designed to undergo iterative cyclical refinement and ongoing development, based primarily on the results of the sub-layer evaluations and the ever-changing staff, student and application landscape in which it is applied. This work has now evolved to become the Learning Technology Evaluation Framework (LTEF.)
It is emphasised that as a result of the ongoing refinement cycle, the nature, validity and applicability of the reviewed literature, the proposed processes, the composition of the survey instruments, and the construction of the sub-layers are all likely to change in the future and undergo further refinement and improvement in order to adapt to the evolving social, technological, and institutional milieu.
Readers are encouraged to use the LTEF documents available through this site as the basis for understanding the aims, goals and structure of the Framework, and to access the most up-to-date information on the Framework refinement, and the latest research and evaluation outcomes.
Questions and suggestions are always welcome – always please feel free to contact me for clarification on any aspect of the TELT (LTEF) documents and its refinements.
About the TELT Evaluation Framework:
Evaluation Framework Vision
To devise a “living” evaluation framework that affords an iterative refinement process to underpin:
- All TELT technology selections and developments
- Efficient tracking of the trajectory of success (and failure) of educational technologies used across faculties and over time
- Enhancement of the divergent L & T approaches and practices in online learning at UNSW
Why the need for a TELT Evaluation Framework?
The rationale and argument for devising a generic evaluation framework is:
- Lack of availability of a widely recognised framework
- The need to avoid the many bias inherent in most frameworks
- The need to position L&T at the top of a hierarchy of evaluation criteria consistent with the TELT vision
- Identify the principles and models on which the framework is to be established.
An extensive literature search conducted over several months has confirmed the difficulty in identifying a comprehensive, generic approach and methodologies for facilitating the selection and evaluation of the effectiveness of educational technology applications and solutions that focus exclusively on serving the needs of learning and teaching. Pedagogical concerns over the actual learning effectiveness of various eLearning applications have increased alongside the meteoric rise in eLearning popularity and implementation.
The chief criticism is that the increasingly feature-rich technological platforms are not properly developed for an appropriate learning context, often improperly integrated and under-utilized, and in many situations simply become an expensive means of indexing ordinary content, adding little educational value. The TELT Evaluation Framework has thus been developed as a comprehensive, flexible and continuous approach to assess and evaluate the suitability of different educational technologies for use within the UNSW TELT Platform, as guided by informed Learning and Teaching practice and research.
What is the TELT Evaluation Framework?
For UNSW, the priority for its TELT evaluation strategy is to devise a theoretical framework from which to assess the suitability of selected educational technologies for inclusion in the TELT platform. The Evaluation Framework is an extensive, multi-faceted tool for providing an evidence-based assessment of learning and teaching technologies that may be used by the University.
Once a technology has passed through a three stage evaluation process and been approved for general use, a perpetual, iterative process of evaluation is initiated to assess the educational effectiveness of that technology in terms of the depth of student comprehension and the measurable improvements to the teaching and learning processes.
What the TELT Evaluation Framework is not
The Framework is designed to be underpinned by educational and eLearning theory and to learn from and to improve on the results and experiences of other eLearning evaluations, with the aim of improving the TELT environment within the University.
It is acknowledged that some staff may misunderstand the Evaluation, perceiving it as an assessment of their skills, teaching practices, or ability. In reality, the Evaluation is focused on understanding and improving eLearning technologies now and for the future, and is thus very critical of the usability, true learning effectiveness, and integration with student learning environments. It is NOT in any way intended to be an assessment of the performance or expertise of individual teaching staff.
Framework Structure Overview
The Framework itself incorporates continual effectiveness re-assessment of the evaluation processes: how well they identify suitable technology applications and tools for inclusion in the platform, and to what extent they improve technology supported learning and teaching. Thus, the Evaluation is conducted over the entire life-cycle of the technologies, delineated across three stages:
- Stage 1: An initial vendor assessment: to determine the suitability, stability and viability of the prospective vendors/open-source solutions.
- Stage 2: A pre-pilot evaluation score-sheet: to determine the suitability of technology for full pilot evaluation testing in Stage 3. Stage 2 assesses the features and functionality of the application from educational and technical perspectives.
- Stage 3: An on-going full-pilot evaluation: to determine suitability for the TELT Platform, as measured by the educational value of the technology application.
Categories of TELT Applications
To permit relevance to different types of TELT applications (categorised for example as large-scale LMS, medium-scale virtual classroom or collaborative suites, and small-scale special purpose software), evaluation methodologies are to be identified to address the specific requirements of each category. The categories of TELT Applications that have been identified, include:
- LMS – Learning Management Systems
- CMC – Computer-Mediated Communication Systems
- PDS – Plagiarism Detection Systems
- CAA – Computer Assisted Assessment Tools
- EMS – Educational Multimedia Software
- LC – Lecture Capture Systems
- Smaller-scale Special Purpose Software
Evaluation Framework – Stage 3
Stage 3 begins an ongoing L&T focused evaluative sub-cycle that aims to facilitate a culture of evidence-based excellence at the faculty and institutional levels to improve UNSW technology supported L&T practices.
Stage 3 determines the suitability of the TELT technology for inclusion in the TELT Platform as a whole. Stage 3 consists of three layers:
- Pedagogy Layer
- Technical Layer
- Business Layer
The Pedagogy Layer is emphasised first and foremost as it is directly informed by the UNSW Graduate Attributes, Learning and Teaching Principles and the 16 Guidelines for Learning and Teaching Practices. The evaluation criteria specified for this layer are also guided by the teaching approaches and models applied in faculties and schools.
The pedagogical criteria for evaluating a TELT technology are further divided into three sub-layers:
- The educational value and usefulness of the selected application
- The learning effectiveness of the TELT technologies
- The learning environments that use TELT technologies
- The Pedagogy Layer as described above
- The Business Layer reflects the formalised processes of organisational change at UNSW that are informed first by pedagogical and then by technological factors.
- The Technical Layer reflects the need to account for the technical requirements, standards and protocols established at UNSW that are informed first by the pedagogical and then the business process criteria.
Generic Usability Evaluation
In conjunction with the overarching concerns over evaluating the actual pedagogical (learning) effectiveness of integrated eLearning environments as assessed through various educational theories and methodologies, it is also important to begin considering usability and usefulness as two fundamental factors that create the foundation for eLearning applications to be both conducive to effective learning and supportive of positive pedagogical strategies, thus having education value.
Prior to undertaking a comprehensive assessment of the learning effectiveness of an eLearning application, a preliminary examination should consider the application’s usability, usefulness, and educational value as a function of usability and usefulness within a learning and teaching context.
If the usability of a particular technology application is a hindrance to the users’ learning and teaching processes, and if it is not evaluated for separately, the entire evaluation could present a biased impression of the overall educational value and subsequent learning effectiveness of the application.
On a practical level, the six-stage heuristic life-cycle forms an evidence-based starting point for the creation of tailored and thematically grouped questions revolving around the entire application usability experience for surveys and other instruments, in turn providing a sound basis for validating the Evaluation process. The questions aim to assess the most common usability, usefulness and educational value attributes, and typical usability problems, as identified through our extensive evaluation literature review, which builds on the review undertaken in the supporting document titled : “Evaluation of the TELT platform: Essential elements and methodologies”.
It is the nexus of various ad-hoc usability elements and assessable attributes identified through research that creates the ephemeral, intuitive “look and feel” of user experience and satisfaction that has been so inherently difficult to isolate and evaluate.
The Generic Usability Evaluation relies heavily on educational and technology literature — please find literature references and supporting documentation for each heuristic usability element, in the comprehensive reports available for download.
Evaluation Framework Documents
Please access and download other documents listed under Publications then LTEF to learn more about the Evaluation Framework (LTEF). The document displayed on this page can be downloaded here.