CC-By-NC-SA

Return to main menu

Evaluation - 'The process of making judgements about the worth (costs and values) of something' (Oliver 2000)

Evaluating the portability and adaptability of your resources is a vital step in the Open Educational Resources release process. The resources need to be sufficiently robust to stand alone and be re-used outside of their original environments, and open enough for others to be able to adapt and build upon. Ensuring you have considered these issues - outlined below - will greatly increase the likelihood of others adopting and adapting your OERs.

Introduction 

In our experience, the first piece of information the potential users of a resource would like is a measure of its worth. "Is it any good? Who else has used it and how did it work for them?" academics often ask. A positive history is crucial to secure further adoption on a wider scale. However, evaluation is often carried out retrospectively (often independently) and opportunities to refine the resource during development are lost. This makes it more difficult to produce a resource which is suitable for use by a wide range of institutions.

There are some excellent resources for teaching available on the internet but it is often difficult to find other users of the same resource and descriptions of how they deploy them, their experiences and the benefits they gained from adopting it within their teaching. We now have significant budget challenges in higher education in 2010 and it is essential that we can maximise the value of all learning and teaching materials.

So, how do we evaluate effectively? How can we reduce the effort and maximise the return? There are some quite advanced frameworks available (Yuan, 2009) but we would initially recommend a simplified approach if you are new to Open Educational Resources. Bear in mind however, that a successful OER has multi-user, multi-author perspectives as it develops.

The first step is to identify the key questions: typically (based on Conole 2000) these questions might be:

• How effective is this?
• Is it more effective than the previous mechanism for teaching this? (in effect, why it was developed)
• How effectively do these [solutions / systems / resources] support learning?
• Is it more effective than face-to-face learning? (is any reduction in effectiveness minimal, and are other benefits significant?)
• How much time does the [adoption or adaption] of the resource take to make a gain?
• What skills do teachers and students need in order to use [the resource] effectively?
• Is support and encouragement available from other users?

Why Evaluate?

Performing an evaluation is essential for an open educational resource to be a success - depending on how we define success. For the resource to be useful in its original point of origin is not enough for an OER. OERs are easily adopted by others in different contexts and situations. Further and true openness is achieved by enabling others to adapt, i.e. to enhance and develop the work further (hopefully with more than one other contributor).

How to evaluate?

Do not be scared off by a potential overhead, there are evaluation frameworks available which will support gathering of information for your stakeholders. These provide a structured list of questions and issues, and tips for how to go about gathering the evidence. The example below was based on one used in an OER pilot project - complete your own questions and decide on what evidence you would capture to answer them.

 Is it worth the additional time and effort?

Yes. If the evaluation is 'bolted-on' afterwards then the retrospective work gathering the evidence can be significant and onerous. However, if the evaluation is planned into the project from the start then the workload is reduced - decisions on how the usage data, feedback, user experiences, teacher experiences etc. are captured can be planned in the development stage; the users and any contributors each have a minor workload on your behalf. The only additional workload is the analysis, which is much easier now you have decided what to capture and how.

The results of the evaluation then become valuable for publications and presentations at meetings and conferences, and may even serve to increase uptake of the resource.

Evaluation towards a goal/purpose

The goal should be for this resource on this topic to become a (hopefully the) reference resource that others take forward and enhance. Wikipedia demonstrates the model for multiple contributors quite well, including its pitfalls. However, few can doubt it has been a significant influence on education. Medpedia is a more academic project by nature and has become very popular with students. Being a wiki it is easy to enhance so you may consider a supporting wiki to be useful, whatever your resource. If you promote it through a repository there is a good chance that some wiki-like functionality might be available to gather feedback, tips and experiences with using your OER. Consider how you might achieve this. For example;

Evaluation of content (quality of materials)

• Identify a critical friend for checking content
• Process critical friend feedback into resource
• Test on pilot consumers
• Process pilot feedback
• Declare level and prior expectations of knowledge for use (educational level for instance)
• Have any 3rd party materials been evaluated separately?
• Have you added an evaluation comment form/weblink to gather 'downstream' feedback?

Evaluation of re-usability i.e. delivery, interface, installation

• Have you identified a critical friend for adoption?
• Have you declared suitable metadata, keywords?
• Have you included any existing 'adopter' reports since the first version of the resource was created?

Other factors

• Have suitable Accessibility issues been addressed?
• Try to avoid creating barriers. Provide cover documentation for non-accessible components
• Have you adhered to suitable technical standards?
• Are you aware of current best practice in this area?

Stakeholder engagement

• How were your stakeholders involved?
• How are they notified of developments?

Is the resource

• Effective?
• Useful?
• Valuable within the discipline?
• Easy to adopt?
• Easy to adapt?
• Of sufficient quality?
• Authentic? 

Was it evaluated formatively DURING development to enable a better final output?

Was it summatively evaluated by an adopter - is this in effect the end of a successful release?

We would also recommend you associate a tag with your resource (such as #bioukoer) so those who reuse your resources, write about it in their blog or discuss it in social media can be identified. 

The above should provide a starting point for you when developing, evaluating and releasing your OERs.

References

Oliver, M. & Conole, G., (2000), Assessing and enhancing quality using toolkits. Journal of Quality Assurance in Education, 8, 1, 32-37.

Conole, G (2004) The role of Evaluation in the Quality Assurance of elearning. Learning and Teaching in Action, 3, 2.
Available at http://www.celt.mmu.ac.uk/ltia/issue8/conole.shtml 

Yuan, Li (2009) Workblog - Developing a Framework for Understanding and Evaluating the Impact of Open Educational Resources

JISC Infonet (2010) Open Educational Resources Infokit

Return to main menu

Number of visitors: