Evaluation Framework Frequently Asked Questions

Evaluation Framework – FAQs

What is an Evaluation Framework?

An Evaluation Framework is designed to provide a coherent approach to evaluating impact. It focuses on the difference a service wants to, or needs to make, then works systematically to identify how you will know you are getting there and what you need to do to make this likely.  The overall emphasis in any impact evaluation framework is on achieving outcomes, where an outcome is defined as:

“Changes in behaviour, relationships, activities or actions of people, groups and organisations with whom a programme works directly”.

(Earl, S. et al. (2001) Outcome Mapping: Building Learning and Reflection into Development Programs.  Ottawa:  International Development Research Centre. p.1)

Why is an Evaluation Framework needed?

The Evaluation Framework has been developed to meet the need to assess progress towards achieving the delivery of Health Education England’s vision for Knowledge for Healthcare and to demonstrate the significant contribution made by library and knowledge services.

What will the Evaluation Framework achieve?

The framework will enable the evaluation of library and knowledge services delivery providing data to demonstrate impact, assure quality of services, measure performance and enable service improvement.

The framework therefore provides an outcomes-based, rather than process based approach, to evaluation and performance.  The evidence collected will also support advocacy and raising the profile of library and knowledge services.

Am I expected to use the Evaluation Framework for my service?

The Evaluation Framework is a good tool to support both service improvement and advocacy for your own service. Many of the indicators do relate to your own service and therefore will show how your service is contributing to the overall vision of Knowledge for Healthcare and against the six outcomes of the framework.  The evidence you will be collecting for national submission will therefore also be valuable in the evaluation of your own service.

How was the Evaluation Framework developed?

The framework development has been facilitated by Sharon Markless using an evidence based process for evaluating the impact of library services which has been tested and adopted across the world.  The methodology is based on an outcomes mapping approach which is used by business worldwide.

The methodology involved three key initial stages

  1. Agreeing the focus by identifying impact areas (where do you want to make a difference?) and then setting a specific objective in each area.
  2. Identifying the impact indicators for each objective that taken together will show whether the desired impact is occurring
  3. Planning evidence collection by identifying the most appropriate impact evidence for indicators.

The second part of the process involved identifying key actions that need to be taken to achieve the desired impacts and how to monitor these.

Who was involved in developing the Evaluation Framework?

The group involved in the development of the framework included HEE’s strategic library and knowledge services leads, Chairs of Knowledge for Healthcare working groups and library and knowledge services managers.

The composition of the group was critical to ensure that the elements of the framework reflected both strategic and operational requirements to demonstrate transformational change

Those involved:

  • Sue Lacey Bryant, Senior Advisor, Knowledge for Health, HEE
  • Clare Edwards, Education Development Programme Lead,  HEE West Midlands
  • Louise Goswami, Head of Library and Knowledge Services Development,  HEE Kent Surrey Sussex
  • Helen Bingham, Head of Knowledge Services and Technology Enhanced Learning, HEE South
  • David Stewart, Director of Health Libraries, HEE North
  • Richard Osborn, Strategic Library Services Development Manager, HEE London
  • Dominic Gilroy, Library and Knowledge Services Development Manager,  HEE Yorkshire and the Humber
  • Doug Knock, Library and Knowledge Services Manager, Kings College London NHS Foundation Trust
  • Alan Fricker, Head of NHS Partnership and Liaison, Kings College London

The work to oversee the implementation of the Evaluation Framework is being led by the Quality and Impact Working Group.

Will the Evaluation Framework replace the Library Quality Assurance Framework (LQAF)?

The Evaluation Framework is not the same as a Quality Assessment Framework.

The Evaluation Framework is concentrating on evaluating progression against the Knowledge for Healthcare vision while the Quality Framework is used to audit the quality of the service and provides a framework to focus, target and improve the quality of library and knowledge services.

A key priority of the Quality and Impact Working Group is to review and develop new quality standards for library and knowledge services across the NHS.  Most of these new standards will align to the Evaluation Framework and the evidence collected will be used to demonstrate progress against both.

The new Evaluation Framework and Quality Standards will integrate in to HEE’s quality framework to assure the quality of library and knowledge services.

Will there be a requirement for more national data/information collections?

The framework will provide a clear purpose for the collection of LKS evidence, activity and statistical collections.  The current collections will be reviewed to identify the data and information requirements and the frequency for collection.   A single evaluation assessment process will be developed to support this process.  It is not expected that the amount of data collected will increase however the focus will change.

What is the difference between the Evaluation Framework and the Value and Impact Toolkit?

The Value and Impact Toolkit helps users to understand what impact is and how best to measure and demonstrate this for different stakeholders.  It offers tools to assist in this process, including templates which can be used to record and demonstrate this impact to the wider organisation and beyond.

Evidence of impact case studies and narratives will be key evidence sources to demonstrate the outcome indicators within the Evaluation Framework.  The Value and Impact Toolkit is therefore key in generating the evidence for the Evaluation Framework.

When will the Evaluation Framework start to be used?

The Evaluation Framework is now published and we will be starting to use it from 2017/18 onwards.  However it is estimated that it will take until March 2020 to fully implement the framework.

The implementation of the framework will begin in July 2017 and there will be an opportunity for involvement in development and piloting of the supporting evaluation processes.

Please add a comment to ask any additional questions which you would like us to address.