Tag Archives: Evaluation

Evaluation Framework – measuring our success

The Evaluation Framework

How do we know that we are progressing towards achieving the delivery of the vision for Knowledge for Healthcare?

The aim for the Quality and Impact work-stream in delivering the vision is to enhance the quality and demonstrate value of library and knowledge services. Working with Sharon Markless, strategic development of an overarching Evaluation Framework for healthcare library and knowledge services has been produced.

The framework will enable the evaluation of library and knowledge service delivery providing data to demonstrate impact, assure quality of services, and enable service improvement.  The framework is based around six objectives defining the contribution Library and knowledge services can make to the achievement of Knowledge for Healthcare.  These have been chosen areas which are in the direct sphere of influence and action by library and knowledge services.

The Six Impact Objectives:

  • Organisations are more effective in mobilising evidence and internally generated knowledge
  • Patients, carers and the public are empowered to use information to make health and well-being choices
  • Improved consistency and increased productivity and efficiency of healthcare library and knowledge services
  • Enhanced quality of healthcare library and knowledge services
  • Partnership working is the norm in delivering knowledge to healthcare
  • Increased capability, confidence and capacity of library and knowledge services workforce

The framework will provide a clear purpose for the collection of evidence, activity and statistical collections.  The implementation of the framework will begin from July 2017 and there will be an opportunity for involvement in development and piloting of the supporting evaluation processes.

The framework and supporting documents have been published on the new blog section Evaluation Framework. There are also some Frequently Asked Questions 


Clare Edwards

clare.edwards@hee.nhs.uk 

Education Development Programme Lead, Health Education England

Chair Quality and Impact Working Group

Clinical librarians really make a difference!

Alison Brettle, Reader in Evidence Based Practice, University of Salford a.brettle@salford.ac.uk
@Brettleali

One of the proudest moments for me professionally this month was the publication of a study which demonstrates the impact of clinical librarians in the North West.  The paper has been a long time coming so it was very exciting to see it finally in print (or should that be online!).  The paper (Brettle, Maden and Payne, 2016) was the result of a number of years work and collaboration between clinical librarians working in the North West and myself.

The project really began in 2009, when I returned from the EBLIP4 conference in North Carolina. As LIHNN had kindly sponsored my conference fees I wrote an article in LIHNNK Up about the conference, expressing some frustration about the lack of evidence within our profession. My practical way forward was to suggest librarians conducted systematic reviews so we would know what evidence there was – and where the gaps were.  I also strongly believed that getting involved in systematic reviews was a good introduction to research. The clinical librarians group in the North West were interested in publishing more about the work they were doing and they got in touch, and to cut a long story short, this was the beginning of a partnership which resulted in the group undertaking a systematic review on evaluating clinical librarian services (Brettle et al, 2011).  The systematic review updated the evidence on effectiveness as well as highlighting what was needed to provide rigorous evidence to demonstrate the impact of clinical librarian services.  The next logical step was to put these findings into practice and conduct an evaluation across the North West.  This was to be the largest clinical librarian study in the UK to date, and all clinical librarians across the region were invited to participate.  Both these studies benefited from small grants from HCLU, which were key to providing a small amount of resources to get the projects off the ground.

Building on the recommendations from our systematic review, our aim was to understand the impacts of CL services within National Health Service (NHS) organisations, by

  • Using a framework that ensured consistent and robust data collection across all participants
  • Testing the Making Alignment a Priority (MAP) Toolkit (https://maptoolkit.wordpress.com) in measuring the CL contribution to organisational objectives
  • Developing research skills amongst the group of librarians involved.

The paper describes the results and the tools used. Using both questionnaires and interviews, we found that the interventions or services provided by CL’s are complex and each contributes to multiple outcomes of importance to their organisation.  So for example each literature search or participation in a journal club or current awareness bulletin could impact on multiple areas and decisions, and will be unique to each encounter.  We found that the questionnaires were useful in providing data about the outcomes to which the librarians contributed, whilst the interviews really brought this data to life, explaining how one piece of information could really contribute in a wide range of ways that are important within the NHS context.

In brief we found that clinical librarians contribute to a wide range of outcomes in the short and longer term and really do make a difference within the NHS. These include direct contributions to choice of intervention (36%) diagnosis (26%) quality of life (25%), increased patient involvement in decision making (26%) and cost savings and risk management including avoiding tests, referrals, readmissions and reducing length of stay (28%).  As well as looking at contributions to patient care, we looked at other outcomes that are important within the NHS (this is where the MAP toolkit came in), so the study is relevant across all types of NHS organisations not just acute patient care.  We were able to show that clinical librarians improve quality and help save money as well as affecting patient care directly – all key outcomes in the current NHS climate.

The third objective of the study was to help improve research skills, and this isn’t really covered in the paper.  The approach we used built on that used in the systematic review project (Brettle et al, 2011) and has since been described as a “hive approach” (Buckley-Woods and Booth, 2013).  Another way of describing it is “doing with” rather than “doing for”. As an experienced researcher I directed and guided the research but it was very much a partnership and mentoring relationship where the clinical librarians really contributed to the research (and it wouldn’t have taken place if they hadn’t done so).  For this project the clinical librarians were invited to take part in the research at a level that worked for them.  For example, some participated in the survey part whereas others took part in the interviews, interview analysis, and writing up the results.  At the questionnaire design stage, a small group drafted the questionnaire, as a group this was discussed and modified to suit everyone’s needs, and then piloted on each service.  Standard documents were developed and provided to all, on how to conduct the pilot and how to obtain ethical and governance approval.  Meetings were used for agreeing procedures and training.  A wiki was used to share and update resources.  At the interview stage, meetings were held to develop the interview schedule and provide training to those taking part in this stage.  Librarians were “buddied” and conducted interviews in each others organisations (to enhance rigour) but with the advantage that the buddies could practice on each other as well as bounce ideas (and fears!).

The tools developed in the project have informed the development of a simpler generic tool for use across all health libraries and have been incorporated in a revised impact toolkit, for those who want to conduct more in-depth, rigorous impact studies.  In terms of further research what we need to do next is find out whether this approach of building research capacity has made a difference in the longer term.  If this is the case we can use this approach more widely to develop the evidence base of health libraries and librarians for the future.

Acknowledgements
This project wouldn’t have been possible without the librarians involved.  Thanks to: Michelle Maden-Jenkins, Clare Payne, Helen Medley, Tracey Pratchett, Michael Reid, Debra Thornton, Rosalind McNally, Pippa Orr, Morag Platt, Denise Thomas, Anne Webb, Riz Zafar

References
Brettle, A., Maden, M., Payne, C. (2016) The impact of clinical librarian services on patients and health care organisations, Health Information and Libraries Journal. Available from: http://dx.doi.org/10.1111/hir.12136

Brettle, A., Maden-Jenkins, M., Anderson, L., McNally, R., Pratchett, T., Tancock, J., Thornton, D. and Webb,A. A 2011, ‘Evaluating clinical librarian services: a systematic review’ , Health Information & Libraries Journal, 28 (1) , pp. 3-22.

Buckley-Woods, H. and Booth, A. (2013) What is the current state of practitioner research: the 2013 LIRG scan.  Library and Information Research, 37(116).  Available from: http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/598

Hello from the Value and Impact Task and Finish Group

Value and Impact is one of two task and finish groups operating within the Quality and Impact work stream of the Knowledge for Healthcare programme.  The group has the remit of refreshing the online Impact Toolkit with the aim of broadening its relevance to library services working outside the acute sector, and to increase the range of tools available to enable the assessment of impacts upon areas such as management and commissioning.

The group first met in May 2015 and decided early in its development that an agreed definition of impact would be valuable in establishing a common baseline.  The adopted definition was taken from ISO 16439:2014 Information and documentation — Methods and procedures for assessing the impact of libraries:

Difference or change in an individual or group resulting from the contact with library services

As preparatory work a scoping literature search was undertaken together with an analysis of fully compliant LQAF submissions for Criterion 1.3C.  These were supplied by LKSL Leads with permission of local managers (3 regions returned, 24 fully compliant submissions).

Consultation with the NHS Library workforce was essential before work on refreshing the toolkit could begin.  A committed and enthusiastic reference group assisted with the creation and testing of a SurveyMonkey-based questionnaire looking at how (if at all) library staff made use of the Impact Toolkit, whether they used any alternatives, what they did with the results, and invited suggestions for improvement of the toolkit resource.

The survey was open during August 2015 and received responses from 136 services (63% based on figures reported in Knowledge for Healthcare) with representation from all the LETBs.

An initial analysis of the responses to the survey shows that although over 95% of respondents already collect impact data, reported usage of the toolkit itself is quite low.  Looking at what impact information is used for, the evaluation and modification of services came out top with 89% while publicity and marketing (79%) and justification for business cases/funding bids (60%) were also popular responses.

Over the next few weeks we will be analysing the written responses to the survey and will report on these in the next blog.  We will then move on to work on refreshing the toolkit based on results and suggestions from the survey and other evidence.  The target date for completion of the toolkit refresh is the end of March 2016.

Membership of the Value and Impact Task and Finish Group

  • Jenny Turner – East Sussex Healthcare NHS Trust (Chair)
  • Stephen Ayre – George Eliot Hospital NHS Trust
  • Douglas Knock – King’s College Hospital NHS Foundation Trust
  • Rebecca Mitchelmore – Isle of Wight NHS Trust
  • Susan Smith – Mid Cheshire Hospitals NHS Foundation Trust
  • Sophie Pattison – University College London
  • Dominic Gilroy – Leeds and York Partnership NHS Foundation Trust