Tag Archives: Evaluation

Can you find out the longer-term impact of the “Making LKS Business Critical” Workshops?

This was the challenge set for the Knowledge for Healthare Continuing Professional Development (CPD) Group. Anne Gray delivered Making LKS Business Critical workshops between January 2017 and Spring 2018 and 148 people attended. This was a perfect opportunity to test the Impact Evaluation Form developed by the group in 2017.

The Evaluation Form was adapted into an online survey and sent to all particpants. 29% responded and here are some of the key findings:

Word cloud of summarised general feedback
  • 87% of respondents said that the information, knowledge or skills gained have led, or might lead, to improving service delivery to customers.  Over 60% respondents said that personal or professional development and sharing information with, or advising , other staff or colleagues were gained.
  • 68% respondents said they gained new knowledge and 58% considered that the information, knowledge or skills gained had generated new ideas.
  • 53% respondents stated that the training had impacted by contributing to service development and delivery and 5% indicated that the training had an impact by saving money or contributed to financial effectiveness
  • Of those that had not used the skills or knowledge from the session 26% said they will probably use it in the future.  16% cited lack of time/capacity, 11% stated that the session did not meet their expectations.
  • Replies to the question What have you done differently as a result of attending this event? were grouped into the following themes:  Approach to searches, outlook about handling this type of enquiry, approach to presenting search information, use of specific resources, skill development and confidence in approaching senior management. 

            “I am more confident in offering and providing this service to the members of  the Trust’s board”

  • When asked about the impact upon service or service users, responses included greater consideration of business objectives, improved quality of service and saving users’ time.  A couple of responses were very specific:

     “In the recent organisational change, I managed to get recognition of the chartered librarian as a job profile and ensure me a role with my teams for the next 3 years.”

     “I was able to do an evidence search for a manager in the Trust shortly after attending the course. I was able to expand the search beyond the usual HDAS which led me to find more pertinent results. I regularly use the additional sources in searches now”

  • Respondents were asked to describe any ideas or work undertaken to consider the specific objective of the course – to generate ideas or work for collaborative resources to support LKS knowledge services to NHS Managers/Commissioners.  10 respondents said they had undertaken work or had ideas and these ranged from working collaboratively with other services to provide a business critical information service to more shared evidence bulletins.

Seeking the longer term impact of training is not always straight-forward – the reponse rate is often low as people have moved roles or are facing a change in cirucmstances since attending the workshop and at least one respondent had difficulty completing the longer term impact evaluation as they felt too much time had lapsed between taking the workshop and completing the evaluation. However, the responses that we did receive showed that some respondents welcomed the opportunity to re-visit the workshop materials and that significant developments had resulted from using the newly acquired skills and knowledge. Click below to download the full evaluation responses:

The CPD Group would like to express thanks to Anne for using her workshops as a”guinea pig” for the impact evaluation and to all of those who took the time to complete the survey.

Don’t forget that the full Development Needs Analysis consultation will begin in the Autumn this year. This will be your opportunity to say what skills and knowledge devevelopment you need for your role now and in the future. Start planning now by taking a look at the Health Professional Knowledge and Skills Base (PKSB) to help you spot any gaps.

On behalf of the CPD Group

Evaluation Framework – measuring our success

The Evaluation Framework

How do we know that we are progressing towards achieving the delivery of the vision for Knowledge for Healthcare?

The aim for the Quality and Impact work-stream in delivering the vision is to enhance the quality and demonstrate value of library and knowledge services. Working with Sharon Markless, strategic development of an overarching Evaluation Framework for healthcare library and knowledge services has been produced.

The framework will enable the evaluation of library and knowledge service delivery providing data to demonstrate impact, assure quality of services, and enable service improvement.  The framework is based around six objectives defining the contribution Library and knowledge services can make to the achievement of Knowledge for Healthcare.  These have been chosen areas which are in the direct sphere of influence and action by library and knowledge services.

The Six Impact Objectives:

  • Organisations are more effective in mobilising evidence and internally generated knowledge
  • Patients, carers and the public are empowered to use information to make health and well-being choices
  • Improved consistency and increased productivity and efficiency of healthcare library and knowledge services
  • Enhanced quality of healthcare library and knowledge services
  • Partnership working is the norm in delivering knowledge to healthcare
  • Increased capability, confidence and capacity of library and knowledge services workforce

The framework will provide a clear purpose for the collection of evidence, activity and statistical collections.  The implementation of the framework will begin from July 2017 and there will be an opportunity for involvement in development and piloting of the supporting evaluation processes.

The framework and supporting documents have been published on the new blog section Evaluation Framework. There are also some Frequently Asked Questions 


Clare Edwards

clare.edwards@hee.nhs.uk 

Education Development Programme Lead, Health Education England

Chair Quality and Impact Working Group

Clinical librarians really make a difference!

Alison Brettle, Reader in Evidence Based Practice, University of Salford a.brettle@salford.ac.uk
@Brettleali

One of the proudest moments for me professionally this month was the publication of a study which demonstrates the impact of clinical librarians in the North West.  The paper has been a long time coming so it was very exciting to see it finally in print (or should that be online!).  The paper (Brettle, Maden and Payne, 2016) was the result of a number of years work and collaboration between clinical librarians working in the North West and myself.

The project really began in 2009, when I returned from the EBLIP4 conference in North Carolina. As LIHNN had kindly sponsored my conference fees I wrote an article in LIHNNK Up about the conference, expressing some frustration about the lack of evidence within our profession. My practical way forward was to suggest librarians conducted systematic reviews so we would know what evidence there was – and where the gaps were.  I also strongly believed that getting involved in systematic reviews was a good introduction to research. The clinical librarians group in the North West were interested in publishing more about the work they were doing and they got in touch, and to cut a long story short, this was the beginning of a partnership which resulted in the group undertaking a systematic review on evaluating clinical librarian services (Brettle et al, 2011).  The systematic review updated the evidence on effectiveness as well as highlighting what was needed to provide rigorous evidence to demonstrate the impact of clinical librarian services.  The next logical step was to put these findings into practice and conduct an evaluation across the North West.  This was to be the largest clinical librarian study in the UK to date, and all clinical librarians across the region were invited to participate.  Both these studies benefited from small grants from HCLU, which were key to providing a small amount of resources to get the projects off the ground.

Building on the recommendations from our systematic review, our aim was to understand the impacts of CL services within National Health Service (NHS) organisations, by

  • Using a framework that ensured consistent and robust data collection across all participants
  • Testing the Making Alignment a Priority (MAP) Toolkit (https://maptoolkit.wordpress.com) in measuring the CL contribution to organisational objectives
  • Developing research skills amongst the group of librarians involved.

The paper describes the results and the tools used. Using both questionnaires and interviews, we found that the interventions or services provided by CL’s are complex and each contributes to multiple outcomes of importance to their organisation.  So for example each literature search or participation in a journal club or current awareness bulletin could impact on multiple areas and decisions, and will be unique to each encounter.  We found that the questionnaires were useful in providing data about the outcomes to which the librarians contributed, whilst the interviews really brought this data to life, explaining how one piece of information could really contribute in a wide range of ways that are important within the NHS context.

In brief we found that clinical librarians contribute to a wide range of outcomes in the short and longer term and really do make a difference within the NHS. These include direct contributions to choice of intervention (36%) diagnosis (26%) quality of life (25%), increased patient involvement in decision making (26%) and cost savings and risk management including avoiding tests, referrals, readmissions and reducing length of stay (28%).  As well as looking at contributions to patient care, we looked at other outcomes that are important within the NHS (this is where the MAP toolkit came in), so the study is relevant across all types of NHS organisations not just acute patient care.  We were able to show that clinical librarians improve quality and help save money as well as affecting patient care directly – all key outcomes in the current NHS climate.

The third objective of the study was to help improve research skills, and this isn’t really covered in the paper.  The approach we used built on that used in the systematic review project (Brettle et al, 2011) and has since been described as a “hive approach” (Buckley-Woods and Booth, 2013).  Another way of describing it is “doing with” rather than “doing for”. As an experienced researcher I directed and guided the research but it was very much a partnership and mentoring relationship where the clinical librarians really contributed to the research (and it wouldn’t have taken place if they hadn’t done so).  For this project the clinical librarians were invited to take part in the research at a level that worked for them.  For example, some participated in the survey part whereas others took part in the interviews, interview analysis, and writing up the results.  At the questionnaire design stage, a small group drafted the questionnaire, as a group this was discussed and modified to suit everyone’s needs, and then piloted on each service.  Standard documents were developed and provided to all, on how to conduct the pilot and how to obtain ethical and governance approval.  Meetings were used for agreeing procedures and training.  A wiki was used to share and update resources.  At the interview stage, meetings were held to develop the interview schedule and provide training to those taking part in this stage.  Librarians were “buddied” and conducted interviews in each others organisations (to enhance rigour) but with the advantage that the buddies could practice on each other as well as bounce ideas (and fears!).

The tools developed in the project have informed the development of a simpler generic tool for use across all health libraries and have been incorporated in a revised impact toolkit, for those who want to conduct more in-depth, rigorous impact studies.  In terms of further research what we need to do next is find out whether this approach of building research capacity has made a difference in the longer term.  If this is the case we can use this approach more widely to develop the evidence base of health libraries and librarians for the future.

Acknowledgements
This project wouldn’t have been possible without the librarians involved.  Thanks to: Michelle Maden-Jenkins, Clare Payne, Helen Medley, Tracey Pratchett, Michael Reid, Debra Thornton, Rosalind McNally, Pippa Orr, Morag Platt, Denise Thomas, Anne Webb, Riz Zafar

References
Brettle, A., Maden, M., Payne, C. (2016) The impact of clinical librarian services on patients and health care organisations, Health Information and Libraries Journal. Available from: http://dx.doi.org/10.1111/hir.12136

Brettle, A., Maden-Jenkins, M., Anderson, L., McNally, R., Pratchett, T., Tancock, J., Thornton, D. and Webb,A. A 2011, ‘Evaluating clinical librarian services: a systematic review’ , Health Information & Libraries Journal, 28 (1) , pp. 3-22.

Buckley-Woods, H. and Booth, A. (2013) What is the current state of practitioner research: the 2013 LIRG scan.  Library and Information Research, 37(116).  Available from: http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/598