Making sense of metrics …. by Tracey Pratchett

Hello from the Metrics Task and Finish Group.

In our last blog post, we provided an update on the results of our survey that you all kindly filled out for us. In this new post, we want to tell you about our progress for defining what a good metric looks like and what it should include.

On Tuesday 15th December the Task and Finish Group met to outline our definition for developing good metrics for NHS library, knowledge and information services. Thanks to Dorothy we had access to wide range of literature, which really helped to inform our discussions – who’d have thought there was so much to consider! It provided a really great basis for our group to discuss metrics and will be summarised in our final report.

To inform our discussion we focussed on two key resources which provided a basic framework for us to adapt for the project:

Fuelled by tea and biscuits and after some interesting debates about what metrics meant to each of our group we adapted Markgraf’s (2015) succinct definitions on the following characteristics for good metrics which should be, in summary:

  • Meaningful
  • Actionable
  • Reproducible
  • Comparable

In order to ensure that these criteria would be applicable for NHS library, knowledge and information service metrics, we tested them on some generic metrics from Knowledge for Healthcare and some Key Performance Indicators (KPIs) from HEE LKSL.

The testing highlighted some key challenges in terms of creating metrics which were meaningful to everyone, something which we discussed at length. It also highlighted issues around the distinction between a metric and a target.

What is your experience of using metrics, do you agree with Markgraf’s criteria, or do you have another take on them?

Please share your comments or feedback via this blog or contact the Task and Finish Group lead Alan, Fricker alan.fricker@kcl.ac.uk

5 thoughts on “Making sense of metrics …. by Tracey Pratchett

  1. A metric is useful if you use it in a meaningful way. There is a danger that we put too much focus on metrics that are easy to gather, rather than metrics that are more difficult to generate. Impact is important if we want to demonstrate to management the value of the service but I would argue that some of the “easy” metrics also serve a valuable function, if they are used in the right way.

    Some of the “easy” metrics can be used as early indicators (EWS for libraries!) that the service is doing something right (or wrong). For example counting book loans and comparing them to the same periods for previous years can give some insight to what is happening within your service and an opportunity to reflect. Have the loans gone down or up? is there a reason for this? Lower footfall? Wrong bookstock? Old bookstock? Are you promoting them enough? have there been changes to your book loans procedures that deter people from borrowing?

    The same applies to inter library loans or teaching/training. If there have been changes in activity levels (good or bad) is there is a reason for this?

    Another point is that although easy metrics tend to focus on activity rather than impact, activity is still an important indicator. If the service has seen a significant increase in activity and more staff are required, the easy metrics are indicators that can help justify this to management.

    Are some metrics more important than others? ……I don’t think there is a right or wrong answer, I think it depends on what you use it for, how you use it and what you interpret it for.

    1. Hi Hugh,

      One of the bits of good practice that showed up in the survey data we received was around people using segmentation to make their data more meaningful. So with your book loan examples – what patterns of use are there amongst professional groups?

      Metrics must be meaningful – to you and to your stakeholders.

      Cheers

      Alan

  2. For me your Service’s performance should be measured against the objectives of the organisations it serves. Developing metrics to compare one Service against another is meaningless to a degree as it highlights differences and similarities in throughputs only.

    Staff should be able to demonstrate that their personal targets link into the organisation’s aims and objectives and/or current business priorities. Scrap counting book loans as they are meaningless and focus upon collecting valuable impact data which demonstrates improving the quality of your organisation’s services and cost improvements. This is the real metrics which senior managers and executives are interested in and will save the axe falling towards our heads.

    If a Service is to survive it has to be able to demonstrate that it is supporting the organisation’s goals.

    1. Thanks Gary – very much agree with need to match to the objectives.

      Book loans do at least have the merit of being something people can easily understand but they are definitely not much of an impact marker.

Leave a Reply

Your email address will not be published. Required fields are marked *