3 things we’ve learned from the Wellcome Success Framework so far

Wellcome was founded on the idea that supporting biomedical research, medical humanities and engaging the public with science would help to improve people’s health in the long run. We can lay out a ‘theory of change’ that leads from those activities to improvements in health, and we can point to examples where that has happened in the past.

But in truth, for most of our 80-odd year history, we haven’t been able to fully test all of those assumptions, evaluate the overall success of our decisions, or substantiate anecdotal evidence of Wellcome’s impact on health.

That’s starting to change. Advances in data platforms and technologies are reaching a point where we can use them to answer difficult questions and gain a more informed understanding about the results of Wellcome’s work. However, step one was to develop the Wellcome Success Framework to bring together all of our work and our ambitions, clarifying what we aim to achieve, and how.

I’m delighted that we’ve now published the first report from the Wellcome Success Framework, which analyses funding outcomes for the five years from 2012 to 2017.

Today’s report is not a scorecard by which to judge Wellcome. Rather it’s somewhat of a baseline – can the framework help us determine whether we’re making good progress toward achieving our ambitions? What can we learn from analysing decisions taken in the recent past? And as our strategy continues to evolve this year, how can the framework inform our decisions in the future?

For us, this is a significant stepping stone on the way to a reliable, systematic way to evaluate the choices we’ve made and the impact they’re having. Here’s what we’ve learned so far.

1. Our data is extremely patchy

We were forced to accept what we suspected for some time now. We collect a lot of data: most of it is informative, though by no means all; some is of high quality, some less good; and there are still many gaps where we have little or no data at all.

The Wellcome Success Framework has helped us to identify the issues. We are now filling the gaps, and assessing existing and potential new data sources and tools. Our aim is that we will soon have a more complete set of reliable data to monitor progress towards each of our ambitions.

2. Measuring success is hard!

It’s relatively easy to measure what Wellcome puts in to our various activities – how much money has been given out in grants, for example. It is harder to track the outcomes of those activities – did the research we funded lead to a breakthrough in knowledge, or new innovations for medical applications? And it is on another level again to measure the impact of our activities on improving human health.

And, partly in response to what the Wellcome Success Framework has already revealed, Wellcome’s strategy is evolving – which may, in turn, mean the Wellcome Success Framework has to adapt again to be of most use in the future.

3. Transparency and openness take courage

Wellcome is not alone in struggling to fully assess the impact of our activities. Far from it. And yet, while we very much want to share our progress, it has taken a few deep breaths to publish our first report, using real data where we have it, and admitting where we have less.

It is our responsibility, however, as a charitable foundation accountable to the society we work in, to be open – not only about our goals, but also about our progress. I hope this report will prompt others to think about our approach and whether there are any other tools or methods we should be consider.

Most of all, I hope this work inspires others to ask the hard questions, and explore new and innovative ways to evaluate their success at an organisation-wide scale. I’m sure we have much to learn from each other.