Guidance for research organisations on how to implement responsible and fair approaches for research assessment

Our open access policy 2021 requires Wellcome-funded organisations to publicly commit to:

• assessing research outputs and other research contributions based on their intrinsic merit 
• discouraging the inappropriate use of proxies or metrics – such as the title or impact factor of the journal in which the work was published.

We believe that research assessment processes used by research organisations and funders in making recruitment, promotion and funding decisions should embody two core principles (‘the principles’) as set out in the San Francisco Declaration on Research Assessment (DORA):  

1. be explicit about the criteria used to evaluate scientific productivity, and clearly highlight that the scientific content of a paper is more important than publication metrics or the identity of the journal in which it is published
2. recognise the value of all relevant research outputs (for example publications, datasets and software), as well as other types of contributions, such as training early-career researchers and influencing policy and practice.

We recognise that many organisations will implement these core principles as part of a broader approach to promoting the responsible and fair use of metrics in line with other equivalent declarations, such as the Leiden Manifesto and the Hong Kong Principles.  

Becoming a signatory to DORA, and/or endorsing equivalent declarations is one way that organisations can demonstrate their commitment to implementing the principles, both to their staff and communities.  

However, there is no requirement for Wellcome-funded organisations to publicly endorse any declaration. We want organisations to implement responsible and fair approaches for research assessment in line with the two principles set out above, whether that is driven by commitments to one or more of these declarations or otherwise.  

As a minimum, by January 2021, Wellcome-funded research organisations should have a:

• statement of commitment to implementing the principles on their website – this should be clear and accessible
• plan for implementing the principles, or a clear process in place for developing a plan (with a specified delivery date) 
• process in place for monitoring and reporting on progress in implementing the principles.

As part of our regular audits, we may ask organisations to demonstrate that they have met these three high-level requirements.

The rest of this guidance is not intended to be prescriptive or exhaustive. We recognise that organisations will take different approaches to put these principles into practice, that reflect their own values, cultures and ways of working. We encourage organisations to consider other activities (that are not detailed below) where they feel they could add value in their own context, and to trial new ideas and approaches generated by their staff on an ongoing basis. 

We also recognise that many organisations have already publicly committed to DORA, or equivalent statements, and have established processes for their implementation. We do not expect organisations to establish new public statements, implementation plans or parallel activities where these are already in place.

Work to implement the principles should be joined-up with the organisation’s broader efforts to enhance research culture and promote research integrity.  

We strongly encourage organisations to be transparent and proactive in sharing their approaches and learnings, and to work together to drive change. Wellcome will work in partnership with organisations to develop and share good practice.

If you have developed approaches to promote fair and responsible assessment that could provide valuable learning and inspiration for other research organisations, please let us know and share with other organisations – such as DORA – who are collating good practice examples and case studies.

1. Statement of commitment

Organisations must have a clear and easily accessible statement on their externally-facing website, highlighting their commitment to implementing the principles as part of a fair and responsible approach for research assessment. This must be in place by January 2021.

The statement does not necessarily need to be published on a stand-alone webpage. For example, it could form part of an existing webpage which addresses the organisation’s broader approach to research evaluation, open research or enhancing research culture.  

Alongside this statement, organisations are encouraged to:

• specify how the principles are aligned to the organisation’s overarching vision, mission and values 
• be explicit on whether they:
  • have signed up to or endorsed DORA, and/or equivalent declarations
  • are actively considering signing up to one or more of these declarations
  • have decided not to sign up to a declaration but will adopt equivalent or enhanced principles and procedures.
• explain the key steps they are taking to implement and embed the principles, linking through to relevant policies, guidance and the implementation plan
• provide detailed materials, guidance and resources to staff through an internally-facing website as appropriate (while considering making these publicly available wherever possible as exemplars of good practice).

2. Implementation plan

Organisations must develop a plan for implementing the principles (where such a plan is not already in place) or have a clear process in place for developing a plan with a specified delivery date, by January 2021. This should incorporate elements from each of the subsections below.

Strategy and leadership

The development and delivery of the implementation plan should be resourced sufficiently and have clear ownership and buy-in across the organisation.  

Organisations should consider:

• making sure one or more senior leaders take ownership for championing the principles and their implementation, as part of broader efforts to enhance research culture
• establishing a cross-organisation working group with a suitably diverse and representative membership – including across research disciplines and career levels – to develop the implementation plan and oversee its delivery
• seeking input and feedback from staff across the organisation during the development and roll out of the implementation plan, where appropriate
• designating one or more staff members to take forward the agreed implementation plan as a key part of their role
• establishing a way for staff to report confidentially if the principles have been breached, and for appropriate remedial action to be taken by senior leaders (which may be through an existing whistleblowing process).

Hiring and promotion practices

Organisations should develop their recruitment, promotion and career-advancement policies and practices in ways that reflect their commitment to the principles.

This might include:

• making sure that the criteria used for recruitment promotion and other decisions on career advancement are clear and transparent, and specifically reference the principles
• developing clear guidance for staff involved in recruitment and promotion decisions which:
  • explicitly caution against the inappropriate use of publication metrics
  • encourage them to value a full and diverse range of research outputs and contributions.
• encouraging candidates to highlight a broad range of research outputs and other contributions, in addition to publications
• asking candidates to highlight a limited number of key achievements (which may be based on one or several outputs), and to provide a narrative description of their significance and the role they played in the research
• prohibiting the use of language in job advertisements which refers directly or indirectly to journal title as a proxy for quality (for example ‘a track record of publication in leading journals’)
• discouraging the inappropriate use of lists of ‘target journals’ for researchers that are based on perceived prestige associated with the journals concerned.

Communication, advocacy and raising awareness

Organisations should make sure that their staff are aware of their organisation’s policy and expectations in relation to the principles, and have the guidance and support needed to implement them.

This might include:

• highlighting ongoing work through the organisation’s communication channels – such as websites, newsletters and social media
• running seminars or workshops to engage staff across the organisation in discussions about the principles and responsible and fair research assessment more broadly, and the associated opportunities and challenges 
• incorporating information about the principles in staff inductions and in specific training for group leaders and others involved in recruiting and promotion
• identifying ‘champions’ across the organisation who can act as advocates of good practice and work with colleagues to identify and address any challenges and concerns.

3. Approach for monitoring and sharing learning

Organisations are required to have a process in place by January 2021 to monitor their progress in implementing the principles and ensure this progress is reviewed over time.

Wellcome will only require organisations to provide assurance that they have such a process in place. We will not require detailed reporting against specific metrics and we recognise that the ability to collect particular types of information will vary between organisations. As noted above, we will also explore how we can bring organisations together to help them develop effective monitoring frameworks and share good practice. 

Organisations should consider:

• gathering quantitative and qualitative information – for example, on hiring and promotion practices and on wider attitudes and practices among staff – to help monitor and report on progress against an initial baseline
• reviewing and adapting their implementation plan on a regular basis, in line with lessons learned and emerging good practice 
• sharing their experiences with other research organisations and the wider research community (for example through DORA and other bodies) – including both successful and unsuccessful approaches.

Since we introduced our open access policy in 2005, Wellcome has adopted a clear position that research publications should be assessed on their intrinsic merit, and not on the journal in which they are published. 

We were one of the first funders to sign the San Francisco Declaration on Research Assessment (DORA) when it was launched in 2013.

There is a clear and growing consensus over the need to reform research assessment practices to help improve research culture and integrity, which is reflected in the Hong Kong Principles, and the need to ensure the metrics that underpin research assessment practices are used in a responsible and appropriate way, as considered in the Metric Tide report.

Many funders, universities and other research organisations have either signed DORA, indicated their support for related declarations such as the Leiden Manifesto, or otherwise started work to put the principles into practice.

In 2019, the cOAlition S group, of which Wellcome is a member confirmed its commitment to assess research outputs based on their intrinsic merit by making it a core principle of Plan S.

To be successful in driving cultural change, funders and research organisations must work together to reform the assessment processes that underpin decisions relating to research funding, recruitment and career progression. Credible and trusted processes need to be put in place, through which researchers are genuinely judged on the value of their work and not on where they have published. Researchers must also be empowered to challenge situations where practices fail to reflect these standards. Many organisations have already made significant steps towards achieving these goals.

Our new open access policy, which comes into force in January 2021, requires the organisations we fund to publicly commit to the two core DORA principles, and that this may be assessed as part of our regular audits from this time. 

The cultural change required to embed the principles will take time, and we are committed to working with our funded organisations to drive this change.

• Open access policy
• Open research
• Research culture: let's reimagine how we work together

If you have any questions about this work, contact Dave Carr d.carr@wellcome.org.