Research Impact

 

Impact beyond Academia

Impact beyond academia, or societal impact, on the other hand, focus on “broader impacts” brought by research, e.g., benefits for the economy, society, environment or culture, beyond the contribution to academic research.

In RAE (Research Assessment Exercise) 2026 in Hong Kong, impact is defined as:

The demonstrable contributions, beneficial effects, valuable changes or advantages that research qualitatively brings to the economy, society, culture, public policy or services, health, the environment or quality of life whether locally, regionally or internationally; and that are beyond academia. Impact in this context includes, but is not limited to –
(a) positive effects on, constructive changes or benefits to the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding, of an audience, beneficiary, community, constituency, organisation or individuals; or
(b) the reduction or prevention of harm, risk, cost or other negative effects.

To find support for impact beyond academia at the University of Hong Kong (HKU), refer to the Knowledge Exchange Office.

To discover impact beyond academia in Hong Kong, check RAE 2020 Impact Case Studies.

Reference

• University Grants Committee. (2024). Research Assessment Exercise 2026 Guidance Notes. https://www.ugc.edu.hk/eng/ugc/activity/research/rae/2026/guidance_notes.html
• Wilsdon, J. (2015). The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management. SAGE Publications. https://doi.org/10.4135/9781473978782

Planning and Tracking Impact

Identifying the impact of your research and gathering evidence can be challenging and time-consuming, especially if attempted long after the research is completed. It would likely be more effective and efficient to tackle this task by planning in an early stage and monitoring impact in real time. 

Planning

Creating an impact plan for research involves several steps to ensure that the work benefits the intended audiences and achieves meaningful outcomes.

1. Identify Beneficiaries
   
   Define who you want to benefit from your research. Conduct a stakeholder analysis to identify groups that might be interested - This helps pinpoint potential benefits and understand why these groups are interested in your research, converting interest into tangible benefits.
    
2. Set Impact Goals
   
   Establish clear impact goals based on the benefits identified in your stakeholder analysis. These goals might be basic initially but can be refined as you develop your plan.
    
3. Plan Activities for Engagement
   
   Determine the activities necessary to engage each identified group. Tailor these activities to the specific interests and characteristics of different sub-groups, ensuring they are relevant and effective.
    
4. Identify Indicators for Success
   
   Develop indicators to measure whether your activities are effective and if you're achieving your impact goals. Identify milestones on the pathway to impact and how you'll know when you've reached your goals. Consider both quantitative and qualitative means of measurement, and determine if additional resources or baseline data are required.
    
5. Refine Impact Goals
   
   Revisit your impact goals to make them more specific and measurable based on the indicators you’ve developed. This refinement enhances the credibility and competitiveness of your goals, especially in funding applications.
    
6. Anticipate Risks
   
   Think about potential risks and barriers to achieving your impact, such as lack of engagement or unintended negative consequences. Plan strategies to mitigate these risks to ensure your activities remain on track.
    
7. Prioritise Goals and Activities
   
   Stand back and assess which impact goals to pursue and which activities to prioritise. Focus on goals that inspire you and activities that are feasible within your available resources and time.
    

 

Tracking

In assessing research impact, two commonly used criteria are significance and reach.

• Significance
  
  The magnitude or intensity of the effect that research has on individuals, groups, or organizations. This involves evaluating how deeply the research influences or changes practices, policies, or understanding within the affected entities.
   
• Reach
  
  The number, extent, or diversity of individuals, groups, or organizations that benefit from the research. It measures how widely the impact is felt across different demographics or sectors.
   

Selecting indicators

Researchers use both qualitative and quantitative methods to depict cause-and-effect relationships. Impact indicators should be:

• Specific (capture the essence of the desired result and able to pick up changes over the time), 
• Measurable in either quantitative or qualitative terms; 
• Achievable (feasible in terms of equipment, funding, competences and time), 
• Relevant (capture what is to be measured accurately and consistently), and 
• Timely (able to provide information in a timely manner).

Some examples of indicators are:

• Numbers of companies, employment or new roles in the workforce
• Numbers of (or profits from) new commercial products or spin-out companies
• Improvements in indicators of social cohesion or social mobility, within a defined perimeter/community
• Time, money, ecosystem variables, or lives saved as a result of new evidence-based practices
• Increase and strengthening of the number of nodes or connections in a social network following a participatory process
• Testimonials or statements from end users (e.g. policy makers) now applying a modelling tool
• Testimonials from practitioners explaining how they gained a higher level of capability and capacity handling daily work thanks to a new guidance (improved skills, understanding, and confidence levels)
• Improvements in variables that indicate the achievement of goals set by a stakeholder or other social group who co-produced research (e.g. number of community members having acquired a particular skill)
• Changes of perception, awareness or attitudes of a social group as a result of engaging with research
• Changes in culture, cultural discourse or appreciation and benefit from cultural artifacts and experiences

Writing a narrative

A narrative of your impact describes the effects of your research. For example, in the UK Research Excellence Framework 2014, impact was assessed through case studies. Research found that high-scoring case studies:

• Established links between research (cause) and impact (effect) by writing in a clear, direct, and coherent way, often included attributional phrases, and used fewer ambiguous or uncertain phrases;
• Provided specific articulations and evidenced high-magnitude of significance (e.g. specifying “the government’s” or “to the House of Commons”) and reach (e.g. specifying “in England” or “in the US”), instead of focusing on pathway to impact;
• Were significantly easier to read and were maintained at interested and educated non-specialist level;
• Focused more on descriptions of research findings or the quality of research, rather than research outputs and processes;
• Used adjectives more appropriately, as inappropriate use of adjectives might lead to an over-claiming or less factual impression.

Scroll down for some useful tools which you may use to discover and demonstrate your success.

Reference

• Reed, M. (2019). Fast Track Impact Planning Template. Fast Track Impact. https://www.fasttrackimpact.com/post/2019/03/18/research-impact-planning

• Reed, M. S., Ferré, M., Martin-Ortega, J., Blanche, R., Lawford-Rolfe, R., Dallimer, M., & Holden, J. (2021). Evaluating impact from research: A methodological framework. Research Policy, 50(4), 104147. https://doi.org/10.1016/j.respol.2020.104147

• Reichard, B., Reed, M. S., Chubb, J., Hall, G., Jowett, L., Peart, A., & Whittle, A. (2020). Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014. Palgrave Communications, 6(1), 31. https://doi.org/10.1057/s41599-020-0394-7

   

Altmetrics

Altmetrics (Alternative metrics) attempts to capture the amount of attention a research output has received in non-academic outlets. It has gained attention as online platforms such as social media, online reference managers, scholarly blogs, and online repositories are deeply embedded into the system of scholarly communication. Different types of altmetric scores, which can be calculated for articles, books, data sets, presentations, and more, can be obtained from a range of commercial providers.

Use and limitations

• Details of the original mentions contributing to the altmetric scores can be useful for a broader examination of research contributions, e.g., highlighting citing policy documents to demonstrate social impact, and highlighting citing patents to demonstrate innovative impact. It may be useful for drafting an impact case study, which include specific, high-magnitude and well-evidenced articulations of significance and reach arising from research beyond academia.

• Often presented as a composite score, it represents a weighted measure of all the attention picked up for a research output (i.e., not a raw total of the number of mentions).
• Some of the activities included, especially those associated with social media, can be prone to being gamed.
• It provides little context for the type and purposes of engagement and thus, difficult to interpret in terms of broader research impact.
• Older papers published in the past may be under-represented, as social media is relatively new.

Access

Altmetric Explorer

1. Conduct a search with title, keywords, or DOI within the full Altmetric database
   
    
2. Identify the desired research output for its Altmetric details page
3. View the Attention Score and the sources, e.g., social media, news, policies, and patents
   

Reference

• Declaration on Research Assessment (DORA). (2024). Guidance on the responsible use of quantitative indicators in research assessment. DORA. https://doi.org/10.5281/zenodo.10979644
• Mingers , J., & Leydesdorff , L. (2015). A review of theory and practice in scientometrics . European Journal of Operational Research, 246(1), 1 19. https://doi.org/10.1016/j.ejor.2015.04.002
• Reichard, B., Reed, M. S., Chubb, J., Hall, G., Jowett, L., Peart, A., & Whittle, A. (2020). Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014. Palgrave Communications, 6(1), 31. https://doi.org/10.1057/s41599-020-0394-7

   

Policy Metrics

Policy data might be used to link research published in scholarly outputs to their use in a policy setting environment, to showcase the impact of research on policy, law and regulation. For example, in Research Assessment Exercise (RAE) 2026 in Hong Kong, "impact" is defined as the demonstrable contributions, beneficial effects, valuable changes or advantages that research qualitatively brings to the economy, society, culture, public policy or services, health, the environment or quality of life; and that are beyond the academia.

Some researchers would quote policies to demonstrate impact, for example, 

• Hong Kong University’s Review of Animal Welfare Legislation in Hong Kong, 
• Development and evaluation of a new service delivery model leads to policy change in preschool education and 
• Preventing suicide in Hong Kong: a multidisciplinary approach.

Use and limitations

• Policy citations could help researchers identify social benefits their research has brought about — the contribution of the research to the social capital of a nation (e.g., stimulating new approaches to social issues, informed public debate, and improved policy making).

• "Policy documents" is not well defined (i.e., what constitutes a policy document?).
• The way to obtain information on missing sources requires significant manual effort.
• To establish a claim on research translated into societal impact, researcher need to tackle problems including: 
  • Causality: What impact can be attributed to what cause?
  • Attribution: As impact can be diffuse, complex, and contingent, what portion of impact should be attributed to a certain research or to other inputs?

Access

Popular platforms include:

Reference

• Bornmann, L. (2013). What is societal impact of research and how can it be assessed? a literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217-233. https://doi.org/10.1002/asi.22803 

• Overton: A bibliometric database of policy document citations. Quantitative Science Studies, 3(3), 624-650. https://doi.org/10.1162/qss_a_00204 

• University Grants Committee. (2023). Research Assessment Exercise (RAE) 2026 Framework. https://www.ugc.edu.hk/doc/eng/ugc/rae/2026/framework.pdf

Patent metrics

A patent is a legal documents which a government grant to an inventor of the right to exclude others from making, using, or selling an invention, usually for a limited period.

Patent metrics include: 

• number of patents owned by researchers
• number of patent documents citing research publications

Patents might be considered a piece of evidence of translation of research into industry, innovation, and technical change. Some researchers would quote patents to demonstrate impact beyond academia, for example, 

• Optimizing devices performance through defect engineering, 
• Next-Generation Bone Implant Technology for Elderly Patients with Osteoporosis and 
• Commercialization of biomarkers for prediction and therapeutic intervention of cardiometabolic diseases: from bench to bedsides and beyond.

Use and limitations

• Patent metrics are sometimes used as evidence of business-related relevance of a piece of research, to demonstrate the impact on economy, technology, and commercialization. They are used when tracking technology transfer or industrial R&D (Research and Development) links.
• Evaluating citations in patents has the advantages that 
  • societal impact can be measured in a similar way to scientific impact;
  • the fact that they are citations means that nonreactive, relatively objective, and extensive data are available; and 
  • they are available in a relatively freely accessible form and can be evaluated with a reasonable amount of effort.

• Citation data are generally skewed. A variety of factors influence citation counts and should be considered in any evaluation.
• Patent citations do not capture knowledge transfers from informal or private collaborations between academics and commercial firms, which are not documented. 
• Citations may underrepresent foundational scientific discoveries, which are less likely to result in patents compared to applied research.
• Patent metrics are just indicators. To establish a claim on research translated into societal impact, researcher need to tackle problems including: 
  • Causality: What impact can be attributed to what cause?
  • Attribution: As impact can be diffuse, complex, and contingent, what portion of impact should be attributed to a certain research or to other inputs?

Access

Popular platforms include:

Reference

• Patent. In Encyclopædia Britannica. Retrieved 4 March 2025, from https://academic.eb.com/levels/collegiate/article/patent/58705

• Bornmann, L. (2013). What is societal impact of research and how can it be assessed? a literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217-233. https://doi.org/10.1002/asi.22803 

• Hammarfelt, B. (2021). Linking science to technology: the “patent paper citation” and the rise of patentometrics in the 1980s. Journal of Documentation, 77(6), 1413-1429. https://doi.org/10.1108/JD-12-2020-0218 

• Roach, M., & Cohen, W. M. (2012). Lens or Prism? Patent Citations as a Measure of Knowledge Flows from Public Research. Management Science, 59(2), 504-525. https://doi.org/10.1287/mnsc.1120.1644 

• Szomszor, M., & Adie, E. (2022). Overton: A bibliometric database of policy document citations. Quantitative Science Studies, 3(3), 624-650. https://doi.org/10.1162/qss_a_00204