Research Metrics

Non-Traditional Research Outputs (NTROs): 

• research outputs beyond the traditional research outputs (academic journal articles, conference papers, academic books, and book chapters), including but not limited to:
  • original creative works
  • live performance of creative works
  • recorded/rendered creative works
  • curated or produced substantial public exhibitions and events
  • research reports for an external body
  • portfolio
• represent a diverse spectrum of creative and practice-based works, reflecting the innovative nature of research, particularly within the humanities and social sciences

In Hong Kong, the RAE (Research Assessment Exercise) 2026 indicates the submission of NTROs cases as acceptable research outputs, with a definition:

Other forms of output that was published or made publicly accessible or effectively shared within the profession, e.g. performance recording, video tape, computer software programme, architectural drawings, or any creative work that can be evaluated for merit and an assessment obtained.

Reference

• Australian Research Council. (2020). State of Australian University Research 2018–19: ERA National Report. https://dataportal.arc.gov.au/era/nationalreport/2018/ 
• University Grants Committee. (2024). Research Assessment Exercise 2026 Guidance Notes. https://www.ugc.edu.hk/eng/ugc/activity/research/rae/2026/guidance_notes.html

 

Capturing the impact of NTROs can be challenging because NTROs, unlike traditional academic outputs, lack a systematic infrastructure for impact measurement.

Researchers can:

• Combine Tools - Multiple tools can be used for a more comprehensive set of measures.
• Provide Qualitative Analysis - Qualitative feedback, reviews, or measures should be provided to highlight the impact. 
• Write a Narrative - Metrics should be combined with a thoughtful narrative to present a clearer, more comprehensive picture of the NTRO’s influence. A compelling narrative contextualises the numbers by telling the story with:  
  • Relevance: The extent to how closely an NTRO aligns with real-world needs, current practices, or ongoing policy development.  
  • Innovation: The degree of originality and creativity in the NTRO, highlighting new developments and practical applications.  
  • Engagement: The level of interaction of the NTRO with the intended audience or community.  
  • Influence: The measurable effect of the NTRO on the field, community, or society.  
  • Longevity: The lasting impact for the NTRO.

Reference

• Deakin University Library. (2025). Research Metrics Toolkit. Retrieved 25 March 2025 from https://deakin.libguides.com/research-metrics/NTRO
• The University of Melbourne Library. (2025). Research impact for Education: A self-help guide. Retrieved 25 March 2025 from https://unimelb.libguides.com/FOE-research-impact/non-standard

The best practice is to keep a record of all the: 

• awards and prizes, 
• commissions or royalties, 
• book or box office sales, 
• grants, 
• reviews, 
• invitation to perform/exhibit, 
• media and news mentions, 
• audience or visitor numbers, 
• prestige of venues, and 
• esteem of performers.

In general,

• Search engines such as Google would be useful. 

Specifically, the following can be used to find evidences related to:

 

Books

• WorldCat is a universal library catalogue to find the libraries which hold the book in their collections.  

 

 

Open Educational Resources (OER) / Textbooks

• The Open Syllabus Project tracks OER and textbook listings in syllabi. 

 

 

Literary works of poetry, drama, and prose

• Literature Online - Criticism helps locating reviews, awards, and prizes.

 

Creative works, exhibitions and events, live performances

• Factiva provides international newspaper clippings, which may help finding reviews.
• WisersOne provides local Hong Kong newspaper clippings.

     

Metrics used for traditional research outputs might be useful as well:

Times Cited

• Citation count, the number of times it is included in the reference list of other articles or books.

 

Altmetrics (Alternative Metrics)

• The amount of attention a research output has received in non-academic outlets, such as: 
  • social media, 
  • online reference managers, 
  • scholarly blogs, and 
  • online repositories.

 

Policy Metrics

• Linking research published in scholarly outputs to their use in a policy setting environment, to showcase the impact of research on policy, law and regulation.

 

Patent Metrics

• For example, number of patents owned by researchers and number of patent documents citing research publications.