Research Metrics

Article Level Metrics

 

Article level metrics are often used to indicate the impact of scholarly journal articles. The most common metrics base on citations, such as times cited and other field normalized metrics. Again, the metrics are only available for the articles indexed in the specific citation databases.

Recently, altmetrics, which captures the immediate response to the publications in online environments (e.g., social media), are gaining increasing attention. It may be useful for drafting an impact case study, which articulates significance and reach arising from research beyond academia.

Times Cited

The citation count of an article (times cited) is the number of times it is included in the reference list of other articles or books. The numbers are available for only the articles indexed in and specific to the citation databases used.

Use and limitations

• As compared with journal-based indicators (e.g., journal impact factor), it can offer information at the relevant level of granularity, i.e., the individual research article.

• There is no indication on whether articles are cited for positive or negative reasons.
• Citation performance is a lagging indicator that takes years to turn into a robust signal (not useful to evaluate recent scholarship).
• It is not well suited to compare researchers at different career stages or in different disciplines.
• Citation patterns can be skewed by author and journal reputations.

Access

Popular platforms include:

• Scopus
  
  
  
  
• Web of Science
  
  

Reference

• Declaration on Research Assessment (DORA). (2024). Guidance on the responsible use of quantitative indicators in research assessment. DORA. https://doi.org/10.5281/zenodo.10979644

Field-normalized citation indicators

Field-normalized citation indicators such as Category Normalized Citation Impact (CNCI) or Field Weighted Citation Impact (FWCI) represent attempts to correct for the citation variability arising from differences between fields, types, and ages of publications. In general:

• A value of 1 is as expected for the world average.
• Values above 1 are above average.
• Values below 1 are below average.

The numbers are available for only the articles indexed in and specific to the citation databases used.

Use and limitations

• They include fields, types, and ages of publications in the calculation.

• It can be difficult to define which papers belong in which fields.
• For datasets comprising only tens or hundreds of papers, such indicators are less reliable because of the impact of highly cited outliers.
• They are mean-based indicators, which cannot represent the often highly skewed distribution of citations across publications.

Access

Popular platforms include:

Reference

• Declaration on Research Assessment (DORA). (2024). Guidance on the responsible use of quantitative indicators in research assessment. DORA. https://doi.org/10.5281/zenodo.10979644

Citations in Context

Some tools, such as "Smart Citations" by Scite and "Citing items by classification" feature in Web of Science, help to reveal how a scientific paper has been cited by providing the context of the citation and a classification system describing whether it provides supporting or contrasting evidence for the cited claim, or if it just mentions it.

Use and limitations

• It adds contextual information on how the citing paper used the citation (e.g., background, basis, or discussion; Or supporting, mentioning or contrasting).

• There are limitations of the machine learning model precision.
• The coverage of articles analyzed by scite may be considered limited. Open access repositories and a variety of open sources as identified by Unpaywall, other relevant open access document sources are utilized. Subscription articles used were available through indexing agreements with over a dozen publishers.

Access

Popular platforms include:

References

• Nicholson, J. M., Mordaunt, M., Lopez, P., Uppala, A., Rosati, D., Rodrigues, N. P., Grabitz, P., & Rife, S. C. (2021). scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. Quantitative Science Studies, 2(3), 882-898. https://doi.org/10.1162/qss_a_00146

• Nicholson, J. M., Uppala, A., Sieber, M., Grabitz, P., Mordaunt, M., & Rife, S. C. (2021). Measuring the quality of scientific references in Wikipedia: an analysis of more than 115M citations to over 800 000 scientific articles. Febs j, 288(14), 4242-4248. https://doi.org/10.1111/febs.15608

   

Altmetrics

Altmetrics (Alternative metrics) attempts to capture the amount of attention a research output has received in non-academic outlets. It has gained attention as online platforms such as social media, online reference managers, scholarly blogs, and online repositories are deeply embedded into the system of scholarly communication. Different types of altmetric scores, which can be calculated for articles, books, data sets, presentations, and more, can be obtained from a range of commercial providers.

Use and limitations

• Details of the original mentions contributing to the altmetric scores can be useful for a broader examination of research contributions, e.g., highlighting citing policy documents to demonstrate social impact, and highlighting citing patents to demonstrate innovative impact. It may be useful for drafting an impact case study, which include specific, high-magnitude and well-evidenced articulations of significance and reach arising from research beyond academia.

• Often presented as a composite score, it represents a weighted measure of all the attention picked up for a research output (i.e., not a raw total of the number of mentions).
• Some of the activities included, especially those associated with social media, can be prone to being gamed.
• It provides little context for the type and purposes of engagement and thus, difficult to interpret in terms of broader research impact.
• Older papers published in the past may be under-represented, as social media is relatively new.

Access

Altmetric Explorer

1. Conduct a search with title, keywords, or DOI within the full Altmetric database
   
    
2. Identify the desired research output for its Altmetric details page
3. View the Attention Score and the sources, e.g., social media, news, policies, and patents
   

Reference

• Declaration on Research Assessment (DORA). (2024). Guidance on the responsible use of quantitative indicators in research assessment. DORA. https://doi.org/10.5281/zenodo.10979644
• Mingers , J., & Leydesdorff , L. (2015). A review of theory and practice in scientometrics . European Journal of Operational Research, 246(1), 1 19. https://doi.org/10.1016/j.ejor.2015.04.002
• Reichard, B., Reed, M. S., Chubb, J., Hall, G., Jowett, L., Peart, A., & Whittle, A. (2020). Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014. Palgrave Communications, 6(1), 31. https://doi.org/10.1057/s41599-020-0394-7

   

Policy Metrics

Policy data might be used to link research published in scholarly outputs to their use in a policy setting environment, to showcase the impact of research on policy, law and regulation. For example, in Research Assessment Exercise (RAE) 2026 in Hong Kong, "impact" is defined as the demonstrable contributions, beneficial effects, valuable changes or advantages that research qualitatively brings to the economy, society, culture, public policy or services, health, the environment or quality of life; and that are beyond the academia.

Some researchers would quote policies to demonstrate impact, for example, 

• Hong Kong University’s Review of Animal Welfare Legislation in Hong Kong, 
• Development and evaluation of a new service delivery model leads to policy change in preschool education and 
• Preventing suicide in Hong Kong: a multidisciplinary approach.

Use and limitations

• Policy citations could help researchers identify social benefits their research has brought about — the contribution of the research to the social capital of a nation (e.g., stimulating new approaches to social issues, informed public debate, and improved policy making).

• "Policy documents" is not well defined (i.e., what constitutes a policy document?).
• The way to obtain information on missing sources requires significant manual effort.
• To establish a claim on research translated into societal impact, researcher need to tackle problems including: 
  • Causality: What impact can be attributed to what cause?
  • Attribution: As impact can be diffuse, complex, and contingent, what portion of impact should be attributed to a certain research or to other inputs?

Access

Popular platforms include:

Reference

• Bornmann, L. (2013). What is societal impact of research and how can it be assessed? a literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217-233. https://doi.org/10.1002/asi.22803 

• Overton: A bibliometric database of policy document citations. Quantitative Science Studies, 3(3), 624-650. https://doi.org/10.1162/qss_a_00204 

• University Grants Committee. (2023). Research Assessment Exercise (RAE) 2026 Framework. https://www.ugc.edu.hk/doc/eng/ugc/rae/2026/framework.pdf

Patent metrics

A patent is a legal documents which a government grant to an inventor of the right to exclude others from making, using, or selling an invention, usually for a limited period.

Patent metrics include: 

• number of patents owned by researchers
• number of patent documents citing research publications

Patents might be considered a piece of evidence of translation of research into industry, innovation, and technical change. Some researchers would quote patents to demonstrate impact beyond academia, for example, 

• Optimizing devices performance through defect engineering, 
• Next-Generation Bone Implant Technology for Elderly Patients with Osteoporosis and 
• Commercialization of biomarkers for prediction and therapeutic intervention of cardiometabolic diseases: from bench to bedsides and beyond.

Use and limitations

• Patent metrics are sometimes used as evidence of business-related relevance of a piece of research, to demonstrate the impact on economy, technology, and commercialization. They are used when tracking technology transfer or industrial R&D links.
• Evaluating citations in patents has the advantages that 
  • societal impact can be measured in a similar way to scientific impact;
  • the fact that they are citations means that nonreactive, relatively objective, and extensive data are available; and 
  • they are available in a relatively freely accessible form and can be evaluated with a reasonable amount of effort.

• Citation data are generally skewed. A variety of factors influence citation counts and should be considered in any evaluation.
• Patent citations do not capture knowledge transfers from informal or private collaborations between academics and commercial firms, which are not documented. 
• Citations may underrepresent foundational scientific discoveries, which are less likely to result in patents compared to applied research.
• Patent metrics are just indicators. To establish a claim on research translated into societal impact, researcher need to tackle problems including: 
  • Causality: What impact can be attributed to what cause?
  • Attribution: As impact can be diffuse, complex, and contingent, what portion of impact should be attributed to a certain research or to other inputs?

Access

Popular platforms include:

Reference

• Patent. In Encyclopædia Britannica. Retrieved 4 March 2025, from https://academic.eb.com/levels/collegiate/article/patent/58705

• Bornmann, L. (2013). What is societal impact of research and how can it be assessed? a literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217-233. https://doi.org/10.1002/asi.22803 

• Hammarfelt, B. (2021). Linking science to technology: the “patent paper citation” and the rise of patentometrics in the 1980s. Journal of Documentation, 77(6), 1413-1429. https://doi.org/10.1108/JD-12-2020-0218 

• Roach, M., & Cohen, W. M. (2012). Lens or Prism? Patent Citations as a Measure of Knowledge Flows from Public Research. Management Science, 59(2), 504-525. https://doi.org/10.1287/mnsc.1120.1644 

• Szomszor, M., & Adie, E. (2022). Overton: A bibliometric database of policy document citations. Quantitative Science Studies, 3(3), 624-650. https://doi.org/10.1162/qss_a_00204