Research Impact

Article Level Metrics

 

Article level metrics are often used to indicate the impact of scholarly journal articles. The most common metrics base on citations, such as times cited and other field normalized metrics. Again, the metrics are only available for the articles indexed in the specific citation databases.

Such indicators may be used for different purposes, such as:

• to assess research in performance-based research funding systems (e.g., Research Assessment Exercise in Hong Kong), and
• to inform decisions in promotion and tenure in faculties.

Recently, altmetrics, which captures the immediate response to the publications in online environments (e.g., social media), are gaining increasing attention. It may be useful for drafting an impact case study, which articulates significance and reach arising from research beyond academia.

Reference

• OECD. (2010). Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings. OECD Publishing. https://doi.org/10.1787/9789264094611-en

• Rice, D. B., Raffoul, H., Ioannidis, J. P. A., & Moher, D. (2020). Academic criteria for promotion and tenure in biomedical sciences faculties: Cross sectional analysis of international sample of universities. BMJ: British Medical Journal, 369. https://doi.org/10.1136/bmj.m2081

Times Cited

The citation count of an article (times cited) is the number of times it is included in the reference list of other articles or books. The numbers are available for only the articles indexed in and specific to the citation databases used.

Use and limitations

• As compared with journal-based indicators (e.g., journal impact factor), it can offer information at the relevant level of granularity, i.e., the individual research article.

• There is no indication on whether articles are cited for positive or negative reasons.
• Citation performance is a lagging indicator that takes years to turn into a robust signal (not useful to evaluate recent scholarship).
• It is not well suited to compare researchers at different career stages or in different disciplines.
• Citation patterns can be skewed by author and journal reputations.

Access

Popular platforms include:

• Scopus
  
  
  
  
• Web of Science
  
  

Other relevant metrics

A number of calculations (or awards and recognition) are based on citation counts. Here are some examples.

Reference

• Declaration on Research Assessment (DORA). (2024). Guidance on the responsible use of quantitative indicators in research assessment. DORA. https://doi.org/10.5281/zenodo.10979644

Field-normalized citation indicators

Field-normalized citation indicators such as Category Normalized Citation Impact (CNCI) or Field Weighted Citation Impact (FWCI) represent attempts to correct for the citation variability arising from differences between fields, types, and ages of publications. In general:

• A value of 1 is as expected for the world average.
• Values above 1 are above average.
• Values below 1 are below average.

The numbers are available for only the articles indexed in and specific to the citation databases used.

Use and limitations

• They include fields, types, and ages of publications in the calculation, and are more useful in terms of comparing research across different fields.

• It can be difficult to define which papers belong in which fields.
• For datasets comprising only tens or hundreds of papers, such indicators are less reliable because of the impact of highly cited outliers.
• They are mean-based indicators, which cannot represent the often highly skewed distribution of citations across publications.

Access

Popular platforms include:

Reference

• Declaration on Research Assessment (DORA). (2024). Guidance on the responsible use of quantitative indicators in research assessment. DORA. https://doi.org/10.5281/zenodo.10979644

Citations in Context

Some tools, such as "Smart Citations" by Scite and "Citing items by classification" feature in Web of Science, help to reveal how a scientific paper has been cited by providing the context of the citation and a classification system describing whether it provides supporting or contrasting evidence for the cited claim, or if it just mentions it.

Use and limitations

• It adds contextual information on how the citing paper used the citation (e.g., background, basis, or discussion; Or supporting, mentioning or contrasting).

• There are limitations of the machine learning model precision.
• The coverage of articles analyzed by scite may be considered limited. Open access repositories and a variety of open sources as identified by Unpaywall, other relevant open access document sources are utilized. Subscription articles used were available through indexing agreements with over a dozen publishers.

Access

Popular platforms include:

References

• Nicholson, J. M., Mordaunt, M., Lopez, P., Uppala, A., Rosati, D., Rodrigues, N. P., Grabitz, P., & Rife, S. C. (2021). scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. Quantitative Science Studies, 2(3), 882-898. https://doi.org/10.1162/qss_a_00146

• Nicholson, J. M., Uppala, A., Sieber, M., Grabitz, P., Mordaunt, M., & Rife, S. C. (2021). Measuring the quality of scientific references in Wikipedia: an analysis of more than 115M citations to over 800 000 scientific articles. Febs j, 288(14), 4242-4248. https://doi.org/10.1111/febs.15608