¶¶ÒõÈÈÃÅ

Re: Request for Information (RFI) on Measuring and Rewarding Scientific Impact (NOT-OD-26-087)

Dear Dr. Jorgensen,

Thank you for the opportunity to respond to the Request for Information (RFI) on Measuring and Rewarding Scientific Impact. ¶¶ÒõÈÈÃÅ is one of the oldest and largest life science societies with over 38,000 members in the U.S. and around the world who are dedicated to conducting robust, rigorous research. 

¶¶ÒõÈÈÃÅ appreciates NIH’s interest in promoting scientific rigor and developing alternative measures for scientific success, beyond traditional measures like publication and citation counts. ¶¶ÒõÈÈÃÅ integrates scientific rigor and reproducibility across our programs and our portfolio of 17 peer reviewed journals. Please see below for ¶¶ÒõÈÈÃÅ’s comments on rigor and reproducibility, data, software and model sharing, training and mentorship, as well as entrepreneurship and translation categories described in the RFI.

Rigor and Reproducibility

Reproducibility is a cornerstone of the scientific process. However, the scientific community has become increasingly aware that some published scientific findings cannot be reproduced. We agree that steps should be taken to address this issue.

One way to measure reproducibility and impact is tracking the frequency with which publications are supported or contradicted by subsequent research, using citation statements classified as supporting, contrasting or merely mentioning prior work. Unlike traditional citation counts, such measures can provide insight into whether findings are being independently validated by the research community. Journal- or researcher-level indicators based on evidence of replication, support and reproducibility could complement existing metrics and help incentivize more robust and reliable science. 

Another potential indicator of scientific quality that is more robust than traditional citation counts is tracking where research is published. Major indexing and archiving services, such as Web of Science, Scopus, the Directory of Open Access Journals and PubMed Central, require journals to meet established standards related to editorial quality, transparency, ethics, technical rigor and accessibility. 

Incentives for researchers are needed to address reproducibility concerns. Replication studies are frequently overlooked because researchers rarely receive dedicated financial support to undertake research whose primary purpose is replication. Instead, the scientific community maintains scientific rigor through the peer review process, which ensures the quality and reliability of scholarly publications. However, robust peer review is time consuming work, done in addition to other research and service duties, disincentivizing the effort needed to thoroughly review articles prior to publication. 

¶¶ÒõÈÈÃÅ suggests the following indicators of rigor and reproducibility:

• Incorporate independent, external measures of research reliability and reproducibility into NIH’s measures of scientific impact, such as the .
• Track how often individual researchers publish in journals indexed by major indexing and archiving services with established standards.

¶¶ÒõÈÈÃÅ suggests the following incentives to promote rigor and reproducibility:

• Create funding streams for replication studies that supplement and do not supplant existing funding streams for original research.
• Recognize and reward researchers’ participation in the peer review process by measuring peer review as part of the researcher’s overall scientific impact.

Data, Software and Model Sharing

Scientific advancement occurs through continual expansion of knowledge, which requires access to accurate and complete data for confirmation of findings and building on previously published knowledge. While sharing data, datasets and software continues to improve, some data or code lack the necessary information and metadata to make them readily usable by other researchers when they are shared. Measuring the quality of the shared data or software, in addition to the frequency of use, and rewarding groups that consistently publish usable, high-quality data and software can begin to change the open-access scientific data and software landscape. 

¶¶ÒõÈÈÃÅ suggests the following indicator to promote data, software and model sharing:

• Implementation of a sharing index that measures how often researchers’ data and software is used and cited, when it is shared and the completeness and FAIR (Findable, Accessible, Interoperable, Reusable) compliance of their data.

¶¶ÒõÈÈÃÅ supports that agencies provide funding incentives to promote data, software and model sharing for labs that produce and share high-quality, well-annotated public data and datasets that become community resources. 

Training and Mentorship

Mentorship of trainees is a cornerstone of the scientific enterprise, yet it is often not evaluated in grant applications and traditional measures of scientific impact and scientific training. ¶¶ÒõÈÈÃÅ suggests agencies provide funding for the training of mentors in leadership, mentoring and managing people.

Entrepreneurship and Translation

Although moving basic science discoveries into clinical and field settings is a key goal of translational research, the translation of science extends beyond the development and commercialization of a finished product. Metrics of translational impact should therefore capture the full range of pathways through which research informs practice, policy, technology and future innovation. Even if the translation fails, capturing the attempt to translate, provided the science was rigorous, can provide indicators of researchers looking to move their science beyond the lab and into practice. This, paired with ongoing funding for programs like the Small Business Innovation Research and Small Business Technology Transfer (SBIR/STTR) programs, can help to encourage researchers to dedicate more of their efforts to exploring translation of their findings for the public good. 

¶¶ÒõÈÈÃÅ suggests the following indicator to measure entrepreneurship and translation:

• Track activities related to the commercialization of research, such as the issuance of patents, investigational new drug applications and validated biomarkers and manuscript reads that may be linked to commercial interest but not result in citations linked to research interest. 

¶¶ÒõÈÈÃÅ suggests the following incentive to promote entrepreneurship and translation:

• Continue robust funding for SBIR/STTR programs and other programs to promote entrepreneurship among researchers, such as the iCorps program.

Conclusion

Thank you for the opportunity to respond to this Request for Information. ¶¶ÒõÈÈÃÅ works to ensure scientific rigor across our programs and looks forward to working with NIH to incentivize and reward high-quality science. If you have any questions or would like to further discuss these comments, please contact Nicole Zimmerman, Senior Federal Affairs Officer, at nzimmerman@asmusa.org.

Thank you,

Amalia Corby
Director of Federal Affairs
¶¶ÒõÈÈÃÅ