Sci-Score, a tool suite to support Rigor and Transparency NIH and Journal Guidelines

ProjectSummary While standards in reporting of scientific methods are absolutely critical to producing reproducible sciencemeeting such standards is tedious and difficultChecklists and instructions are tough to follow often resulting in low and inconsistent complianceScientific journals and societies as well as the National Institutes of Health are now actively proposing general guidelines to address reproducibility issuesparticularly in the reporting of methodsbutthetrickierpartistotrainthebiomedicalcommunitytousethesestandardstoeffectivelyTo support new standards in methods reportingespecially the RRID standard for Rigor and Transparency of Key Biological Resourceswe propose to build Sci Score a text mining based tool suite to help authors meet the standardSci Score will provide an automated check on compliance with the RRID standard already implemented by overjournals including CellJournal of Neuroscienceand eLife and other Rigor and transparency standards put forward by the NIHThe innovation behind Sci score is the provision of a scorewhich can be obtained by individual investigators or journalsThis score reflects an aspect of quality of methods reportingWe posit that the score will serve as a tool that investigators can use to compete with themselvesandeachotherthewaytheycurrentlycompeteonmetricsofpopularityi etheH indexIn Phase I of this project and beforeour group has successfully developed a text mining algorithm that can detect antibodiescell linesorganisms and digital resourcesallRRID typesand has created a preliminary scoreWe propose to extend this approach to all research inputslike chemicals and plasmids that are requested as part of Cell pressSTAR Methodshttpwww cell com star methodsWe also propose to build a set of algorithms to detect whether authors discuss the major sources of irreproducibility outlined by NIHincluding investigator blindingproper randomization and sufficient reporting of sex and other biological variablesResource identification along with other quality metrics will be used to score the quality of scientific methods section textIf successfulthe tool could be used by editorsreviewersand investigators to improve thequalityofthescientificpaperOur Phase II specific aims includeenhancing and hardening the core natural language processing pipelines to recognize a broader range of sentences in near real timebuilding a set of modular tools that will be provided for different groups of users to take advantage of the text mining capability we develop in aimAt the end of Phase IIwe should have a commercially viable product that will be able to be licensed to serve the needsofthepublishersandthebroaderresearchcommunity Standardsforscientificmethodsreportingareabsolutelycriticaltoproducingreproduciblesciencebutmeeting suchstandardsisdifficultChecklistsandinstructionsaretoughtofollowoftenresultinginlowandinconsistent complianceTosupportnewstandardsinmethodsreportingespeciallytheRRIDstandardforRigorand TransparencyweproposetobuildSci ScoreatextminingbasedtoolsuitetohelpauthorsmeetthestandardSci ScorewillprovideanautomatedcheckoncompliancewiththeRRIDstandardimplementedbyoverjournalsincludingCellJournalofNeuroscienceandeLifeSci scoreprovidesanautomatedratingthequality ofmethodsreportinginsubmittedarticleswhichprovidesfeedbacktoauthorsreviewersandeditorsonhow toimprovecompliancewithRRIDsandotherstandards