New algorithm and multi step process for achieving better global project rankings

Under this Small Business Innovation Research SBIR Phase I project Research Topic the contractor will investigate the effectiveness of new peer review methods Concordance Index based Global Ranking CIGR and Modified Borda Count MCB to enhance professional evaluation and decision making in life sciences management and administration The intended result is a novel set of metrics that can be used by NGOs disease foundations advocacy groups research funders policy makers and by academic institutional bodies This SBIR phase I project has the following objectives Simulation studies to investigate the effectiveness of the method used in the NSF pilot study and the contractor s new method CIGR The contractor will perform simulation studies to investigate the effectiveness of the new review method in terms of how it can recover the true ranking of proposals or papers The contractor will also study which parameter settings are optimal for the new review method Testing the new approach in real evaluation settingsThe contractor has implemented the method used in the NSF s pilot study as an online platform at http pevals com The contractor will implement our new ranking method CIGR in the phase I project and test the new peer review approach in graduate student course project essay evaluation at Florida State University In these tests the true rankings and scores of the project reports or essays will be obtained from the course instructor and used as the true rankings After collecting the ranking from individuals we will use both CIGR and the modified Borda Count MBC score proposed originally for the new peer review method and also used in the NSF s pilot study to obtain the final rankings and compare the accuracy of the two different ranking methods based on their agreement with the true rankings This test is unique in the sense that it has the true rankings obtained from the course instructors which will help us to evaluate the effectiveness of different ranking algorithms This test of using graduate student course projects will establish solid basis for our future tests on scientific papers and proposals The contractor will also test our method for a conference paper evaluation for a workshop in IEEE BIBM conference