Please note that the opinions, conclusions, and recommendations expressed by the authors of ASPCS papers and volumes do not necessarily reflect the opinions of the ASP, its Board, or its funders.
Summary of key conclusions:
1. The unique contribution of this book is to calibrate the interpretation of metrics derived from the SAO/NASA Astrophysics Data System using mean judgments provided by 22 experienced astronomers about the career impact of 510 faculty researchers at 17 highly-ranked universities world-wide.
2. It is more important to write high-impact papers than to write many papers.
3. Paper counts tell us little about research impact. If the aim is to rank candidates for a resource such as a job, then counting papers is not a good strategy.
4. Citation counts are the most reliable impact indicator for non-big-team people. If candidates who work substantially in teams of > 30 people are included in a search, then the most reliable metrics are normalized citation counts.
5. Eight metrics are calibrated: citations of all papers, citations of refereed papers, citations normalized by number of authors, tori index=twice-normalized citations, first-author citations, first-author citations received in 2013 – 2017, I100 index, and paper reads. Two metrics are analyzed with two different techniques each to demonstrate that conclusions are robust. Different combinations of metrics work best for different cohorts.
6. Instrumentalists are measured essentially as well as people in other fields, if metrics are chosen carefully.
7. “Research impact quotient” riq is an age-independent measure of research quality. It is a useful complement to career-integrated metrics such as citation counts.
8. Hirsch (2005) index H correlates so well with the square root of citation numbers that, for most people, it provides no additional information. However, outliers to this correlation who have unusually small H for their citation numbers are people who have made outstandingly important contributions (e. g., Nobel Prize winners).
9. Metrics measured early in a career (after 10 years of post-PhD ramp-up) can be used to predict impact later in that career. This is shown for three metrics, citations of refereed papers, normalized citations, and citations of first-author papers.
10. All metrics are probabilistic in the sense that a small fraction of people are not well measured. Correlations of impact with individual metrics have scatter that is quantified here, plus almost every correlation has a few major outliers. Therefore:
11. Metric combinations measure impact more accurately than does any single metric.
12. Standards for tenure-stream jobs are shown in Chapters 7.3 and 7.4 (publications) and Chapter 10 (citations). Chapter 14 summarizes advice for young scientists.