Reproducibility means many things to many people. This session will cover 1) various uses of ‘replication’ and ‘reproducibility’ in different communities 2) how Nature commissions and edits opinion articles on improving science and 3) how to pitch and write articles for a general scientific audience
Monya Baker Editor and writer at Nature, Nature Journal
Monya Baker commissions and edits opinion articles on improving science for Nature magazine, where she has worked since 2007. Her work has appeared in Nature, Science, Wired, The Economist, Slate, New Scientist and elsewhere. She has an Ed.M from Harvard University and a B.A. in biology from Carleton College.
The wisdom of crowds hinges on the independence and diversity of their members’ information and approach. Here I explore the wisdom of scientific crowds for discovery, showing how findings established by more distinct methods and researchers are much more likely to replicate, how a population of diverse small teams advances science more rapidly than large ones, and how diverse prior experience are critical for punctuated advance. Artificial Intelligence has typically been designed to substitute for human expertise rather than complement it, limiting its capacity for human benefit. I demonstrate how incorporating the distribution of human expertise into AI models allows us to design diversity that complements and corrects for collective human bias, generating “alien” hypotheses unlikely to be imagined or pursued without intervention in materials discovery, drug development and COVID-19 vaccines. I discuss how to design diverse human and artificial intelligence collectives that can expand our imaginations and reach past our limits together.
James Evans Professor, Department of Sociology, University of Chicago, University of Chicago
James Evans research focuses on the collective system of thinking and knowing, ranging from the distribution of attention and intuition, the origin of ideas and shared habits of reasoning to processes of agreement (and dispute), accumulation of certainty (and doubt), and the texture—novelty, ambiguity, topology—of understanding. He is especially interested in innovation—how new ideas and practices emerge—and the role that social and technical institutions (e.g., the Internet, markets, collaborations) play in collective cognition and discovery. Much of his work has focused on areas of modern science and technology, but he is also interested in other domains of knowledge—news, law, religion, gossip, hunches, machine, and historical modes of thinking and knowing.
Science is a cumulative activity, in which past knowledge serves as a foundation for new knowledge. One of the mechanisms through which the cumulative nature of science manifests itself is the act of citing. However, citations are also central to research evaluation, thus creating incentive for researchers to cite their own work. Therefore, such self-citations have been one of the most constant criticism against the use of citation indicators for the measurement of research impact. Using a dataset containing millions of papers and disambiguated authors, this talk will examine the relative importance of self-citations and self-references in the scholarly communication landscape, their relationship with age and gender of authors, as well as their effects on various research evaluation indicators. it will also present the results of a comparison of the content of cited and citing papers, thus making it possible to test whether researchers cite their own work in order to inflate their impact indicators. The talk with conclude with a discussion of the role of self-citations in the research ecosystem.
Vincent Larivière Full Professor, Canada Research Chair on the Transformations of Scholarly Communication, l’Université de Montréal
Vincent Larivière is full professor of information science at the École de bibliothéconomie et des sciences de l'information, l’Université de Montréal, where he teaches research methods and bibliometrics. He is also the scientific director of the Érudit journal platform, associate scientific director of the Observatoire des sciences et des technologies and a regular member of the Centre interuniversitaire de recherche sur la science et la technologie. He holds a B.A. in Science, Technology and Society (UQAM), an M.A. in history of science (UQAM) and a Ph.D. in information science (McGill), and has performed postdoctoral work at Indiana University’s Department of Information and Library Science.
Scientific enterprise has undergone a major transformation since World War II. In order to help solve increasingly complex outstanding problems and questions, contemporary science has adopted new approaches to knowledge production that are predominantly team based and are less confined by disciplinary boundaries. The character of the social, institutional and intellectual aspects of science and their interplay is very complex, and requires new approaches. In this talk I will showcase a number of studies that use the data from scientific publications to shed light on contemporary research practices, such as the formation of research teams, research workforce, interdisciplinarity, productivity and citation dynamics.
Staša Milojević Associate Professor, Department of Informatics, Indiana University, Bloomington, Indiana University
Staša Milojević is an Associate Professor in the Luddy School of Informatics, Computing, and Engineering, the director of Center for Complex Networks and Systems Research, a core faculty of Cognitive Science program, and a fellow of Rob Kling Center for Social Informatics at Indiana University, Bloomington.
The goal of this lecture is to introduce students to the tools that are used to estimate sociodemographic characteristics of authors from large-scale bibliometric data, with an emphasis on intersectionality. These approaches will be critiqued and the limitations demonstrated through validation studies. The lecture will then demonstrate the insights that can be drawn from the data, when the analysis are applied rigorously.
Cassidy Sugimoto Professor, School Chair, and Tom and Marie Patton Chair in the School of Public Policy at Georgia Institute of Technology, Georgia Institute of Technology
Dr. Cassidy R. Sugimoto is Professor and Tom and Marie Patton School Chair in the School of Public Policy at Georgia Institute of Technology. Her research examines the formal and informal ways in which knowledge is produced, disseminated, consumed, and supported, with an emphasis on issues of diversity, equity, and inclusion. Sugimoto was a professor of Informatics in the School of Informatics, Computing, and Engineering at Indiana University Bloomington from 2010-2021 and served as the Program Director for the Science of Science and Innovation Policy program at the National Science Foundation from 2018-2020. She has received the Indiana University Trustees Teaching award (2014), a national service award from the Association for Information Science and Technology (2009), and a Bicentennial Award for service from Indiana University (2020). She holds a bachelor’s in Music Performance, a master’s in Library Science, and a doctoral degree in Information and Library Science all from the University of North Carolina at Chapel Hill.
Hot streaks dominate the main impact of creative careers. Despite their ubiquitous nature across a wide range of creative domains, it remains unclear if there is any regularity underlying the beginning of hot streaks. Here, we develop computational methods using deep learning and network science and apply them to novel, large-scale datasets tracing the career outputs of artists, film directors, and scientists, allowing us to build high-dimensional representations of the artworks, films, and scientific publications they produce. By examining individuals’ career trajectories within the underlying creative space, we find that across all three domains, individuals tend to explore diverse styles or topics before their hot streak, but become notably more focused in what they work on after the hot streak begins. Crucially, we find that hot streaks are associated with neither exploration nor exploitation behavior in isolation, but a particular sequence of exploration followed by exploitation, where the transition from exploration to exploitation closely traces the onset of a hot streak. Overall, these results unveil among the first identifiable regularity underlying the onset of hot streaks, which appears universal across diverse creative domains, suggesting that a sequential view of creative strategies that balances experimentation and implementation may be particularly powerful for producing long-lasting contributions, which may have broad implications for identifying and nurturing creative talents.
Dashun Wang Associate Professor, Kellogg School of Management, Northwestern University, Northwestern University
Dashun Wang is an Associate Professor of Management and Organizations at the Kellogg School of Management, and (by courtesy) the McCormick School of Engineering, at Northwestern University. His current research focus is on Science of Science, a quest to turn the scientific methods and curiosities upon ourselves, hoping to use and develop tools from complexity sciences and artificial intelligence to broadly explore the opportunities for innovation and promises of prosperity offered by the recent data explosion in science.
I describe two developing, publicly-available, interlinked data initiatives that provide unprecedented visibility into the STEM enterprise. UMETRICS uses data on payments on sponsored research projects from 72 university campuses comprising 40% of academic R&D in the U.S. to identify the full teams of researchers conducting research. These data have been linked to a wide range of researcher characteristics and research outputs, including author-ity data on biomedical publications. We describe results on the production, diffusion, and value of research with a particular emphasis on underrepresentation.
Bruce Weinberg Professor, Department of Economics, The Ohio State University, The Ohio State University
Bruce A. Weinberg received his Ph.D. from the University of Chicago in 1996 before joining the faculty at the Ohio State University, where he is now Professor of Economics and Public Affairs. His research has been published in journals including The American Economic Review, The Journal of Political Economy, The Proceedings of the National Academy of the Sciences, and Science. This research spans three areas. The first is the economics of innovation and creativity. This work studies the production, diffusion, and value of innovations, including how creativity varies over the life cycle, how an individual’s own creativity is affected by the presence of other important innovators, and disparities among researchers. The second area is family and neighborhood determinants of youth outcomes and behavior. This work studies how youth behaviors, including employment, delinquency, cognitive development, and risky behaviors, are affected by their families and peer groups. The third research area concerns technological change, industrial shifts, and the wage structure. This work studies how computerization and the shift from manufacturing to services have affected the gender wage gap, the racial wage gap, and the returns to experience.
Digitization of the scientific literature has been transformative. Researchers and the public can now instantaneously access millions of papers—unquestionably a good thing for science and society. However, this access generally filters through search engines, social media, and recommender systems. It is less clear whether, and in what ways, these filters are benefiting and changing the practice of scientific discovery and knowledge dissemination. Are they increasing access to a broader range of the literature and thereby democratizing science? Or are scientists reading a more concentrated set of papers? And how are these filters and potential echo chambers impacting public understanding of science? This talk will highlight a set of studies that investigate the extent of citation and usage-based concentration, the changing role of journals and gatekeepers of science, and how these changing roles impact trust and perception of science during a pandemic engulfed by a simultaneous infodemic.
Jevin West Associate Professor, Information School, University of Washington, University of Washington
Jevin West is an Associate Professor in the Information School at the University of Washington. he co-founded the DataLab and direct the Center for an Informed Public. he studies the Science of Science and worries about the spread of misinformation. My laboratory consists of millions of scholarly papers and the billions of links that connect these papers. he develops knowledge discovery tools to both studies and facilitates science. In particular, Iheam interested in the origin of scholarly disciplines and how sociological and economic factors drive and slow the evolution of science.
Daniel Acuña Assistant Professor in the School of Information Studies at Syracuse University, Syracuse University
Daniel Acuña is an Assistant Professor in the School of Information Studies at Syracuse University, Syracuse, NY. The goal of his current research is to understand decision making in science—from helping hiring committees to predict future academic success to removing the potential biases that scientists and funding agencies commit during peer review.
Stephen David Associate Professor of Otolaryngology - Head and Neck Surgery, School of Medicine, Oregon Health & Science University
Stephen David joined the OHSU faculty in February 2012. Before coming to OHSU, he received his Ph.D. in Bioengineering from the University of California, Berkeley in 2006 and subsequently completed postdoctoral work in the Institute for Systems Research at the University of Maryland, College Park.
The COVID-19 pandemic made apparent many of the strengths and breaking points in the scientific enterprise. On the positive side, it demonstrated that the scientific community can come together to solve challenging problems in a fast and efficient manner. On the negative side, it revealed all the ways in which the system is broken and rotten at its core. From politicization of scientific knowledge and uncertainty, to the outrageous work demands put on the most exploited members of the community, there is much to rally against. It is upon each and every one of us to not accept this status quo and to fight for a scientific enterprise that works for the good of humankind and the environment.
Luís A. Nunes Amaral co-Director, Northwestern Institute on Complex Systems Professor of Chemical & Biological Engineering, Northwestern Institute
Professor Amaral, a native of Portugal, conducts and directs research that provides insight into the emergence, evolution, and stability of complex social and biological systems. His research aims to address some of the most pressing challenges facing human societies and the world’s ecosystems, including the mitigation of errors in healthcare settings, the characterization of the conditions fostering innovation and creativity, or the growth limits imposed by sustainability.
Any understanding of teamwork and its success is incomplete if we ignore the notion of power. Power manifests through cooperation. as long as there are more than one person working on a given task, power starts to influence every step of communication and activities. Power is pervasive, complex, and often disguised in our research community. But power is an essential, unavoidable, yet unrecognized element of scientific collaboration. This talk will highlight some thoughts about team power, how to quantify team power in science of science, and the potential relationship between the dynamics of team power and team success.
Ying Ding Bill & Lewis Suit Professor School of Information University of Texas at Austin, University of Texas at Austin
Dr. Ying Ding is Bill & Lewis Suit Professor at School of Information, University of Texas at Austin. Before that, she was a professor and director of graduate studies for data science program at School of Informatics, Computing, and Engineering at Indiana University. She has led the effort to develop the online data science graduate program for Indiana University. She also worked as a senior researcher at Department of Computer Science, University of Innsburck (Austria) and Free University of Amsterdam (the Netherlands). She has been involved in various NIH, NSF and European-Union funded projects. She has published 280+ papers in journals, conferences, and workshops, and served as the program committee member for 250+ international conferences. She is the co-editor of book series called Semantic Web Synthesis by Morgan & Claypool publisher, the co-editor-in-chief for Data Intelligence published by MIT Press and Chinese Academy of Sciences, and serves as the editorial board member for several top journals in Information Science and Semantic Web. She is the co-founder of Data2Discovery company advancing cutting edge AI technologies in drug discovery and healthcare. Her current research interests include bibliometrics, data-driven science of science, team collaboration, AI in health, knowledge graph, and scholarly communication.
The information professions need a paradigmatic shift to examine the ways we have systematically undermined knowledge systems falling outside of Western traditions. Epistemicide is the killing, silencing, annihilation, or devaluing of a knowledge system. Epistemicide happens when epistemic injustices are persistent, systematic, and collectively work as a structured oppression of particular ways of knowing. addressing epistemicide is critical for information professionals because we task ourselves with handling knowledge from every field. There has to be a reckoning before the paradigm can truly shift; if there is no acknowledgement of injustice, there is no room for justice.
Beth Patin Assistant Professor at Syracuse University’s School of Information Studies, Syracuse University
Dr. Beth Patin is an Assistant Professor at Syracuse University’s School of Information Studies. Beth’s research agenda focuses on the equity of information in two research streams: crisis informatics and cultural competence. She is the co-founder of the Library Information Investigative Team research group. Currently, she is working on projects about epistemicide and at the intersection of disability and race in youth literature. Additionally, she is a member of the Advisory Board on the Laura Bush Foundation for America’s Libraries.
Data production processes have rarely been included in quantitative study of science not only due to the difficulty in gathering evidence and wide variant data production methods and methods in different fields, but also due to its low status in scholarly publishing hierarchy. The enablers of research collaboration include cyberinfrastructure, science policy, and sci-tech human capital. to evaluate how effectively these enablers support and foster research collaboration needs to take more than publication metadata into the metrics. The abundance of scientific data repositories and the metadata in these repositories in particular offer a great opportunity for us to reexamine the measure and methods for impact assessment, or assessment of the effectiveness of enablers in fostering collaboration and accelerating the transition from data to knowledge. This presentation talks about the theory development of the collaboration capacity framework through using the GenBank metadata as empirical evidence in the data-intensive biomedical research.
Jian Qin Professor at the iSchool, Syracuse University, Syracuse University
Jian Qin is Professor at the iSchool, Syracuse University. The areas of her research interest include metadata, knowledge and data modeling, scientific communication, research networks, and research data management. She received funding from IMLS to develop an eScience librarianship curriculum and from NSF for the Science Data Literacy project. Jian Qin directs a Metadata Lab that focuses on big metadata analytics and metadata modeling and linking.
Keeping up with scientific literature is like drinking from a firehose. as more papers are published each year, automated methods for discovering, reading, and understanding the literature are needed to help us identify the latest and most applicable findings. This is especially the case in biomedical literature, where novel developments may have direct and immediate ramifications to patient care. in this talk, I will provide an overview of several projects that focus on scientific document understanding. I will discuss recent work on classifying text in scientific papers by combining document visual layout and semantics, which can produce high-fidelity, machine-readable representations of scientific papers that are useful for a variety of downstream applications. I will then provide several examples of how NLP and deep learning methods can be applied to specific problems in biomedical text mining such as relation extraction, fact checking, and multi-document summarization. These problems involve interactions between many documents, and the success of our models depends on our ability to accurately retrieve and extract information from all relevant documents. I will close with a discussion of ongoing challenges in this domain, and how we are hoping to address these challenges going forward.
Lucy Wang Postdoctoral Investigator, Allen Institute for AI (AI2), Allen Institute for AI (AI2)
I am a Postdoctoral Investigator at the Allen Institute for AI (AI2) in the Semantic Scholar research group. I work on KR and biomedical ontologies, text mining and NLP for biomedical and scientific text, open access, and meta-science. I completed my PhD in the Department of Biomedical Informatics and Medical Education (BIME) at the University of Washington in Seattle, WA. I also hold degrees in Biomedical Engineering and Physics from Johns Hopkins and MIT respectively.
The goal of most empirical studies in policy research and medical research is to determine whether an alteration in an intervention or a treatment will cause a change in the desired outcome response. Unlike randomized designs, establishing the causal relationship based on observational studies is a challenging problem because the ceteris paribus condition is violated. When the covariates of interest are measured with errors, evaluating the causal effects becomes a thorny issue. Additional challenge arises from confounding variables which are often of high dimensional or correlated with the error-prone covariates. Most of the existing methods for estimating the average causal effect heavily rely on parametric assumptions about the propensity score or the outcome regression model one way or the other. In reality, both models are prone to misspecification, which can have undue influence on the estimated average causal effect. To the best of our knowledge, all the existing methods cannot handle high-dimensional covariates in the presence of error-prone covariates. We propose a semiparametric method to establish the causal relationship, which yields a consistent estimator of the average causal effect. The method we proposed results in efficient estimators of the covariate effects. We investigate their theoretical properties and demonstrate their finite sample performance through extensive simulation studies.View presentation
Jianxuan Liu Assistant Professor, Department of Mathematics, Syracuse University, Syracuse University
Jianxuan Liu is an Assistant Professor of Statistics in the Department of Mathematics, and a Senior Research Affiliate in the Center for Policy Research, Syracuse University. Her research interests include causal inference, semiparametric modeling, high-dimensional data, measurement error models, and case-control study. She has a wide range of statistical consulting experiences in industry, non-profit research institutes, and academia.