Report of a Workshop at the University of Amsterdam, 21-24 May 1996.
“Measuring is knowing” is a truism in the sciences. In the interdisciplinary field of science and technology studies (STS), however, a large gap exists between quantitative and qualitative approaches. There are a few exceptions, but most scholars restrict themselves to one of these repertoires. As a consequence, scientometric studies are sometimes sociologically naive while qualitative analyses often neglect possibilities to test their hypotheses. This divide - might it be coined a great divide in science and technology studies? - is visible in methodology sections of articles, in conference selections, in the composition of personal networks of STS scholars, and, lastly, in their citation behaviour.
The quest for more integration between qualitative and quantitative perspectives was central to a workshop organized by the Network of European Centers in Science and Technology Studies (NECSTS). NECSTS has organized an annual series of ERASMUS workshops since 1992 designed to stimulate the vitality, social relevance, and intellectual coherence of the interdisciplinary field of science and technology studies (STS). Each of these workshops focuses on a specific specialty or discipline contributing to the wider field of study. Previous workshops have addressed “university- industry relations,” “public understanding of science,” “history of science and medicine,” etc. In stimulating discussions in working groups, those attending have ample opportunity to reflect on the usefulness of the pertinent approach for their own research.
In 1996, the workshop focused on scientometrics, the quantitative study of science and technology. As is well-known, the emergence of scientometrics as a specialty within the STS field is strongly associated with the creation of the Science Citation Index SCI in the 1960s. The importance of the SCI was immediately recognized by sociologists and historians of science like Derek de Solla Price and Robert King Merton. They contributed to the organization of the citation index (Elkana et al. 1978; cf. Wouters 1996), especially in the early years of its existence. The SCI is still the dominant data source for many scientometric studies.
During the 1980s, other databases and new approaches to databases became available. Crosslinks have been firmly established between scientometrics and the quantitative study of patent statistics (“technometrics”). Scientometrics has also been influenced by approaches used in other fields focusing on measurement (like informetrics, with its roots in library and computer sciences). From the perspective of scientometrics, three interfaces can currently be distinguished: - The interface with theorizing in the sociology, history, and philosophy of science. Problems of probabilistic inferencing, of organizing large datasets, and of testing for the significance of differences are pervasive in these disciplines, and the researcher may wish to turn to the more quantitative side of the field for expertise (cf. Leydesdorff 1995). - The interface with science and technology policy issues. Increasingly, sophisticated evaluation and forecasting studies are called for by policy makers. Scientometricians have been asked to map the future possibilities in specific fields, e.g., the choices on the agenda for Dutch physics (Gl„nzel et al. 1995). The relation between evaluation and forecasting is a crucial issue in debates at this interface (cf. Irvine & Martin 1984). In methodological terms, researchers focus on questions about the reliability and validity of indicators. - The increased use of S&T in advanced knowledge-based economies raises a third set of problems. How do these complex systems evolve? Is it possible to model their interactions? Recently, Gibbons et al. (1994) have distinguished a traditional “Mode 1” type of science from a new “Mode 2” type at the interfaces. Others have suggested a triple helix model of university-industry relations (Etzkowitz & Leydesdorff 1995; Leydesdorff & Etzkowitz 1996). Indicators of these developments must develop means to incorporate changes in the social environments of science and technology: for example, what counts as “biotechnology” is different in 1995 from its delineation in 1980.
Furthermore, the spread of the desktop environment, i.e. using the personal computer and workstation in networked enviroments, is revolutionizing scientometrics like many other fields of science. One tends to become less dependent on mainframe computers and established research institutions. The specificities of studying cognitive interactions like in science and science-based technologies, however, require methods different from those readily available in computer software for the social sciences (although there is some overlap). In summary, now is a good time to make these techniques available to more qualitatively oriented colleagues and policy makers so that frameworks for future collaborations can be explored.
The Amsterdam workshop was structured along these intellectual lines. Forty-four participants discussed these issues in both plenary sessions and smaller groups. Most participants were from Western European research centres; additionally, there were participants from Russia, India, Rumania, Hungary, and Australia. Each workshop was devoted to a specific topic: the social conditions of scientometrics, scientometrics and European science policies, scientometrics and STS theory, and scientometrics as a policy instrument. Special attention was also given to Ph.D. projects using scientometric techniques.
Arie Rip (University of Twente, The Netherlands) faced the central questions in his opening presentation by focusing on the relationships between scientometrics and the practices of science. He distinguished three consecutive phases in the development of scientometrics: the times of Derek de Solla Price that Rip characterized as “data-driven”; the late seventies and the eighties when science indicators became “the real business”; and present-day scientometrics.
What might be the difference between present- day scientometrics and scientometrics of the previous period? Rip formulated it as follows: “In my view, the third phase is one in which representation is becoming less interesting; nowadays, the added value of indicators and maps (for various audiences, scholarly and otherwise) counts.” But how should one assess the validity of indicators if they are no longer underpinned by claims of “objective” representations?
Rip proposed to distinguish between “natural” and “artificial” aggregations in science and the study of science for solving this problem. The sciences themselves are systems of communication that continuously organize and reorganize these communications in order to make them available for evaluation. The creation of databases like the Science Citation Index has made it possible to confront these “natural” aggregations with aggregations based on “artificial” reflections of the science system. This raises a number of questions, such as the priority of the various aggregations, the congruency among them, and the validity of the one aggregation (e.g., peer review) vis-…-vis the other (e.g., scientometric evaluation). The speaker noted that in some cases the independent evaluation by policy makers using scientometric information has already had a significant effect on the underlying processes of knowledge generation. This feedback loop induces discussions of quality control on the side of scientometrics that require discussions with non-scientometric colleagues in STS and policy analysis (cf. Glänzel 1996).
Rémi Barré (Observatoire des Sciences et Technologies, Paris) analyzed the complexity of the system of mediation between science & technology and policy making in terms of functions that have to be fulfilled in advanced industrial systems. These functions are carried out differently in various national systems in terms of the institutional divisions of labour between bureaucracies, universities, and para-university research. The author emphasized the need for entrenchement in national systems because of the availability of specific statistical resources, while national experiences need to be communicated so that international standards can be developed and codified in order to facilitate comparisons.
Ulrich Schmoch (Fraunhofer Institute, Karlsruhe, FRG) elaborated this perspective with a discussion of the specificity of the technology side of the system. Patents are different from publications since they refer to the capitalization of scientific knowledge. Recent advances in the dynamic modelling of these interactions were discussed. Although they occur along a different axis, Schmoch also emphasized the systemic nature of these developments, and the need to define units of analysis, levels of aggregation, and to hypothesize mechanisms for the transfer of knowledge. Tibor Braun from the ISSRU-unit in Budapest (Hungary) explained the independent processing of the Science Citation Index at his unit. He showed how the data is processed into indicators in various dimensions and time horizons. The trade-off between capturing the complexities of the system and the need to provide policy makers with comprehensive information was discussed in terms of the dimensionality of the representations. Braun suggested that the development of three-dimensional indicators be considered.
Possible applications of the noted development of “desktop scientometrics” were demonstrated by Sylvan Katz from the Science Policy Research Unit at the University of Sussex (UK). Katz presented his findings along these lines, using UK-data. He noted that the various national centres are “cleaning the data” for their respective countries. Perhaps one should agree on standards and organize the data internationally using internet and Unix as well as Windows-type programming. The audience agreed that desktop scientometrics provides policy makers with a laptop-model for asking specific questions, while such a system would provide the community of scientometricians with almost unlimited domains to discuss interesting questions concerning the research systems under study. However, intellectual property rights make it necessary to pursue these discussions in gremia other than this workshop.
Katz’s suggestions were underscored by a report by Paul Cunningham (PREST, Manchester, UK) about the use of scientometric evaluations in the various EU S&T programs. Although these programmes are committed to high-quality evaluation procedures, their use of scientometric data is not systematic. The effect of independently reached conclusions is sometimes dramatic, but this remains the exception. Thus, a rapprochement between scientometric discourse and policy discourse is still very much in need.
On the third day of the conference, the issue of differences among discourses was discussed in more theoretical terms. Barend Van der Meulen (Foresight Committee, The Netherlands) suggested that scientometricians are using a naive model of communication. The relevance of a scientometric communication is determined not in terms of its enlightenment of the policy process, but in terms of its usefulness to this process. Anthony Van Raan (Leiden, The Netherlands) accordingly emphasized the essential role of the client in developing this field. Scientometricians should understand their role as a service based on applied research. But in academia the applications have to be firmly rooted in basic research. Thus, the field should develop in terms of these two dimensions: basic as well as applied, customer-driven as well as academic. Additionally, Van Raan provided the audience with a comprehensive state-of-the-art review of the field of scientometrics.
Different dimensions of the communication, such as “utilization”, “codification”, etc., were distinguished by Loet Leydesdorff (Amsterdam, The Netherlands). He used aggregated citation networks among STS journals to show this effect reflexively at the level of STS as a field. Since 1986, the various core journals of STS (Social Studies of Science, Scientometrics, and Research Policy)1 have grown apart in terms of aggregated citation behaviour to such an extent that they can no longer be considered as a single grouping. The integration of such a (functionally?) differentiated system of communications can only be achieved by making a reflexive step forward. In his opinion, there is a need for models that take into account the highly skewed distributions that are produced over the years, as well as the insights into the underlying mechanisms provided by qualitative theoretical and empirical approaches. Thus, a second function of quantitative approaches in STS can be distinguished in addition to the data-driven approaches presently dominant in scientometrics. Theoretically informed indicators might enable policy makers to distinguish the various dimensions in the complex dynamics of S&T systems. Wouter van Rossum (Groningen, The Netherlands) had fallen ill prior to the workshop, but his paper was presented nevertheless. He focused on the dynamic overlay of university-industry-government relations which is emerging in advanced industrial countries. This raises new questions about the position of indicators. If the environments change, one has to be clear about what one is measuring if one keeps the yardsticks stable. The discussion at the workshop concentrated on the distinction between relational indicators that capture the operation of the system and positional indicators that try to capture its structure.
On the last day of the workshop, a number of advanced graduate students of the Dutch Graduate School for STS presented their thesis work, using scientometrics for the understanding of science and technology. Gertrud Blauwhof (Amsterdam) presented a longitudinal study of patent and science statistics in the case of “telephone switching systems.” Floor Rikken (Groningen) demonstrated the retrieval of developments in a pharmaco-medical setting using the words associated with so-called adverse drug reactions. Ed Noyons (Leiden) presented simulations of the development of neural network research using co-word models. Other, more detailed studies were presented by a number of participants in the various workshops during the conference.
The workshop concluded with a discussion about the future of scientometrics, introduced by Paul Wouters, who presented three possible scenarios: “a glorious future,” “impending death”, and “muddling through.” These scenarios were obtained by varying the development of the three interfaces mentioned above. Thus, the specialty of scientometrics may be expected to develop differently as its environments (science and technology studies, science policy, evolutionary economics) exert differential influences.
One topic discussed in this and several other workshop sessions was the divergent development of the quantitative and the qualitative study of science and technology. In the early years of scientometrics, a “natural coalition” seemed to exist between Mertonian sociology of science and scientometrics. Since the early eighties, this bond has been severed due to a number of intellectual, practical, and economic developments. The field of scientometrics seems to have moved towards a more intimate connection with the quantitative library sciences and related specialties in information sciences. Consequently, academic communication among science and technology students is no longer as seamless as it was in the past.
Quantitative modelling of the science/technology system, however, is in need of more advanced theoretical underpinning. From this perspective, future research should focus on combining scientometric research with both sociological insights (from the qualitative specialties in science and technology studies) and computational expertise from the various library, information, and data-structural sciences. According to the participants, this workshop has clearly contributed to this goal. An intensive exchange of opinions and insights has occurred among leading scholars in the field, including a non-neglizable Dutch community, European colleages at both the graduate and postgraduate levels, and relevant policy makers. Papers from the workshop will be published in a special issue of Scientometrics co-edited by the organizers of the workshop and the managing editor of this international journal.
We are pleased to acknowledge the financial support of the Netherlands Ministry of Education, Culture & Science, the European Union DG XII, the Netherlands Organization for Scientific Research, and the Royal Academy of Science. The workshop was co- organized by the European Association for the Study of Science and Technology (EASST) and the Netherlands Graduate School of Science, Technology, and Modern Culture. Note 1. STHV has a different position, since this journal has become the journal of 4S since 1988. In terms of citation behaviour, it can be considered as belonging to the same grouping as Social Studies of Science.
References
Elkana, Yehuda, Joshua Lederberg, Robert K. Merton, Arnold Thackray, and Harriet Zuckerman (1978). Toward a Metric of Science: The advent of science indicators (New York, etc.: Wiley).
Etzkowitz, Henry & Loet Leydesdorff (1995). “The Triple Helix of University-Industry-Government Relations: A Laboratory for Knowledge Based Economic Development,” EASST Review 14 (1), 11-19.
Gibbons, Michael, Camille Limoges, Helga Nowotny, Simon Schwartzman, Peter Scott, and Martin Trow (1994). The new production of knowledge: the dynamics of science and research in contemporary societies (London, Sage).
Glänzel, Wolfgang (1996); (Ed.) “Proceedings of the Workshop on ‘Bibliometric Standards’,” Scientometrics 35 (2), 165-277.
Glänzel, W., A. Schubert, T. Braun, E. Rinia, & M. Brocken (1995). Physics in the European Union in the 80s Utrecht: FOM).
Irvine, John & Ben R. Martin (1984). Foresight in Science: Picking the Winners (London: Frances Pinter).
Leydesdorff, Loet (1995). The Challenge of Scientometrics: The development, measurement, and self-organization of scientific communications (Leiden: DSWO Press, Leiden University).
Leydesdorff, Loet & Henry Etzkowitz (forthcoming 1996). “Emergence of a Triple Helix of University-Industry-Government Relations,” Science & Public Policy.
Wouters, Paul (forthcoming 1996). The Citation Culture. Ph.D. Thesis, University of Amsterdam.