Thoughts on Libraries and Information

Haustein, S. (2014). Readership metrics. In B. Cronin & C.R. Sugimoto (Eds.), Beyond bibliometrics: Metrics-based evaluation of research (pages tbd). Boston, MA: MIT Press.

Haustein argues that citation-based metrics for journals fail to capture information about the “pure readership” of publications—thus missing many instances of “impact”—instances which are better measurable by altmetrics. The question of whether to include these readers in the judgment of science returns to an argument we have also had about citations themselves—that bad science gets cited negatively. Research has shown that wholly negative citations are exceedingly rare, but it’s likely that bad research is much, much more often read than it is cited. Therefore, it could be problematic to include readership and access data in metrics that will, undoubtedly, be co-opted to assign concrete value measurements to science.

Interestlingly, Haustein points to the work of librarians in collection decisions to support her readership metrics argument. Although undoubtedly readership should be included in collection management decisions, those decisions are rare now that big deals and approval plans account for the vast majority of purchases in under-staffed or over-extended libraries.

Haustein seems to focus on the incompleteness of traditional data (citations don’t show readership) but I would argue that readership and impact are not as synonymous as her assumptions make them out to be. Surely many scientists read and are impacted by research from fields other than their own and often don’t cite that work—but this is a failure on the part of the researchers since those same connections could be vital to others who read the output that inter-disciplinary exposure inspires. It may be more effective to encourage stronger citation habits than simply counting readers because readers might come to a paper for any reason, but scientists will generally only cite work that contributes to and directly impacts science. If unique download counts are used to measure science, the mediocre author of ten mediocre papers who teaches Chemistry 100 to several hundred students each semester could vastly inflate his own impact scores by simply assigning readings for his tests. Certainly readership information is indicative of readership; but does it really tell us how impactful a paper is?

Davis, P. (2013). Visualizing Article Performance — Altmetrics Searches for Appropriate Display. The Scholarly Kitchen, September 30, 2013. Accessed April 30, 2014.

As Davis points out, the rise of quantitative measures of scientific output has resulted from an ever-growing number of scientists over the past several decades and the increasing difficulty of keeping track of individuals and individual pieces of their output. 

Whereas NISO may help standardize how things are counted and reported, they cannot help determine what these new metrics mean. Whether tweets, for example, measure popularity or prestige or simply the noise of narcissistic authors, bored academics, and automated bots will need to be decided by individual communities for specific functions: for instance, by a research university for the purposes of promotion and tenure; by a teaching hospital for the purposes of education; or by a government health department for the purposes of public engagement. 

Davis explains that this difficulty results in a focus on displaying information as opposed to interpreting it—let each visitor to a page come to his or her own conclusions about the meaning of the number and distribution of each type of online interaction (tweets, posts, downloads, views, etc.). Whereas the number with three decimal places used to represent an impact factor makes the score seem precise, a difference of .001 is basically meaningless—new altmetrics services like Altmetric and F1000 provide less precise scores and, in the case of the former, they also share exact numbers with the user (although the algorithm by which those numbers are transformed into an Altmetric point value is more obscure). F1000 only shares its base data with members of its “faculty.”  

Altmetrics produces its point value based on a complex algorithm that intends to eliminate bias:

[To produce an Altmetrics score, ] each data source starts with a base score, which is then weighted based on the author of the mention. The weight of each author is based on three components: bias, promiscuity, and reach. For Twitter, bias is measured by how often an author tweets about the same journal or DOI prefix; promiscuity is measured by how often an author tweets about papers; and reach is measured by an author’s number of Twitter followers as well as how many of those authors also tweet about papers.

Carpenter, T. (2013). Stick to Your Ribs: Altmetrics — Replacing the Impact Factor Is Not the Only Point. The Scholarly Kitchen, January 24, 2014. Accessed April 30, 2014. http://scholarlykitchen.sspnet.org/2014/01/23/stick-to-your-ribs-altmetrics-replacing-the-impact-factor-is-not-the-only-point/

Carpenter begins by outlining the traditional metrics by which scholarship is judged (press imprimatur, citations, journal reputation, and journal impact factor), but continues on to an interesting discussion of the inspiration for scholarly communication’s current push away from those metrics and toward new ones based on web-based data and communication practices.

Scholarly Communication researchers and others with dogs in the journal metrics fight are, according to Carpenter and Joe Esposito, interested in altmetrics—at least partially—because they present the first opportunity in 50 years to displace journal impact factor and re-focus the attention of scholars and institutions on actual quality instead of a skewed and often manipulated representation of journals as containers for scholarly potential. The JIF choke-hold on publishing has rewarded reputation and longevity with a metric that seems to reflect high quality science, but may be only indicating name recognition or manipulation; in such a system, new journals have a difficult time gaining readership, subscribers, prestige, and importance—regardless of the quality of the science they publish.

Priem, J. (2014). Altmetrics. In B. Cronin & C.R. Sugimoto (Eds.), Beyond bibliometrics: Metrics-based evaluation of research (pages tbd.). Boston, MA: MIT Press.

Priem argues that bibliometrics in the traditional sense are no longer (if they ever were) capable of tracking interactions between scientists and the work that influences them. As evidence he presents:

only about 15-20% of scientist in the United States have authored a refereed article (King & Tenopir, 2004); 

many important artifacts like datasets are not cited (MacRoberts & MacRoberts, 2010);

many highly useful journals are infrequently cited (Garfield, 1972);

citations produce outcomes that are counterproductive to Mertonian norms (Bornmann & Daniel, 2008);

citations only reflect the rigid formal communication system, whereas science itself often happens in informal systems (Price & Beaver, 1966)

Given the rapid expansion of social networks like twitter into the work lives of young researchers, we should begin thinking about how we can trace science interactions through these documented, informal communication systems. Priem points out that the adoption of these new technologies is not significantly changing the process by which science is done—but providing a new and differently effective means for communication; the most significant change new communication technologies afford to science is that they are traceable and produce easily quantifiable data. It is the ease of gathering and analyzing these massive new data sets that Priem sees as the main difference between earlier citation alternatives.

For the most part, Priem’s altmetrics are data about views or sharing, but he also mentions Faculty of 1000, a review service managed by a core “faculty” of selected scholars as well as scholarly blogging. These two sources of data for altmetrics are examples of how web 2.0 technologies facilitate the practice of applying human-level critical assessment for impactful research—the greatest shortcoming of quantitative citation analysis.

In addition to applying altmetrics to the work that has traditionally been evaluated with citation analysis, Priem points out that we can also utilize many emerging methods to assess new forms of scholarly output that don’t cite and are rarely cited themselves. Given that these new forms of output are highly non-standardized, however, the process of assessing the relative impacts of different forms of communication will still be difficult.

We still know very little about how sociological factors affect citation rates. . . . On the other hand, we know that citation rates say something about the contribution made by an individual’s work, at least in terms of the utility and interest the rest of the scientific community finds in it. (Garfield, 1979, p. 372) 

Nisonger, T.E. (2004). The benefits and drawbacks of impact factor for journal collection management in libraries. Serials Librarian, 47(1-2), 57-75.

Nisonger outlines the different arguments being made about citation counts and Journal Impact Factor from different perspectives. He concludes that JIF is a useful tool but must be used in combination with other criteria—but his focus is on collection development decisions in academic libraries. 

Nisonger wrote this in 2004—meaning he is writing from the perspective that JIF is still one of the most valuable metrics for journal usefulness. Since 2004, librarians have gained access to countless new metrics and statistics to better gauge the value of each journal to their patrons; JIF is only capable of gauging a journal’s value to its field on the largest scale—for example, the particular chemists at a certain university may operate within a small niche that isn’t served by the most cited Chemistry journals, but instead, by a few small niche journals.

Journal Impact Factor is calculated by counting the number of citations last year to any item published in the two previous  years, then dividing that by the number of citable items published over the same two years. It is important to note that citations to reviews and other non-article parts of journals count for the numerator, but do not count for the denominator (a fact Nisonger seems to disagree with in Figure 1)—this allows journals to inflate their JIF by publishing high quality non-article citable pieces that are likely to be cited but do not increase the denominator of the equation. 

Nisonger points out that immediacy index, cited half life, and total citations are also useful (if myopic) indicators of each journal’s engagement with the scholarly community—and that all of these should be considered in addition to JIF. Impact Factors are not often combined with other metrics. This makes their applications to the many fields Nisonger outlines (journal collection management decisions in libraries, journal rankings, journal decision models, evaluation and selection of full-text databases, measuring the acceptance of electronic journals, use of impact factor by editors, publishers, and authors, research about journals, performance evaluation, and adaptation for other contexts). 

Archambault, E. & Lariviere, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 1-15.

Archambault and Larivière start by making the perennial argument against JIF as a measure of the quality of science—that it was created to help librarians make collection decisions for fields with which they’re not yet familiar. Given that librarians rarely have complete control over serials collections due to big deal subscriptions or consortial obligations (not to mention accreditation requirements), the journal impact factor’s use in collection decisions is no longer required. Add to this the fact that librarians now have extensive records about ILL requests and downloads, as well as data about how many users attempted to download articles from journals to which they don’t subscribe—and the journal impact factor is suddenly a powerless measure compared to what publishers themselves will freely deliver to the librarian’s inbox with any frequency he or she requests.

The real problem is that JIF isn’t used for collection development anymore; it may be mentioned during big deal negotiations, but far more important to librarians is the actual number of requests received for articles from the journal or the cost per use. JIF is now used as a metric for the value of a journal; an extension of its original purpose that at least makes sense—it’s still related to how librarians collect serials. The less benign extension of JIF-as-value-judgment is the use of a journal’s impact factor to evaluate individual works of the researchers who publish in them. By adding the JIF of the journal in which each paper was published in his or her tenure dossier, the researcher implicates his own research in the usefulness of the journal as a whole—which works just as well for the author of the journal’s best paper as it does for that of its worst paper.

The authors point out that even if used appropriately, the power we place in JIF makes it a prime target for manipulation and abuse. Journals themselves can artificially inflate their JIF by publishing fewer articles and more citable non-articles (reviews, reactions, etc.). In addition to intentional inflation, the effect that JIF has had on publishing decisions since the 1960s should not be ignored—extremely high JIFs mean publishers can be very selective with the work they publish and their reputations will continue to increase at the expense of new entrants into the market who can’t rest on established JIFs.

The authors suggest we should either:

Abandon the use of the Journal Citation Report altogether,

Find some way to ensure it is used appropriately and does not lead people to making assumptions about the quality of articles published in certain journals, or

Redesign the JCR to account for enough measures that it actually serves as an accurate proxy for journal quality.

Baum, J. A. (2013). The excess-tail ratio: correcting journal impact factors for citation distributions. M@ n@ gement, 16(5).

Baum begins by clarifying the original purpose of Journal Impact Factor—to aid librarians in making collection development decisions by quantifying the probability that papers published in each journal will be useful for research. Baum proposes that a better gauge of the usefulness of journals would account for the distribution of infrequently cited articles instead of rewarding journals that publish few papers and happen to have a few hits.

By measuring the impact of the journal’s least cited papers as well as that of the most cited ones, Baum’s proposal addresses the free rider problem of work that is not particularly useful being deemed more important simply for having been published in a journal that has published a handful of heavily cited papers. That problem is lessened when one sees not just the JIF if the publishing journal, but also the paper’s individual citation statistics. An author’s average citation counts will often paint a different picture of his or her career than just the impact factors of the journals he or she publishes in. 

Bowley, C., & Vandegrift, M. (2014). Librarian, Heal Thyself: A Scholarly Communication Analysis of LIS Journals.

I am wary of any research that takes a firm stance on the “is open access good” debate—because the contention around open access is well founded on both sides. The cotton candy and daisies view held by most with an opinion on the matter vastly oversimplifies the current state of scholarly publishing. This article points specifically to respected journals maintained by scholarly societies to shame their continued paid subscription model but ignores the fact that, without income streams from the publication of journals, many scholarly societies would cease to exist. Does science depend on the generosity of libraries to fund conferences, grants, and other (sometimes open access) publications? Absolutely. But this benefit does not go unnoticed by the members of those societies and I can guarantee that library money is being flushed down many less worthwhile toilets than subscribing to JASIST.

This study also bases its argument on the logic that librarians prefer free materials, so they must want to publish their own work in journals that are free. While this is true, some factors must be remembered:

LIS journals are generally far less expensive than those of other fields (average cost in 2011 for LIS journals was $616, whereas every single hard science discipline had an average of more than $1000, with Chemistry leading the pack at $4044, almost 7 times as expensive as LIS) (Bosch, Henderson, & Klusendorf, 2011)

Many libraries do not bother collecting LIS materials since there are very few LIS programs in the US—so librarians aren’t likely to have to purchase the material back anyway

Librarians and information scientists are among the most likely researchers to contribute to their fields in ways that completely circumvent the publishing industry (Digital Humanities, alternatives to traditional publishing-based promotion and tenure models, and vast numbers of conference posters and presentations all speak to this last point)

My issue with research that brandishes a stick at any discipline to demand that it embrace open access is that disciplines develop naturally and fluidly, they are complex things that were not built to withstand huge shocks. The support structure for switching every discipline’s research to open access simply does not exist yet. During the transition to full open access, libraries will be double-burdened, both supporting authors who are able to publish in open access journals by contributing to their publishing fees AND purchasing subscriptions to the closed journals that are absolutely required for many disciplines (whether for research or, more explicitly, by accreditation bodies). This paper certainly isn’t the most sensationalist call to action for open access, but its “why don’t we just…” attitude gives the wrong impression to readers and leaves them expecting over-night changes that simply aren’t possible. 

Open access is winning the publishing war slowly but surely. It is not the place of any author to tell a discipline to accept the expense of a major disruptive shift at the risk of leaving its scholarly societies in shambles, its libraries double-burdened, and its system of ensuring and preserving access to the scholarly record in question. Sometimes, it's best to just back off and let disciplines adjust to open access in their own ways at a pace that is minimally disruptive—open access will ultimately displace the traditional publishing model whether we rush it or not.

References:

Merritt, D. J. (2000). Scholarly Influence in a Diverse Legal Academy: Race, Sex, and Citation Counts. The Journal of Legal Studies, 29(S1), 345-368.

Merritt discusses the high number of female and minority entrants into law school faculties in the late 1980s. She examined the sample of all 815 professors who began tenure-track positions at US law schools between 1986 and 1991 and were still on the tenure track in the fall of 1998. Merritt questions the reliability of her statistics and suspects that the number of women may be overstated. She goes on to explain that the number of minority professors was too low to accurately assess different races—showing that even if citation counts indicate fairness, the employment practices of the universities 

Merritt found that in the field of law, new faculty members are cited with less bias than in many other fields. She found demographic indicators that could explain the slight difference between the white women and women of color and their white male colleagues—these demographic differences (Merritt lists parents’ levels of education as a prominent one) could also be linked to the effects of affirmative action placements, but that connection would be difficult to argue since Merritt’s sample comprised every single new faculty member at accredited US Law schools. 

This study’s greatest strength is its inclusivity and longitudinal nature. By excluding those individuals who fell out during the tenure process, however, she assumes that biased practices in academia did not result in departure from the academy. Her methodology lacks the precision afforded by current indexes of citations, but was at least well thought-out. It would be interesting to see how the same 815 professors’ citation counts would compare now (if adjusted for the length of their tenure). Ultimately, the number of variables studied overcomplicate the findings.

White women, women of color, and men of color joined law school faculties in substantial numbers only during the last generation. With the weight of cultural history against them, stereotypes still in play, conflicting demands from home, and less traditional credentials than some of their white male counterparts, it would not be surprising if these academic newcomers had difficulty establishing strong scholarly reputations. Instead, the analyses in this paper suggest an impressive level of scholarly success for the white women, women of color, and men of color who joined law faculties during the late 1980s. Average citation counts for these professors lag somewhat behind counts for their white male peers, but four factors counsel optimism about the gap.

These four factors are:

The gap is smaller than some expected

Sex and race differences are much smaller than differences between other factors like subject or the religion of the professor

for women, the gap can be traced to factors like socioeconomic background, religion, prestige of tenure track institution, subject taught, educational background, and number of prewiring publications (meaning that the difference may not be linked to sex at all)

Thomas, J. B. (1980). Scholarly Productivity in Psychology: a criticism of citation count research. British Educational Research Journal, 6(1), 91-95.

Thomas aims to outline the difficulties of using citation counts for analyzing changes in the practices of different disciplines (a field called Disciplinary Dynamics). Among Thomas’s problems with the practice are the difficulty of collecting adequate data, the unreliability of the data that was available in the case study (which, granted, was done almost 30 years ago), and the lack of understanding about the validity of the statistical approach to the study of citation practices in the first place.

Some of Thomas’s laundry list of issues are no longer valid arguments—his problem with data reliability would not exist given the current offerings of standardized citation data through aggregators. Additionally, computing technology, in combination with born digital data, can eliminate the issues he sees with operationalization of data analysis. 

The most striking argument he makes is that we simply don’t understand what citation counts actually mean (if anything). This same argument is made about the processes used by grant funding agencies for assigning funding to different research projects. The amount of money or trust we invest in these processes essential to science would demand a strong understanding of the logic that defines practices—but in neither case is the standard based on sound, researched, proven logic.

Lindsey, D. (1989). Using citation counts as a measure of quality in science: Measuring what’s measurable rather than what’s valid. Scientometrics, 15(3-4), 189-203.

Lindsey briefly covers the benefits of citation counts over earlier systems for gauging the quality of science. The greatest benefit of citation counts is that the author himself cannot artificially enhance his citation count since citations are created by other scientists. This assumes that self citations (one of the most common types of citations) are negated in the analysis. Lindsey notes that this benefit also assumes only weak relationships between scientists—that is to say, “a scientist most often doesn’t know the individual scientist who cites his or her work and will likely never meet the individual.” This assumption disagrees with the findings of other scholars which indicate that invisible colleges and sociotechnical networks comprise a large portion of the researchers any scientist is likely to cite. Because of this, citation counts benefit those scientists whose networks are broad and whose work applies broadly to the work of those in many fields.

Other complications exist—including an increased likelihood of citing authors who share the citer’s sex (or, presumably, other demographics); it is also postulated that longer works, since they contain more citable units of information, are more likely to amass citations than shorter works. Finally, other studies have shown that works which are cited most heavily tend to be compendia of knowledge—i.e., textbooks and other large reference resources that scientists can cite for hundreds or thousands of different proofs. Popular science, as Lindsey points out, is also disproportionately cited.

Lindsey is careful to point out that citation counts are very convenient and do indicate relationships between authors and efficiencies/peculiarities in the publication of science—but they are insufficient as measures of the quality of individual works or researchers. The information we can find in citation counts tells us about:

invisible colleges

the formation of disciplines

social networks among scientists

Bornmann, L., & Daniel, H.D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45-80.

The article outlines the many inherent issues with using citation counts for quality judgments. The point Bornmann and Daniel are making with their laundry list is that citations are simply impacted by too many variables unrelated to the quality of the work to serve as useful proxies for the value of research. Scientists follow different conventions for citation and make biased decisions about whom to cite (with men and close relations of the author being over-represented in citations). Add this to the disciplinary differences that shape citation practices and the increase in inter-disciplinary studies (and citations), and we’re left with a metric that describes relationships and connections—but not value or quality.

The Chubin and Moitra study found no evidence of entirely negative citations and only about 5% somewhat negative citations, which indicates that the problem of negative citations inflating citation counts is small but still exists. The sample size for this study was too small to be conclusive, but I expect a similar distribution for scholarship as a whole.

Vinkler’s study of citation practices in Chemistry found that most citations could be classified as necessary to the completeness of the research (with the second most common reason being that the source paper’s content was crucial to the findings in the citing paper)—but that the third most common reason for citation was self-citing of the author’s own previous work.

Crow, R. (2009). Income models for open access: An overview of current practice. Washington: SPARC.

This SPARC report outlines the types of decisions OA journals must make to build viable business models without traditional subscription income (or only basic supplementation from traditional journal products and services). Overall, SPARC encourages publishers to approach the question of financial viability from multiple different directions. Among the options for funding OA are Fees to authors, versioning income, voluntary fees from users, subsidies, relationships with institutions, and grants.

Established journals do not need to build brand recognition or respect within their scholarly communities, but also must deal with engrained financial strategies. On the other hand, new publications can custom build their economic strategy without worrying about precedence or tradition--but must also make a strong value proposition and convince scholars of their quality.

I worry that many of the strategies outlined in this report are patches to support otherwise viable publishing operations instead of strategies for fully supporting OA. Some of the suggestions (like author funding) serve as actual funding models, but most are supplemental options at best--with a referral fee rate of 6% and a site traffic of 100,000 unique visitors per year, even if 3% of visitors purchased from the advertiser, the publisher would receive only $5,000 in ad revenue (about half the per-article publishing cost of a high impact journal)--and traffic that high is unlikely for new or niche publishers.

Kaiser, J. (2010). Free journals grow amid ongoing debate. Science, 329(5994), 896-898.

Kaiser points to different studies that peg the number of articles available for free at somewhere between 7 and 20%. She seems to conclude that large official mandates are most influential way to broaden OA offerings. She focuses on PLoS and NIH as strong evidence that policy changes can support OA and cites various studies that show how OA can save money for the government and universities.

Kaiser points out that PLoS ONE was set to publish 7500 papers in 2010 (about 69% of submissions) and uses a publishing model based on ensuring the scientific rigor instead of the importance of the papers submitted for publications. While ONE's financial stability helps support the OA mission of the other PLoS publications, it is hard to argue that a journal publishing that many articles and openly admitting to less rigorous peer review is making a strong contribution to science. ONE's offer to publish articles regardless of their importance to the scientific enterprise would seem to support science that is basic, replicative, or otherwise not exciting, but those articles are still considered important to scholarly communication, so I have to ask whether these articles are being published despite their lack of "flashiness" or if ONE is actually publishing unnecessary papers as Kaiser seems to suggest.

Among Kaiser's other strong points are:

Science and Nature report a per-article cost of $10,000

Only 25% of the visits to PubMed Central were from University computers (maybe indicating that the public is accessing the work)

A study by Philip Davis found that after 2 years of randomly assigned open access in traditional journals, citations were not significantly affected

This last point speaks to Kaiser's argument that high citation rates for open access content actually reflects authors' willingness to paying OA fees for their best work.

Morris, S. (2006). When is a journal not a journal? A closer look at the DOAJ. Learned Publishing, 19(1), 73-76.

Morris set out to survey the offerings of the DOAJ and test the assumption that open access journals are less stabile just because of their youth. She examined 1,213 of the 1,443 journals in the DOAJ as of 2006 and found that:

20 were completely inaccessible

18 were partly inaccessible

10 were no longer OA

14 were not even journals (they aggregated links or other publications but did not produce original content

Peaks in new DOAJ journals were found in 1997 (when many journals went online for the first time) and again in 2000, presumably due to increased interest in open access.

People often point out that open access journals are problematic because they are unreliable. Morris points out that in addition to being transient and unreliable, a considerable number of OA journals don't even publish regularly. 140 journals surveyed had published fewer than 20 articles, 26 of those publishing five or fewer articles during their life span. 110 (almost 10%) journals had been inactive for three years prior to the survey.

Laakso, M., Welling, P., Bukvova, H., Nyman, L., Bjork, B-C., & Hedlund, T. (2011). The development of open access journal publishing from 1993 to 2009. PLoS ONE. Available at: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0020961

The authors sought to investigate growth and change among open access journals between 1993 and 2009. They examined both small and large journals as well as different waves of open access, from the pioneering years (1993-1999) to the innovation years (2000-2004) to the consolidation years (2005-2009).

The authors choose to examine changes over this time period because the factors that impact the success of open access publishing are heavily related to economic benefits of publishing online—which grow with time as technology becomes less expensive. The authors study only direct open access, which is only one of the many flavors that constitute the open access landscape. This is problematic because gold open access is not generally considered to be the singular solution to the serials publishing crisis. Other articles I have read find that green open access is a more tenable option—the journals following embargo or self archiving paths are ignored entirely by this study.

From the beginning of the authors’ discussion of their sample, it is clear that they found a large number of open access journals that had failed. They include such journals but never speak to the effect such failures have on the open access movement. The authors later mention that 8% of the 500 or so pioneering journals established between 1993 and 1999 had disappeared entirely (i.e. without a trace of their website or the articles they published) by 2009. This problematizes not only the study but the argument that open access journals are capable of preserving knowledge with any certainty.