Altmetrics for Librarians and Institutions: Part III

Part III: Altmetrics in the wider academic context

In addition to being indicators of quality and impact across academic content (whether it is a journal article, dataset, slideshow or so on), there are some other real ways in which altmetrics may be able to contribute to important decision-making within an academic context. We identified three opportunities and once again discussed them with Heather Piwowar and Jason Priem.

1. Tenure, promotion and grant awards: In certain countries, the traditional journal impact factor has been and is being used to evaluate researchers capabilities, based on the level of impact that the journals they publish in have. There are obvious and well documented drawbacks of this approach. When impact factor metrics is used as key decision drivers for tenure and funding, the issues get serious. Altmetrics could fill an important gap here, once they have developed their focus to provide specific solutions to specific problems, and proven to be relevant to various research outcomes and institutional-level objectives. The response from Priem and Piwowar was that there are several opportunities for academia including “tenure committees looking at the broader impacts of research by examining altmetrics. Not to the exclusion of traditional evaluation techniques of course, but as a supplement.” Jason cited his own institution (University of North Carolina, Chapel Hill) as considering adding blogging as a key research outcome; he continued, “but how is anyone supposed to assess the impact of a blog? Altmetrics like downloads, tweets, social bookmarks, and wikipedia citations can help a lot here.” He continued that altmetrics “can [also] help tenure, promotion, and awards committees reward increasingly important products like software and datasets.”


2. Wider, measurable impact: Why wouldn’t an institution, department or team want to demonstrate the contribution they have made to the overall research objectives of the organisation, either in an outreach sense, or to meet funding targets? Altmetrics will allow them to do this, and since much of the data is open source, complete transparency would allow benchmarking against other institutions. Performance frameworks (such as REF or ERA mentioned in the first altmetrics post) could incorporate these kind of scores into their evaluation and funding decisions.


3. Institutional (and non-institutional) repositories: as Priem put it, “an IR using altmetrics to highlight content the community has adjudged important, via their tweets, blog posts, bookmarks, and so on. There’s great potential here.” Total-Impact runs on open data from various sources, relating to a fixed web location of a particular object. Non-journal assets that are stored in open repositories can also have value. Priem adds, “altmetrics can also help librarians make the case that alternative publishing venues (like Dryad and figshare for data, GitHub for software, and F1000 or PeerJ for articles) as places where research can have an impact in ways that don’t have “Factor” at the end.”

Are there any drawbacks?
It is undeniable that altmetrics have some way to go to prove their relevance to a host of different problems. There are going to be many experiments in the foreseeable future to determine specific value for specific groups. We wanted to look at the significance of building and using a single score to be used as a quality measure, and also the implications of applying altmetrics to eBooks.

"broader impacts of research by examining altmetrics"

Relevance – I asked Piwowar and Priem about the feasibility of a single score (in the same vein as Klout, Kred or PeerIndex which are all algorithm-based scoring systems for online social influence) that could be attributed to a journal, as an indicator of ‘quality’ or ‘impact’. Librarians could use this in gauging the relative value of journal holdings. The answer was one of caution:
“I think users, not creators of data, should be the one who get to decide what matters to them. Let’s imagine an algorithm that ranked journals based on 3 metrics (to use a simplified example): citations, Mendeley bookmarks, and tweets. I suppose many folks might like a weighting of 70:20:10 or similar. But what happens when we want to evaluate that journal’s performance over the last 6 months? Suddenly Mendeley inclusion (which happens much more quickly than citations) is a bigger deal. What happens when an author wants a public health study to reach a broad audience, fast? Suddenly the Twitter stats matter a lot more.”

eBooks – can altmetrics be successfully applied to books / eBooks / monographs as well? The answer is almost definitely yes, but the focus for altmetrics researchers and developers has so far been focussed on primary research output. Priem says,
“in principle, yes, but in practice no. There are lots of interesting metrics one could gather for monographs; for instance, Amazon reviews. However, so far altmetrics have really focused on either articles (in the case of PLoS ALM and altmetric.com) or articles and “alternative products” like datasets and software (in the case of total-impact). I think it’s just a question of this very young field maturing; I’d expect to see interesting book metrics surfacing in the next 12-18mo.”

"it’s just a question of this very young field maturing"

Altmetrics have an obvious, albeit not yet 100% defined, purpose in the academic world, where demonstrable impact is essential from the article level, right up to overall institutional goals. Tools and services like Mendeley (and Mendeley Institutional Edition), Total-Impact and Altmetric.com among others are pioneers in this new territory. The potential value these measures possess for librarians is only just starting to come to light. Once larger communities see the possibilities of mining the social web, the acceptance of altmetrics into the mainstream will accelerate. Priem sums it up nicely:

“I think that, used cautiously, altmetrics data can certainly help drive conversations about wider impacts, and those conversations need to be taking place more than they are.  We just also need to be realistic about the state of the art."

Read the first two parts of the Altmetrics for Librarians and Institutions blog series here:

• Part I: Altmetrics landscape, impact and value
• Part II: Librarians and the benefits of altmetrics

Leave us your comments below.