campusreview. com. au industry & research
Please be more specific
What exactly constitutes a scholarly article isn’ t nearly as straightforward as it should be – and that’ s a costly problem.
By Ralf Buckley
Every modern civilisation relies on its universities. And governments count visible products – graduates and publications – to determine the value of an institution. That’ s why universities report publications annually to the Department of Education.
To standardise the statistics and prevent rorts, the government issues Higher Education Research Data Collection( HERDC) Specifications defining what counts and what doesn’ t. There are penalties for misreporting. Reported grants are audited, but reported publications are not.
These specifications are necessary to maintain quality. Academics write popular as well as scholarly articles and books. Scholarly journals publish book reviews and conference reports as well as research. Some journals are fake.
The government relies on universities to interpret the specifications consistently. But they don’ t. Both between and within universities, there are widely different interpretations. So reported publication counts are unreliable as an indicator of how much material meets quality standards.
Does it matter? Well, yes. The amount at stake, calculated below, is about $ 4 billion a year. This is not wasted. The research is done and published, available to industry and government. It’ s about claiming credit, for academics and universities, in rankings and research assessment exercises, and calculating returns on public investment.
Why $ 4 billion? Most academic workloads contain equal teaching and research, universities and the NTEU state. Australian universities receive about $ 11 billion annually from government, and that is 45 per cent of their total budget, about $ 24 billion, with $ 12 billion of that going to produce publications.
Of this, about one-third, or $ 4 billion worth, is in publication categories with inconsistent classification: research notes; letters; commentaries; perspectives, etc. How do we know? We can simply count them. Academic journals publish many different types of articles. Across a wide range of high-volume high-tier journals in multiple disciplines, about a third – in number of items, not page count – are of types where there are different interpretations of the HERDC specifications. That means we can’ t be sure of whether they’ re up to standard.
The 2012 Excellence in Research for Australia evaluation states that Australian universities produce about 66,000 research publications annually. So the average cost per publication is $ 12 billion divided by 66,000, or about $ 180,000. That is what all university funders, jointly, invest to produce each publication. Prolific publishers are valuable!
OK, now to the critical issue. Why are the specifications interpreted inconsistently? Academic disciplines have different publishing styles and traditions, and journals use a range of names for publications of different types and lengths. But the specifications have to define research publications in terms that apply equally across all disciplines.
To comply with the specifications, a publication must be in an academic, rather than a popular, publication, and it must meet seven standard requirements for“ substantial scholarly activity”: literature review; precedents; citations; originality; peer review; addition to knowledge; and formal publication in print, electronically or online.
Applying these seven criteria is not always straightforward. To help out, the specifications contain two lists – one of publication types that generally will comply and one of types that won’ t. For example, the lists distinguish between“ letters to a journal” and“ letters to the editor”, and between“ commentary” and“ brief commentary”. But journals use these terms differently, or not at all. And what does brief mean when word counts are not specified?
It appears universities misinterpret how to use these lists, and this is where the uncertainty originates. The correct way to read the specifications is according to the principles of statutory legal interpretation. The critical issue is that the lists are simply examples of the seven criteria; they do not supersede them. That means any item, no matter how long or short, and no matter how it is labelled, will count as compliant if and only if it appears in an academic publication and meets all seven criteria.
Perhaps some universities do indeed classify all their publications correctly into compliant and non-compliant categories. We don’ t know. But we do know that at least some universities, and even different faculties within the same university, have different interpretations for the same items. We know there is substantial uncertainty, worth up to $ 4 billion a year.
What can be done to overcome the problem? There are several obvious options. The Department of Education could publish some explanatory notes to the specifications. It could also publish lists of all compliant publications annually. Universities could compare notes independently on what they think does or doesn’ t count. They don’ t, because they have not yet appreciated how much is at stake. But academic publishing has changed much recently. The issue deserves closer attention. n
Professor Ralf Buckley holds the international chair in ecotourism research at Griffith University.
17