campusreview . com . au industry & research weighted apportioned research outputs over the six-year output reference period is less than 50 in any four-digit or two-digit FoR at an institution , then ERA will not evaluate that FoR at that institution ’ ( ERA23 Guidelines , p . 12 ). For a FoR to reach that threshold , it might submit 10 scholarly books . Alternatively it might submit 50 journal articles and scholarly book chapters . Or perhaps a combination of all three .
However , in FoR codes where major creative works feature prominently in the submission , how might a threshold be reached ? Currently , by awarding a single point for major creative works , the time a researcher has devoted to the development and creation of the work , the effort that the researcher has put in , and the achievement of the work , are not commensurately recognised .
In other words , by awarding 5 points for scholarly books and only 1 point for major works delivered in other forms , the research contributions ( and associated labour ) of the latter are seriously misrepresented .
There is no logical argument to justify why a major creative work should be valued at 1 while a scholarly book is valued at 5 . It is becoming increasingly clear that major works of scholarship , no matter what the media-form of their delivery , should be weighted at 5 points if they contribute an equivalent amount of new knowledge as would a scholarly book .
Towards assessment criteria to measure NTRO scale , impact and success Australian universities differ when it comes to measuring the research contribution of NTROs . This measuring process is done so as to ascertain whether a researcher is meeting their requisite internal institutional KPIs – hence each specific research ‘ output ’ is assessed and valued to determine the long-term performance and productivity of individual academics .
To measure this performance most universities have committees that assess each NTRO according to an internal scale . Some universities use a 1 or 5 scale that accords with the metric of ‘ traditional ’ research ; other institutions have used a 1 , 3 , 5 scale ; and a few employ a 1 , 2 , 3 , 4 , 5 system .
We think that a survey of these various attempts to calibrate the research contribution of specific NTROs could be useful in developing a more equitable ERA exercise – potentially addressing the comparative ease of rating an output against a notional ‘ world standard ’ by citation measures as opposed to peer review evaluation measures , and the documented tendency of creative practitioners to judge harshly in their own fields .
For example , Dr Karen Pearlman , senior lecturer and a director of the Creative Documentary Research Centre , has developed the following list of criteria as a starting point for discussion .
The first two points allow assessors to evaluate how many research points to allocate against the research time and effort a researcher has devoted to a work . 1 . Duration or size of output 2 . Effort and duration of process ( e . g . fieldwork , archival research , or production processes )
On the 5 point / 1 point equivalency model , a 5 under criterion # 1 might be : 40 minutes or longer in duration for a film , theatre , dance , sound or music work ; a substantial book of poems ; a novel or book-length work of creative nonfiction ; or a large-scale exhibition . Under criterion # 2 we note that it would unlikely for a researcher to be able to produce more than one work of a ‘ 5-scale ’ within a two-year period .
Principles three and four , below , move beyond metrics . 3 . Research question clarity : identification of context and kind of question . Is the research seeking knowledge of creative practice ? Or is the research question seeking knowledge of something other than creative practice which can only be secured through creative practice ? Is the context for this clearly articulated and has the researcher identified what s / he / they are addressing ? 4 . Research investigation depth : Depth of investigation of a question / sub-questions . This is helpful to consider if , say , the length of work does not seem commensurate with content , or when different forms express themselves in radically different durations .
The clarity of research question , including clarity about the kind of knowledge it is aiming to uncover , also makes evaluation of the research contribution possible .
Research contribution is identified / described in the research statements to guide the evaluator ’ s eye towards the knowledge contributions that are not explicitly stated as such . Therefore , qualitative considerations of contribution might include :
Creative arts outputs have been consistently ( and concretely ) under-valued as research .
5 . Research ’ s creative contribution to knowledge : How the creative work as a creative work generates new knowledge . 6 . Aesthetic achievement in form , expressiveness , creating and conveying of ideas through means other than direct statement .
Clear articulation of question and contribution also makes possible an evaluation of innovation or whether the knowledge generated is new knowledge . Here it is once again important to determine whether the research is asking a question about creative practice or through creative practice . 7 . Innovation in creative practice technology ; creation or refinement of a method or process ; or innovation in form 8 . Innovation , insights , apprehension , or comprehension generated through achievement in form
Finally , measures of significance are provided in the research statements . Except for point nine , which requires some disciplinary expertise to discern , these are more clear-cut evaluative measures . Like a Scimago ranking , these culturally determined measures of significance are used by ARC evaluators to augment their own views with the expertise of the curators , judges , critics or publishers that have evaluated the work publicly . 9 . Significance of the new knowledge to the field 10 . Significance of venue , publisher , broadcaster , or funding source 11 . Recognition by experts or position of work in the field .
This article represents the individual authors ’ views and not those of Macquarie University . ■
Bronwen Neil FAHA is professor of Ancient History , Karen Pearlman is a senior lecturer in Screen Practice and Production , Kate Rossmanith is associate professor of Media and Creative Practice , and Tom Murray is associate professor of Screen Media and Creative Practice at Macquarie University .
21