industry & research campusreview . com . au
Up with the times
Research in the 21 st Century : legacy forms , ‘ new ’ media and ERA .
By Bronwen Neil , Karen Pearlman , Kate Rossmanith and Tom Murray
As Australian universities prepare their submissions for the 2023 round of Excellence in Research Australia ( ERA ), the national research evaluation framework , it is a good moment to reflect and review this thing we call ‘ research ’.
Indeed the ERA exercise , which is run every three years to evaluate how different disciplines across the Australian research landscape are performing against an ambiguous ‘ world standard ’, is currently seeking feedback-submissions on the recently released draft ERA23 submission guidelines .
We , at Macquarie University ’ s Creative Documentary Research Centre , take this as an opportunity to highlight a significant anomaly in research assessment that seems increasingly anachronistic in the 21 st Century .
The specific context is that much of the major research that our members produce is in a form that is quaintly described as ‘ Non- Traditional Research Outputs ’, or NTROs defined in distinction to the more common ‘ Traditional Research Outputs ’. While NTROs within universities are produced across many Fields of Research ( FoR ) codes , and can also take the form of commissioned reports , the main subject of our attention here is Creative Arts and Writing ( now Field of Research code 36 , previously FoR19 ).
Approximately half of the outputs submitted in Creative Arts and Writing ( then FoR 19 ) in the ERA15 and ERA18 exercises were NTROs . They include works such as films , novels , short stories , poetry , paintings , photography , videos , dance and music
Production still from The Skin of Others , 2020 ( c ) Tarpaulin Productions Pty Ltd . A feature documentary by Tom Murray . Image : supplied .
compositions . The argument has been decades in the making , but there can be no doubt now that creative-arts based research methods and forms are capable of uncovering important new knowledge that contributes to scholarship across many fields of research ( for some historical context see Jenny Wilson , 2011 ).
In the case of recent work produced by CDRC members , these contributions cover fields including psychology , cognition , anthropology , geography , feminism and gender studies , Australian history , Indigenous studies , early childhood learning , the criminal justice system , and various creative-arts disciplines . This deeply researched work , sometimes taking nearly a decade to produce , has engaged large audiences in film festivals and through television broadcast screenings . Our members have also written major fiction and non-fiction works .
In each case these research outputs were funded ( in whole or part ) through the Australian Research Council , university post-doctoral research funds , and internal university research funds . Yet , due to ‘ legacy ’ understandings that currently underpin the idea of ‘ research ’, these works – when submitted during the ERA process – are evaluated as worth only a fraction of what an equivalent ‘ scholarly ’ book is worth .
Both the ERA18 Guidelines and ERA23 Draft Guidelines state that ‘ books have a weighting of 5:1 compared to other research outputs ’ ( ERA23 Guidelines , p . 12 ). By contrast , all ‘ original creative works ’ ( i . e . NTROs ), no matter the size , have a weighting of 1 point each . ( There are cases too where a portfolio of works is submitted together as a single output , attracting 1 point .)
Is this important ? After ERA18 , Canberra University scholars Jen Webb and Ross Gibson usefully reflected on the consistent ‘ lacklustre ’ ERA results in the creative arts ( NiTRO , 2019 ). While every other FoR had increased its average score in each ERA round , FoR19 ( now 36 ) was the only code in which the average score across the sector had decreased ( McKee 2018 ).
Webb and Gibson wondered why this was the case : Australian creative practiceled research is , after all , recognised as world-leading . They concluded that the underwhelming ERA results in this FoR were due to a failure of creative arts researchers to sufficiently address why their work constituted research ; that evaluation measures in ERA remain science focused (‘ espoused knowledge is more positively valued than a sensed understanding ’); and that members of the creative academic community are , perhaps , inordinately tough on one another .
We suggest that a further reason for these poor ERA results in the creative arts has to do with the way that creative arts outputs have been consistently ( and concretely ) under-valued as research , in both quantitative and qualitative terms . At the same time , evidence ( both UK and Australian data ) indicates that the numbers of NTROs being produced by universities are rising . It is therefore more urgent than ever that this inequity be understood and redressed .
How does this impact the Australian research landscape ? This inequity has the potential to significantly skew the status of ERA submissions in FoR codes , and in research institutions , where large-scale NTROs are a substantial element of the submission . NTROs currently appear in a number of two-digit FoRs : e . g . Anthropology or History , for example , might , as part of their submissions , include outputs such as a documentary podcast or an ethnographic film . Such outputs might solely fall under ‘ Anthropology ’ or ‘ History ’ because their knowledge-contributions sits squarely in these areas . In these two-digit FoRs , NTROs are well and truly in the minority . Not so in Creative Arts and Writing : in ERA2015 , 52 % of outputs were NTROs ; in ERA2018 , 49 % were NTROs ( data : ERA18 report ).
Currently in the ERA assessment process , there is a low volume threshold for submissions : for FoRs that are evaluated using peer review , the threshold is the equivalent of 50 weighted apportioned research outputs . ‘ If the number of
20