Serious flaws in how PISA measured student behaviour ( continued )
ARTICLES
Serious flaws in how PISA measured student behaviour ( continued )
Then , for each of the five statements , students had to tick one of the boxes on a four-point scale from ( a ) never or hardly ever ; ( b ) in some lessons ; ( c ) in most lessons ; and ( d ) in all lessons .
Problems with the PISA process and interpretation of data
Even before we look at what is done with the results of the questions posed in PISA about classroom discipline , alarm bells would be ringing for many educators reading this blog .
No rationale for what is a good classroom environment
For a start , the five statements listed above are based on some unexplained pedagogical assumptions . They imply that a ‘ disciplined ’ classroom environment is one that is quiet and teacher directed , but there is no rationale provided for why such a view has been adopted . Nor is it explained why the five features of such an environment have been selected above other possible features . They are simply named as the arbiters of ‘ disciplinary climate ’ in schools .
Problem of possible interpretation
However , let ’ s accept for the moment that the five statements represent a contemporary view of classroom disciplinary climate . The next problem is one of interpretation . Is it not possible that students from across 72 countries might understand some of these statements differently ? Might it not be that the diversity of languages and cultures of so many countries produces some varying interpretations of what is meant by the statements , for example that :
• for some students , ‘ don ’ t listen to what the teacher says ’, might mean ‘ I don ’ t listen ’ or for others ‘ they don ’ t listen ’; or that students have completely different interpretations of ‘ not listening ’;
• what constitutes ‘ noise and disorder ’ in one context / culture might differ from another ;
• for different students , a teacher ‘ waiting a long time ’ for quiet might vary from 10 seconds to 10 minutes ;
• ‘ students cannot work well ’ might be interpreted by some as ‘ I cannot work well ’ and by others as ‘ they cannot work well ’; or that some interpret ‘ work well ’ to refer to the quality of work rather than the capacity to undertake that work ; and so on .
These possible difficulties appear not to trouble the designers . From this point on , certainty enters the equation .
Statisticians standardise the questionable data gathered
The five questionnaire items are inverted and standardised with a mean of 0 and a standard deviation of 1 , to define the index of disciplinary climate in science classes . Students ’ views on how conducive classrooms are to learning are then combined to develop a composite index – a measurement of the disciplinary climate in their schools . Positive values on this index indicate more positive levels of disciplinary climate in science classes .
Once combined , the next step is to construct a table purporting to show the disciplinary climate in the science classes of 15 year olds in each country . The table comprises an alphabetical list of countries , with the mean index score listed alongside each country , so allowing for easy comparison . This is followed by a series of tables containing overall disciplinary climate scores broken down by each of the disciplinary ‘ problems ’, correlated with such factors as performance in the PISA Science test , schools and students socio-economic profile , type of school ( eg public or private ), location ( urban or rural ) and so on .
ACER reports the results ‘ from an Australian perspective ’
The ACER report summarises these research findings from an Australian perspective . First , it compares Australia ’ s ‘ mean disciplinary climate index score ’ to selected comparison cities / countries such as Hong Kong , Singapore , Japan , and Finland . It reports that :
Students in Japan had the highest levels of positive disciplinary climate in science classes with a mean index score of 0.83 , followed by students in Hong Kong ( China ) ( mean index score : 0.35 ). Students in Australia and New Zealand reported the lowest levels of positive disciplinary climate in their science classes with mean index scores of - 0.19 and - 0.15 respectively , which were significantly lower than the OECD average of 0.00 ( Thomson , Bortoli and Underwood , 2017 , p . 277 ).
Then the ACER report compares scores within Australia by State and Territory ; by ‘ disciplinary problem ’; and by socio-economic background . The report concludes that :
Even in the more advantaged schools , almost one third of students reported that in most or every lesson students don ’ t listen to what the teacher says . One third of students in more advantaged schools and one half of the students in lower socioeconomic schools also reported that there is noise and disorder in the classroom ( Thomson et al , 2017 , p . 280 ).
What can we make of this research ?
You will note from the description above , that there would need to be a number of caveats placed on the research outcomes . First , the data relate to a quite specific student cohort who are 15 years old of age , and are based only on science classes . That is , the research findings cannot be used to generalise about other subjects in the same year level , let alone about primary and / or secondary schooling .
31 SCIENCE EDUCATIONAL NEWS VOL 66 NO 4