Serious flaws in how PISA measured student behaviour( continued)
ARTICLES
Serious flaws in how PISA measured student behaviour( continued)
Then, for each of the five statements, students had to tick one of the boxes on a four-point scale from( a) never or hardly ever;( b) in some lessons;( c) in most lessons; and( d) in all lessons.
Problems with the PISA process and interpretation of data
Even before we look at what is done with the results of the questions posed in PISA about classroom discipline, alarm bells would be ringing for many educators reading this blog.
No rationale for what is a good classroom environment
For a start, the five statements listed above are based on some unexplained pedagogical assumptions. They imply that a‘ disciplined’ classroom environment is one that is quiet and teacher directed, but there is no rationale provided for why such a view has been adopted. Nor is it explained why the five features of such an environment have been selected above other possible features. They are simply named as the arbiters of‘ disciplinary climate’ in schools.
Problem of possible interpretation
However, let’ s accept for the moment that the five statements represent a contemporary view of classroom disciplinary climate. The next problem is one of interpretation. Is it not possible that students from across 72 countries might understand some of these statements differently? Might it not be that the diversity of languages and cultures of so many countries produces some varying interpretations of what is meant by the statements, for example that:
• for some students,‘ don’ t listen to what the teacher says’, might mean‘ I don’ t listen’ or for others‘ they don’ t listen’; or that students have completely different interpretations of‘ not listening’;
• what constitutes‘ noise and disorder’ in one context / culture might differ from another;
• for different students, a teacher‘ waiting a long time’ for quiet might vary from 10 seconds to 10 minutes;
•‘ students cannot work well’ might be interpreted by some as‘ I cannot work well’ and by others as‘ they cannot work well’; or that some interpret‘ work well’ to refer to the quality of work rather than the capacity to undertake that work; and so on.
These possible difficulties appear not to trouble the designers. From this point on, certainty enters the equation.
Statisticians standardise the questionable data gathered
The five questionnaire items are inverted and standardised with a mean of 0 and a standard deviation of 1, to define the index of disciplinary climate in science classes. Students’ views on how conducive classrooms are to learning are then combined to develop a composite index – a measurement of the disciplinary climate in their schools. Positive values on this index indicate more positive levels of disciplinary climate in science classes.
Once combined, the next step is to construct a table purporting to show the disciplinary climate in the science classes of 15 year olds in each country. The table comprises an alphabetical list of countries, with the mean index score listed alongside each country, so allowing for easy comparison. This is followed by a series of tables containing overall disciplinary climate scores broken down by each of the disciplinary‘ problems’, correlated with such factors as performance in the PISA Science test, schools and students socio-economic profile, type of school( eg public or private), location( urban or rural) and so on.
ACER reports the results‘ from an Australian perspective’
The ACER report summarises these research findings from an Australian perspective. First, it compares Australia’ s‘ mean disciplinary climate index score’ to selected comparison cities / countries such as Hong Kong, Singapore, Japan, and Finland. It reports that:
Students in Japan had the highest levels of positive disciplinary climate in science classes with a mean index score of 0.83, followed by students in Hong Kong( China)( mean index score: 0.35). Students in Australia and New Zealand reported the lowest levels of positive disciplinary climate in their science classes with mean index scores of- 0.19 and- 0.15 respectively, which were significantly lower than the OECD average of 0.00( Thomson, Bortoli and Underwood, 2017, p. 277).
Then the ACER report compares scores within Australia by State and Territory; by‘ disciplinary problem’; and by socio-economic background. The report concludes that:
Even in the more advantaged schools, almost one third of students reported that in most or every lesson students don’ t listen to what the teacher says. One third of students in more advantaged schools and one half of the students in lower socioeconomic schools also reported that there is noise and disorder in the classroom( Thomson et al, 2017, p. 280).
What can we make of this research?
You will note from the description above, that there would need to be a number of caveats placed on the research outcomes. First, the data relate to a quite specific student cohort who are 15 years old of age, and are based only on science classes. That is, the research findings cannot be used to generalise about other subjects in the same year level, let alone about primary and / or secondary schooling.
31 SCIENCE EDUCATIONAL NEWS VOL 66 NO 4