TIMSS and PISA results: Seeing past the headlines

Education research does not often make newspaper headlines. Even less often does it make headlines in multiple countries around the world. In just a few weeks time we will see a rare exception.

In the space of a week, the latest set of results from both TIMSS (29th November) and PISA (6th December) – two international surveys that compare the performance of education systems around the world – will be published. The results are a closely guarded secret until then, but two things can be guaranteed: there will be newspaper headlines, and the headlines will be misleading.

I know there will be headlines, because I have just spent the week in Oslo at the General Assembly of the IEA – the organization responsible for TIMSS, together with the TIMSS & PIRLS International Study Center at Boston College. Not only have we had a preview of the results, but we have also spent much of the week discussing how to ensure the results are reported widely and accurately in the media, and are understood and applied by policymakers and the wider sector (my blog from the What Works Global Summit last month discusses exactly this topic).

And I know the headlines will be misleading. Most notably, they will focus on the rankings of countries and will fail to grasp the notion of statistical significance. A country’s ranking may change from year to year simply because the number of countries participating in the study has changed. Everyone knows the old joke about coming second in a beauty contest with only two contestants (rather a different achievement to coming second out of 50!)

And differences between countries, or changes over time, are not always indicative of fundamental differences/changes. In football they say the league table never lies. But at any given point in time, some teams will be punching above or below their weight (to mix metaphors!) simply because of an unusually good or bad run of form. Small differences are sometimes just the result of these blips.

NFER has been involved in the international studies for the past twenty years, so we’re familiar with the good, the bad and the ugly of how they’re reported. Over the coming weeks we will play our part in helping to explain the results and debunk the myths.

Despite the challenges, attending IEA’s General Assembly has emphasized two ways in which the value of international studies will be realized long after the initial flurry of headlines.

Firstly, the data generated goes far beyond measuring overall performance. Student and school questionnaires provide a rich source of information on students’ attitudes and experiences of school; teachers, professional development and school organization. The TIMSS encyclopedia (published in October 2016) also contains a wide range of comparative data on curricula, teacher training, school starting age, and more.

By comparing countries and tracking changes over time, key education issues can be explored in ways that are not possible with any other data source. Indeed, NFER is currently undertaking a project using international data to provide an important new perspective on social mobility in England.

Secondly, the international studies promote cooperation and learning between educators around the world. You will rarely find such a coming together of people from education ministries and research institutions from such a diverse set of countries. The conversations and relationships formed through networks such as the IEA enable genuine dialogue and learning to take place – not just an over-simplified policy borrowing from the top performing nations.

IEA’s strapline is ‘Researching education, improving learning’, which aligns well with NFER’s own ‘Evidence for Excellence in Education’. Those are two headlines I’d be happy to see appearing later this year.

Ben Durbin is Head of International Education at NFER. He can be contacted at b.durbin@nfer.ac.uk


One thought on “TIMSS and PISA results: Seeing past the headlines

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s