Impact of Education Quality Research on Policy Is Not as Significant as You Might Think

© Mikhail Dmitriev/ HSE

At a seminar held at HSE as part of the Days of the International Academy of Education in Moscow, Professor Gustavo E. Fishman (University of Arizona) likened international comparative studies of education quality to horse racing and discussed how these studies do not have as significant an impact on educational policy as is commonly believed.

In his opening remarks to the seminar, Andrey Zakharov, Head of HSE’s International Laboratory for Education Policy Analysis, noted that the results of comparative studies on the quality of education garner a lot of interest from politicians and the general public. Low results are a cause for concern. For example, in the 1990s, Germany and Poland dealt with so-called PISA-shock—their results in the comparative PISA study (an assessment of the functional literacy of 15-year-old students) were lower than expected. Japan experienced a similar situation when the results of another study—the TIMSS (an assessment of student preparedness in grades 4 and 8 in math and science)—were published. The question then arises: what we should change in the education system in order to improve it?

This question, albeit belatedly, has become a timely one for Russia, which has been participating in these studies since the 1990s, and whose results have now become a topic of active discussion. Most concerning among the results is Russian students’ less than satisfactory performance in PISA. However, this is not cause to conclude that these studies have had significant influence on Russian education policy. ‘Only some localized changes can be observed,’ Andrey Zakharov said.

In order to improve the quality of education in mathematics, statistics was added to the Russian school curriculum

Gustavo E. Fishman likened countries’ participation in international studies to horse racing, wherein contestants do not even know who they are competing with. It is believed that a country should be ahead of the curve, so a lot of time and money is spent on test preparation and assessment. At the same time, says the researcher, comparative studies were conceived in the 1960s without competition in mind, but for purposes of education quality assessment, and there was not as much interest in them. Now more than 140 countries participate in these studies which are growing in number, conducted by a wider variety of influential organizations (such as UNESCO, OECD and others), and yielding a larger amount of publications.

Gustavo E. Fishman

Professor at the University of Arizona

‘A new industry has appeared—an entire economic sector with professionals working in the area of international education quality studies. Even if you are not engaged in their realization and development, read about them, publish articles, and participate in the academic process.’

At the same time, the researcher believes, this entire industry exists on its own and has little impact on education policy. Comparative studies do not provide tools for improving the quality of education. But some people take dishonest measures to improve results in order to look better than their competitors.

In some countries, only the best students are selected to participate in the international testing

And this is not the only undesirable consequence.

An analysis of the educational policies of several countries participating in international studies showed that after the publication of their results, they implemented diametrically opposed changes. In one case, they centralized decision-making; in another, they decided to privatize schools; in a third, it was decided to increase state support for education and ensure that teachers meet high standards; and in yet another, it was decided to weaken state support of schools. ‘The studies were the same, but the resulting policies were different,’ said the researcher.

If a country performs poorly in an international study, the teachers are blamed, and the most common course of action is to fire the bad teachers.

In the media, teachers are usually criticized, Gustavo E. Fishman noted. This is at odds with the data we have from opinion polls about the role of teachers in people’s lives. In these surveys, only a small portion of respondents noted being dissatisfied with their teachers.

‘Bad’ teachers in schools

25-40%

according to media data

5%

according to survey respondents

 

Fishman’s main conclusion? Thanks to comparative studies of the quality of education, we have a wealth of interesting and informative data, the application of which remains an open question. Politicians use this data to justify their actions, but if countries demonstrating similar results in these studies implement opposite policies, can we trust their justifications? With every year, the assessment system becomes more perfected. But it does not help us come up with solutions to the problems that arise from it. Nor does it help us identify the problems, Fishman says.