‘The Idea is to Produce Excellent Researchers’

The International Laboratory for Educational Policy Research has been working at the HSE for more than a year. Its academic supervisor, Stanford University professor Martin Carnoy told us about the results of this year’s research, their forthcoming plans and the future of the laboratory.

— Professor Carnoy, how would you define the key research goals of your laboratory?

— Our main objective is to do something that is not in the Russian tradition, at least, in the more recent past. Which is to carry out real policy research in education. Education is a big sector which has usually simply been thought of this as a social service, a state investment – and it is true – but with very general goals. Since a lot of money is spent on education, the new emphasis in the world is whether it can be spent better. The second global question is whether we can improve the quality of the services being delivered. We are also interested in pursuing other goals, such as better equity, or better job prospects for the very best students. It depends on what their goals are, but this requires empirical data to show which direction to move in. And if you pursue certain goals, you can reach those goals.

The methodology of doing this kind of research has advanced tremendously in the past 20 years, so it’s only recently that really good work has been done. Education is a very difficult topic to analyze, because the output is unclear. In an automobile factory, you know more or less what a good automobile is, so you can measure the output. In education there are many outputs, and this is a difficult thing to measure. The education process is quite complicated and depends on many factors, such as student motivation, teacher quality, different methods of teaching and so on.

Here in the laboratory we are just beginning to approach this topic systematically. The data in Russia, by the way, is better than in many countries now, in part because the HSE is making an effort to collect such data. In many countries there is very little data, and you have to collect it yourself. This is also the case here: most of the data has to be collected in some form. There are international tests, and Russia participates in those tests, and we are analyzing both the PISA (Programme for International Student Assessment) data and the TIMSS (Trends in International Mathematics and Science Study) data. They are not the best data to use for this kind of analysis, but we are doing our best and I think we are making a lot of progress.

— What goals do you see as your nearest tasks?

— There are two kinds of progress we want to make. Firstly, we want to train excellent researchers who really know what good research is, and secondly, we want to produce results that are helpful for policy. Those are our two objectives.

I cannot say we started from zero last year, because some very good data had already been collected. But in terms of analyzing this data and producing world-class results we are just finishing the first stage. We have produced a couple of very good papers. We still have some work to do on them, but over the next two or three months days we will try to put them into a condition where we can publish them in respected international journals. There are a number of other projects that we are working on, from this and other data which will hopefully also produce very good papers.

Then we are going to take the results of those papers and try to put them into short policy briefs. They are just a few pages which summarize the main topics of research so that they can be taken and be used to influence policy.

— How is the work in the laboratory organized? How long do you usually come here for?

— I generally come for 10-14 days at a time. Actually, this is better, because what happens is that we work very intensively with people on their papers. Ultimately, they have to learn how to use this interaction to raise their own game. I’m not going to take them to the gym and help them do their exercises – I’m going to show them the exercises they need to do. The idea is to produce researchers who are independent, who ultimately don’t need me.

— What projects are currently underway in the laboratory?

— There are quite a few of them.

In Russia one interesting thing is that until now we knew very little about the process of what produces higher achievement in schools. There is a very good paper by Andrey Zakharov, who took a sample of schools in three regions and received very strong results showing some obvious things and some more surprising things about improving the results of Russian language and mathematics in university entrance exams. We are doing a similar project using TIMSS data, but the TIMSS data is not as good as the data Andrey has got. Even so, we have some interesting results there.

We are trying to figure out why Russian students do worse on the PISA tests than they do on the TIMSS test. The TIMSS test is more like a Russian test, compared to the PISA test. So the question is, is it the test? Is it that they are not used to taking that kind of tests, so they don’t take it very seriously? It could be: they are 15 years old and the test has no meaning or relevance to them, so there are lots of reasons. We are still trying to figure this out.

There are lots of problems with international tests in general. We are working on a very interesting paper comparing the results for Eastern European countries and some Western European countries with Russia and trying to understand what’s going on. This is not a statistically very sophisticated paper, but, in a way, it is a very interesting paper because it will ask a very important question: how seriously should we take these results as a measure of student knowledge in Russia?

We are also working on some papers about teacher expectations and their effect on student results. We have found that teachers overestimate the results for students from higher social classes, and they underestimate how well students from lower social classes will do. If they genuinely believe that, does that affect how well the students really do? If your teacher does not believe you are going to score very highly, that might affect how hard you stud. We are looking at the tutoring issue – whether tutoring actually improves your score on the test that gets you into a university, does it really help? We have a very good statistical method for doing that.

There is a very large survey, which is now on its second round, on student progress through school. We’ll find out more about which students leave school, why they leave school, etc. We are also going to do more work on effective teachers using that survey.

We have a very interesting paper using the PISA data on how much a year of schooling is worth in different countries. It appears that in Russia an extra year of schooling is actually worth a lot less than in Czech Republic or Slovak Republic, about the same as in Hungary. We studied the 9^th-10^th year of school, and our research says something about the efficiency of schooling at that level. You can begin to tell why, for example, the Hungarians start out much higher in the 9^th grade and then add very little in the 10^th grade. The Russians don’t start out that high in the 9^th grade and add very little in the 10^th grade. So it may very well be that one of the reasons that Russians don’t score so highly in these international tests is because their schooling does not really add that much in a year. I think that this has important implications: why doesn’t it add that much during the year?

We are also working on a paper on whether there are certain parts of the entrance exam for university, the USE exam, which actually predict student performance in the university, and other parts which are very bad predictors. So maybe we only have to look at one or two parts, or maybe we only have to look at one, so you don’t need to have the entire exam.

We are going to begin a project now that is trying to measure how much it costs to provide schooling in Russia. Believe it or not, it’s not very clear. And we don’t know much about the variation between regions.

One of our most important projects is a book in which the HSE is one of four participants. We have been working on it for the past three years, and this year this book is coming out through The Stanford Press. The book compares universities in the BRIC countries. And we will publish probably four or five papers from this book in international journals. It’s a great book, and the Russian part is really important. We had a big conference in Stanford at the end of April about this book and people from all over the world could sign up and watch it online on the web.

There is also another project that is looking at how different public universities are responding to the decline in average test scores of people entering university. The young population is declining, so a higher and higher percentage of young people are going to universities. The variation is increasing in who goes to university, and then the question follows, do universities even notice this? Are some universities having trouble actually recruiting students? How are they responding? We are carrying out a whole project on how the universities are responding to this new situation.

— A year has passed since you became head of the laboratory. What have you achieved, and has it been worth it?

— You have to be patient, because I think the expectations were that we would suddenly start producing world-class papers in one year. In fact, we have produced a couple of really good papers, that’s true. But it is like a very big freight train: it takes a while to get moving, but once it gets moving, it’s very hard to stop. Right now we are just trying to get the freight train moving. We are also doing a lot of other things, for example, we are teaching methodology courses to the students. But Rome wasn’t built in a day. It’s not even a three-year process, it’s really a ten-year process. You have to lift the whole standard and, as I said, you have to get people to be able to work on their own.

We can produce papers, but I think they are not the measure. It’s like teaching the test. You can certainly improve test scores by teaching the test, but that doesn’t mean you increase knowledge. And we can do that, we can get those scores up, we can produce some papers. But then we go away and all of a sudden the papers go away. That’s not the idea. The idea is to produce excellent researchers. We have very smart people here working on this, and we want to turn them into people who can really conduct their own research. The number of papers as a measure of success is a very Soviet view of what it means to be good. But it didn’t work well then and it doesn’t work well now.

— Does the laboratory need ‘fresh blood’?

— We are looking for students. We have people who need students to work on their projects, so if anyone wants to join us, they just need to get in touch with the laboratory.

Oleg Seregin, HSE News Service