CMM predicted: PSU scores would not increase much by not penalizing for incorrect answers

CMM predicted: PSU scores would not increase much by not penalizing for incorrect answers

“Our estimations were quite correct. Scores did not increase extremely, as some experts predicted”, explained Jorge Amaya. This researcher of the Center for Mathematical Modeling (CMM) of Universidad de Chile, along with the academics Jaime San Martín and Nancy Lacourly, participated in the redesign of the University Selection Test (PSU, because of its acronym in Spanish) 2014, which had an unprecedented trait: for first time since 1966, one correct answer would not be deducted for every incorrect four ones.

The measure was controversial at the time and some people were against, because, presumably, scores would significantly increase. Nevertheless, this idea was refused by the study carried out by CMM researchers. San Martín explains that the rise in 15% of the highest score, from 220 in 2013 to 253 in 2014, is framed within the variations of the last years. On average, the quantity of people obtaining these points has been of 292 since 2004. “It is enough to have a question a bit easier or more difficult to get such a fluctuation”, highlighted the expert.

“We turned to CMM because it has top level academics “, pointed out Sergio Caruman, director of the Department of Evaluation, Measuring and Educational Register (Demre). With this entity, the CMM signed one year ago an agreement to predict the students´ reaction towards this change that has been introduced in the PSU since last year. Thus, the results of this study were successfully applied in the pretest and PSU of 2014, which predicted that the national scores would not increase significantly in spite to eliminate the penalization for incorrect answers.

CMM team, composed by Meeaime San Martín, Jorge MAmaya and Nancy Lacourly,

Meeting between the director of Demre, Sergio Caruman, the  director of CMM, Alejandro Jofre, and research team of this center, composed by Jaime San Martín, Jorge Amaya and Nancy Lacourly

 

The relation between Demre and the CMM arose out of the evaluation of the PSU that was made in 2013 by the British company Pearson commissioned by the Ministry of Education and the Council of University Presidents (Cruch). This study detected methodological problems in the structure and assessment of the test, for instance, in how the items were constructed. In order to redesign the test based on the recommendations, Demre asked for methodological support to certain research institutions of the Universidad de Chile.

One of the Pearson´ statements was to eliminate the penalization for incorrect answers. Indeed, psychologists of all over the world agree with this measure, as it is part of the learning process.

“Working with Demre is a challenge that we have taken with great interest. Not only the issues that we are facing are difficult from a theoretical point of view, but also they have a lot of impact in the society”, pointed out San Martín. He added that global trends do not penalize incorrect answers, due to psychological and theoretical reasons, as the omission introduces biases in the statistics of the qualities of the test.

CMM proposed to relate the difficulty of the questions, the performance of the students and the omission. It was relevant to study how the difficulty and the biserial coefficient vary once incorrect answers are not longer penalized. The first one is about the percentage of students who answers correctly an item. However, the second one measures the performance of the item in relation with the test entirely.

The future of the PSU

Not penalizing incorrect answers have been one of the first modifications introduced in the PSU out if the Pearson Report. In the next years, it is expected to bring the other changes.

CMM will continue the task started in 2014 thanks to a new agreement being signed and will consider further working lines. Particularly, the Center will support Demre in addressing more recommendations of the Pearson Report.

Consequently, CMM will contribute to the main aim of Demre, whose focus is to produce tests with the highest possible standards. “PSU is a radiography of the socio-economic reality in Chile and the education quality. The selection is not easy, specially if we take into consideration the economic inequality in the country”, concluded Caruman.

 

Photo Source frontpage: Wikimedia Commons

Posted on Mar 19, 2015 in Frontpage, News