Evaluation of the new theoretical test
Date: 0000-00-00
The changes which were undertaken were evaluated and in the following section some of the strategies for evaluation as well as some findings are described. This evaluation of the test was designed to answer questions about the early experiences of the new test, matters of measurement and content, educational outcomes and criterion issues.
The first part of the evaluation was a statistical description of the test and the test results for the starting-period. It was based on all tests made during the fourth and the fifth weeks in 1990, altogether 3391 tests. The result showed a remarkable decline in the proportion that passed the new test, compared with the old one. On average 24 % of the examinees passed the test during this first period. About 90% passed the old test. One explanation for this dramatic decrease was that the new curriculum contained some new areas that, after a while, were supposed to be more familiar to the test takers than during the first period. As a consequence of the increased familiarity the pass/fail quota was also supposed to increase.
The results from the evaluation also indicated that some items should be changed in order to work properly. The difficulty of the six catalogues was not the same, defined by the proportion passing (18% - 31%), even though the average mean scores seemed to be about the same (26.0 - 27.3 out of 40). On the basis of these results and the comments from the local testings some items were also modified or replaced.
In May 1990, data were once again collected and now the proportion of examinees passing had increased to 53%. This increase continued for another two or three months before it reached the level it still has, i.e. about 70%. This implied that the test and the ideas of the new test and item format had become quite familiar both to the test takers and the teachers after about four months.
After the May collection of test data new item-analyses were conducted, all based on classical test theory. These analyses indicated that some items seemed to work better, probably because of the changes in the education and in the attitudes among the test takers, along with a growing familiarity with the new curriculum. Others did still not work satisfactorily and were, in many cases, replaced or changed as a result of the item analyses.
In order to describe different properties of the new test, different models were used. Since the new test was to some extent based on the old test these models for description can be called "theoretical reconstructions". Descriptions of the content in terms of the parts of the curriculum, the complex criterion structure and the separate cutoff scores, gave some indications about how different abilities were valued. The part named "Traffic Rules", for example, was obviously considered to be the most important part of the course, since 14 out of 40 items were related to this part and since the cutoff score was set to 79%. The part named "Knowledge About the Vehicle" was considered the least important with only three items and the cutoff score set to 33%.
This kind of information is considered to be important when the ambition is to maintain the standards already set or even to improve them. In theory, the test is considered as a criterion- referenced test, where the demands are defined by the specifications made in the curriculum and where the cutoff limits define the required competence level. Most items will be correctly answered by nearly all examinees if they are well-educated. This also means that the classical statistical methods used for evaluating a single item will, in most cases, be too blunt. Other methods have to be used in order to get valid differences between uneducated and educated people in this specific content area.
Return to the main article: "The driving license examination in Sweden"
Comments to this article have been disabled.
