1.

INTRODUCTION

The main defects of subjective test result evaluation are: different raters often get different results; The test results depend on the difficulty of the test questions, because the questions with different difficulty are not equivalent; Lack of scientific methods to evaluate errors. Therefore, using scores to evaluate subjective tests is not very scientific. This paper introduces a method of objectively evaluating students’ achievements, which can overcome the above defects.

2.

SCORING OF SUBJECTIVE QUESTIONS

An example is discussed below¹.

Solving the equations:

Because the examinee’s responses at each step have two results: right or wrong, the examinee’s response at each step is a random event.

Suppose; B : x < 0 or x > 1; , through the analysis of the solution process of case 1, it is not difficult to find that no matter whether the subject’s answer to equation (1) is right or wrong, it will not affect the subject’s answer to equation (2). In other words, regardless of whether the examinee’s answer to equation (2) is right or wrong, it will not affect the examinee’s answer to Inequality (1). It can be seen that event A and event B are independent of each other. However, only after event A and event B occur at the same time, event C may occur. If event A or event B does not occur, event C must not occur, thus (A ∩ B) ⊃ C.

Through the above analysis, it is not difficult to find that the three events A, B, and C satisfy the following relationships: (1) event A, B are independent; 2) (A ∩ B) ⊃ C. According to the relationship between A, B, C, the following results may appear in the process of problem solving¹:

Assume that , the Ω contains all possible outcomes of the solution of case 1. Suppose that X = {0,1,2,3,4}, and define a 1 to 1 mapping function f : Ω → X, the function is specified such that each “score” in X corresponds to an event in the same position in Ω.

Through function f, events that may occur during problem solving are linked to a set of numbers², this number set X can be used as the scoring step of the quiz item. In other words, case 1 can be divided into 4 different scoring steps, each step is scored as 1 point, 2 points, 3 points, 4 points. If the scenarios in which the subjects scored 0 are also counted, then case 1 can be divided into 5 different scoring levels. Its advantage is that the test score of the subjects has nothing to do with the reviewers. For the same topic, different reviewers can also score the same score, avoiding the error caused by the reviewers’ personal reasons.

3.

INVARIANTS IN THE EXAMINATION

In the test T. Let X₁,X₂,…,X_N denote the scores (true scores) of N examinee’s on test T, respectively, X_i represents the score of the i-th subject in the test, N_i represents the total number of test scores not greater than X_i, and represents the percentile of the examinee i in the test. When N → ∞, the limit value of P_i is the percentile of the examinee in the overall test³.

It is further assumed that the test T will preserve the order of the examiners, the meaning of preserve the order of the examiners can be explained that: suppose T₁, T₂ are any two tests in the same attribute set, A, B are any two examiners, the scores of examiner A on T₁, T₂ are X₁ and X₂ respectively, examiner B on T₁, T₂ are Y₁ and Y₂, then the corresponding relation of X₂, Y₂ is as follows: (1) ; (2) ; (3) . Under the above assumptions, percentile P_i is an invariant parameter⁴.

Let , θ_i ∈ R. From the definition of θ_і, we can know that θ_і is uniquely determined by P_i. Because P_i is an invariant parameter, θ_і is also an invariant parameter⁵. It is easy to know from the knowledge of mathematical statistics, the estimation of ability parameter θ has the following large sample properties: (1) The ability parameter θ is consistent estimation. (2) The ability parameter θ obey the normal distribution N(0, 1).

4.

INVARIANT FRACTION

Now we discuss how to convert the ability parameter into people’s habitual “score”⁶. Let because θ ~ N(0, 1), according to the nature of normal distribution, it is not difficult to get . Due to θ ~ N(0, 1), from the nature of normal distribution, the probability of falling within the interval (–2.5,2.5) is slightly 0.995, the probability of falling within interval (–∞,–2.5) or (2.5,+∞) is about 0.05. Utilize the property, θ = –2.5 or θ = 2.5 can be approximately regarded as an infinity point. When h = –2.5, that is, θ = –2.5 regarded as the zero point of X. Since the length of interval (–2.5,2.5) is 5, if the test score is set to 100 points, can be taken. Take , so under the above assumption, X ~ N(50,400), and the value range of X is [0,100]. Here X is called the ability score. The ability score obtained through the above method conversion also has all the properties of the ability parameter. Specifically, ability scores have the following characteristics⁷.

5.

APPLICATION EXAMPLES

As an application of the above method, we will discuss an example below¹.

Table 1 is the estimated value of item parameters obtained from a practical test. There are 24 test questions in this test, including 8 selection questions, 8 filling questions, and 8 multi-level scoring questions. A total of 1226 participants participated in the test. The average score of the test is 60, and the standard deviation is 24.44.

Table 1.

Parameter estimates for a test item.

From Figure 1, we can see that in this test, the distribution of test scores showed a gentle upward trend, with a certain proportion of examinee in each score range. This shows that the distribution of examinee with high or low scores in the test is relatively uniform, and the proportion of examinee with high scores is relatively large. Among them, examinee with scores higher than 60 accounted for 53% of the total, and examinee with scores higher than 80 accounted for 26.6% of the total. In the range of 80 to 90, the proportion of examinee reached the peak. On the whole, the distribution of scores showed a negative skew trend.

Figure 1.

The distribution of test scores for 1226 candidates in the test.

From the distribution of ability scores shown in Figure 2, we can see that there is a great difference between the distribution of ability scores and test scores. Also, we can see that examinees with scores above 60 account for 24.6% of the total, which is 27.4% points lower than 53% of the test scores. The examinees who scored more than 80 accounted for 2.6% of the total. The difference from the test score of 26.6% is 24% points. Its peak value appears in the range of 50 to 60 points, and the distribution of ability score shows a significant normal distribution trend, with a variance of about 400.

Figure 2.

The ability score distribution diagram of 1226 examinees after converting the test score of each subject into ability score according to the method described in this paper.

It is worth noting that the distribution of test scores is rarely normally distributed, because test scores are strictly dependent on the difficulty of the test, and the distribution of test scores is not the same for different difficulty tests.

Because the ability score has nothing to do with the difficulty of the test, it is always normal in the case of large sample random sampling¹⁰.

The ability score not only has the intuitive and easy to understand advantages of the test score, but also retains all the large sample properties of the ability parameters. Therefore, the ability score is a very ideal scoring method.

6.

TESTING ERROR

In the classical test theory, the test error is mainly described by reliability and test standard error. However, reliability is a general and rough indicator, which only roughly reflects the average difference between the real score and the test score of the subjects, and cannot reflect the test error of the examinees at different abilities. Therefore, reliability and test standard error are not a good way to characterize the test error. In this paper, we will use another method to characterize the test error.

Assuming that a test is composed of n items, the ability score estimates of subjects with ability score X₀ on n items are X₁, X₂, L, X_n, is the average score, according to mathematical statistics, when n → ∞, , let , Z ~ N(0, 1) can be obtained. Let confidence level α = 0.05, can be obtained after calculation, therefore, in the 95% probability meaning, , that is . So the 95% confidence interval is .Using the above formula, the confidence interval of the estimated ability score can be obtained. The confidence interval for the estimated capacity score shown in Figure 1 is shown in Figure 3.

Figure 3.

The confidence interval of the estimated capacity score.

Note: The abscissa represents the estimated value of ability score, and the ordinate represents the true value of ability score. The lower curve represents the left-end curve of the confidence interval, and the upper curve represents the right-end curve of the confidence interval.

Figure 3 is obtained as follows¹¹:

In this case, due to the small number of samples in each ability interval, the error between the standard deviation of the sample S and the overall standard deviation of the interval is large, so the error between the confidence interval is also large. If you can guarantee enough samples in ability interval, the resulting confidence interval will be more accurate.

From the strip region of Figure 3, we can see that in this test, for the examinees whose ability scores are between 50 and 70, the estimation error is the smallest. It is noted that the difficulty of most examinees in this test is between 0.5 and 0.7, while the test error is the smallest between 50 and 70, indicating that the test error is closely related to the difficulty of test items.

In addition, from the expression of the left and right endpoints of the confidence interval, we can see that when the standard deviation is constant, the length of the confidence interval will become shorter with the increase of the number of test items n. When n approaches infinity, the length of the confidence interval will approach zero. This shows that appropriate increase in the number of test items can effectively improve the test accuracy. Thus, the confidence interval can not only better reflect the test error at different abilities, but also better reflect the relationship between the test error and the number of test items¹.

In the classical measurement theory, the estimation of measurement error is mainly described by reliability and test standard error. From the approximate calculation formula of reliability , the reliability of this test can be calculated as r = 0.85, and the test standard error is .

Compared with the confidence interval, the method of using test reliability or test standard error to describe test error has the following defects: (1) Test reliability or test standard error only reflects the average difference between the test scores and the real scores of all examinees in the test. In this case, which is too rough and general, and is far less detailed than the confidence interval. (2) Test reliability or test standard error is based on strictly parallel test. In practice, strictly parallel test does not exist, so the practical significance of test reliability or test standard error is greatly reduced. (3) Test reliability or test standard error does not reflect the relationship between test number (even if strictly parallel tests do exist) and test item number and test error. That is to say, even if strictly parallel tests do exist, test reliability or test standard error basically remains unchanged no matter how many tests are conducted or how many test items are added. Therefore, test reliability or test standard error is a rigid indicator.

Due to the above defects in test reliability or test standard error, it is not a good way to describe test error with test reliability and test standard error. Relatively speaking, it is a more scientific and reliable method to describe test error with confidence interval.

7.

CONCLUSION

The characteristic of this paper is to give a new evaluation method under the guidance of item response theory (IRT). Since the ability parameter or ability score defined according to the method described in this paper is a constant parameter, in the case of large sample random sampling, the ability scores from different tests can be directly compared. For example, for the college entrance examination scores from different years, it can be directly compared in the sense of ability score, and the error is also controllable. Its popularization and application are of great significance to improve the scientificity and reliability of various tests.