Test-item Analysis
Create and analyse tests with the Authoring ToolGetting started with Edu-Suite
Edu-Suite’s Authoring tool offers the possibility to create various types of tests and run extensive analyses of test items.
Test item analysis (tia) is an ideal tool for test developers, publishers and authors. It aims to provide a clear understanding of the reliability, validity and distinctiveness of tests, the questions on the test and the theoretical knowledge needed to pass the test successfully.
Below, we explain in more detail the ins and outs of analysing tests and test items using Edu-Suite’s Authoring tool.
Ins & outs of Edu-Suite’s test-item analysis
Varied test
By using the interaction types and question types in Edu-Suite’s Authoring tool, you can create varied and high-quality tests. This can be a one-off, or you can create a test matrix to have your test questions converted to a test on-demand.
Quality-check
With the test item analysis tool, you can check the quality of the test questions based on a snapcheck.
Method-level analysis
The test-item analysis in Edu-Suite functions on a method-level basis. In this, the results of all students who completed the test are analysed within a set of predefined parameters. Several variables can be examined in the tia, for example:
- Power of differentiation of a question: to what degree it separates good students from weaker ones (rit-value and rir-value)
- Difficulty of the question (p-value) (the easier the question, the lower the p-value)
- Attraction of the distractor (a-value)
- Differentiation ability of a question (rar-value)
The results of the tia are presented in a transparent chart where the different values for each item are listed and poor items or those requiring attention are marked by colour. Red (insufficient score), Orange (moderate score) and Green (sufficient score). This allows you to immediately see where the focus areas and improvements are.
Improving with a tia
Om een tia uit te kunnen voeren zijn minstens 50 afnames van een toets nodig, bij voorkeur meer dan 100. De resultaten van de tia kun je gebruiken om de toets-vragen in een toets te verbeteren door:
- Modifying a lead-off question
- Making the questions more specific
- Correcting a key error
- Removing a poor question from the test
Test-item analysis: explanatory list of terms
Number: Number of elaborations of the test task within the analysis, the number of students who chose the multiple-choice option.
Answer/Deduction: Indicates whether the choice is the correct answer or a deducer.
Discrimination: The discrimination index is the difference between the average item scores of the top 33% and the bottom 33% of students. It is desired that the discrimination is above 0.3.
Frequency (%): Indicates the percentage of students who chose the multiple-choice option.
Average test score: The average test score of the students who chose the multiple-choice option.
Item: The sequential number of the test task.
Item Alpha: Alpha says something about the reliability of the test. A re-measurement records (approximately) the same result. Alpha indicates the level of reliability for the item concerned. The higher the better.
Item Alpha rest: Alpha rest is the reliability of a test minus the item in question. So if an item is removed from a test, the Alpha rest is the reliability of the remaining test.
Choice Sequence: The sequence number of the multiple-choice option.
Choice: The multiple-choice option.
P-Value: The percentage of students who answered this item correctly. Desired P-value is between 0.3 and 0.7. If the P-value is below 0.3, the question is too difficult, and if the P-value is above 0.7, the question is too easy.
Q1 to Q4 (quartiles): Group is divided into 4 quartiles, best scoring group in Q4 (25%) and only scoring is Q1 (25%) Related to P-value
Rar: Detractor-rest correlate. Indicates how strongly choosing this answer correlates with the score on the test. Desired: positive for correct answers, negative for distractors.
RIR: The RIR is a similar index to the RIT. The RIT is about the correlation between a question and the whole test, including the question itself, the RIR is about the correlation of the question and the rest of the test, that is the whole test minus the question in question
RIT: The RIT value is the correlation between the score of a task and the total score of the test (including the task in question) The RIT indicates the extent to which a task has succeeded in differentiating between ‘good’ and ‘poor’ students. By “good” we mean students who achieved a high test score and by poor we mean a low test score. The RIT is a discrimination index. A high means that many candidates with a high test score answered the question correctly and many candidates with a low test score answered the question incorrectly. Furthermore, a high RIT means that the question contributes relatively much to the reliability of the test. To put it another way, a high RIT means that a question is a representative part of the overall test.
SD: Standard Deviation is a measurement of the variance of scores within a task. The higher the SD value; the greater the variance and, in general, the better the quality of the task.
TIA: test-item analysis
Interested in Edu-Suite’s CMS and Authoring tools?
Want to know more about what our advanced CMS can do for your business?
Give us a call or send an e-mail and we’ll help you find the solution that suits your needs.