19 of 45
Lesson 1:
Outline
|
Notes
|
Glossary
|
Presentation
|
Activities
|
Directed Questions
|
Assessment
Activities for Reliability
Multiple Choice
Attention: ONLINE RESPONDING IS DISABLED
Javascripting is either turned off or is not recognized by your browser.
1.
For this activity you will need to analyze some classroom assessment data and judge the inter-rater reliability of the exam. A 12-item test contained some supply items that had more than one correct answer. As these items are subjectively scored, there may be some concern about inter-rater reliability. Imagine that two teachers both graded the same five tests independently following the same scoring rubric. The scores they assigned the five students are shown.
Student
Mr.Dixon
Ms.Valdez
Vicki
9
8
Tony
11
11
Billy
8
8
Jasmine
11
12
Fernando
10
10
What is the inter-rater reliability as expressed by percentage of exact agreement for this test?
example:
3/5 or 60%
2.
How would you interpret this level of inter-rater reliability?
example:
Not great. You aim for at least 80% agreement.
3.
Mr. Dixon and Ms. Valdez scored 20 more students tests using the same scoring rubric. The inter-rater correlation was 0.92. How would you interpret this level of inter-rater reliability?
example:
Very good. The value of 0.92 shows a strong, positive correlation.
4.
Overall, what, if any, can you conclude about the reliability and validity of the test?
example:
The reliability of the test is good when you include more students. Nothing can be stated about the validity of the test.
Principles of Measurement
