Fleiss' kappa is a statistical method of evaluating agreement among three or more raters rating any number of items. For example, suppose Abigail, Bob and Charlie each rate three paintings, by Picasso, Rembrandt and Monet, on a scale from 1 to 5. Fleiss' kappa corrects for the amount of chance variation.
- Skill level:
Other People Are Reading
Count, for each item, the number of times each rating was given. For example, how many times was the Rembrandt painting given a rating of 5? If r is the number or raters and q is the number of rating categories, you should have q*r computations.
Square these numbers.
Add up the squared numbers.
Subtract qr from the result. In the example, qr is 15. This result will be the numerator in a fraction
Multiply rq by the number of raters - 1. In the example 152 = 30. This is the denominator of the fraction.
Divide the numerator by the denominator. This is Fleiss' kappa.
Tips and warnings
- You can get Fleiss kappa automatically in statistical programs such as R, SAS or SPSS.
- 20 of the funniest online reviews ever
- 14 Biggest lies people tell in online dating sites
- Hilarious things Google thinks you're trying to search for