Fleiss' kappa is a statistical method of evaluating agreement among three or more raters rating any number of items. For example, suppose Abigail, Bob and Charlie each rate three paintings, by Picasso, Rembrandt and Monet, on a scale from 1 to 5. Fleiss' kappa corrects for the amount of chance variation.

- Skill level:
- Challenging

### Other People Are Reading

## Instructions

- 1
Count, for each item, the number of times each rating was given. For example, how many times was the Rembrandt painting given a rating of 5? If r is the number or raters and q is the number of rating categories, you should have q*r computations.

- 2
Square these numbers.

- 3
Add up the squared numbers.

- 4
Subtract q

*r from the result. In the example, q*r is 15. This result will be the numerator in a fraction - 5
Multiply r

*q by the number of raters - 1. In the example 15*2 = 30. This is the denominator of the fraction. - 6
Divide the numerator by the denominator. This is Fleiss' kappa.

#### Tips and warnings

- You can get Fleiss kappa automatically in statistical programs such as R, SAS or SPSS.