How to Calculate Fleiss' Kappa

Written by peter flom
• Share
• Tweet
• Share
• Email

Fleiss' kappa is a statistical method of evaluating agreement among three or more raters rating any number of items. For example, suppose Abigail, Bob and Charlie each rate three paintings, by Picasso, Rembrandt and Monet, on a scale from 1 to 5. Fleiss' kappa corrects for the amount of chance variation.

Skill level:
Challenging

Instructions

1. 1

Count, for each item, the number of times each rating was given. For example, how many times was the Rembrandt painting given a rating of 5? If r is the number or raters and q is the number of rating categories, you should have q*r computations.

2. 2

Square these numbers.

3. 3

4. 4

Subtract qr from the result. In the example, qr is 15. This result will be the numerator in a fraction

5. 5

Multiply rq by the number of raters - 1. In the example 152 = 30. This is the denominator of the fraction.

6. 6

Divide the numerator by the denominator. This is Fleiss' kappa.

Tips and warnings

• You can get Fleiss kappa automatically in statistical programs such as R, SAS or SPSS.

Don't Miss

References

• All types
• Articles
• Slideshows
• Videos
Sort:
• Most relevant
• Most popular
• Most recent