Kappa is a statistical measure of agreement in ratings between two raters. For example, if two doctors evaluate a patient as either "sick" or "healthy" then kappa measures the extent of their agreement. It is adjusted for agreement due to chance; that is, the agreement we would expect from flipping coins. It contains variants for ordinal data (that is, a variable that has a natural order). Although widely used, kappa has its share of critics.
- Skill level:
Things you need
- Computer with Excel
Open Microsoft Excel on your and enter headers in adjacent cells in rows and columns. In the example, you would enter "Healthy" in cells A2 and B1, and "Sick" In cells A3 and C1.
Enter data. Each cell represents ratings from two raters. In the example, there might be 100 in cell B2 (both rate healthy), 20 in cell C2 (row doctor says "sick" column doctor says "healthy"), 10 in cell B3 (just the reverse) and 30 in cell C3 (both say sick).
Calculate proportions. Add all four cells, then divide each of the four cells by the resulting total.
Go to a new cell and calculate the observed agreement. In the example, this is "=b2+c3." Enter this in cell c6.
Calculate row and column totals. In the example, enter "=b2+b3" in cell b4, "=c2+c3" in cell c4, "=b2+c2" in cell d2 and "=b3+c3" in cell d3.
Go to a new cell and calculate expected agreement In the example, this is "=(b4d2)+(x2d3)". Enter this in cell c7.
Calculate kappa: (observed agreement - expected agreement)/(1-expected agreement). In the example, enter "=(A6-A7)/(1-A7)" in cell A8 to get kappa.
Tips and warnings
- Statistical programs such as SAS or R or SPSS can do this very easily.
- 20 of the funniest online reviews ever
- 14 Biggest lies people tell in online dating sites
- Hilarious things Google thinks you're trying to search for