Online Kappa Calculator
The Online Kappa Calculator can be used to calculate kappa--a chance-adjusted measure of agreement--for any number of cases, categories, or raters. Two variations of kappa are provided: Siegel and Castellan's (1988) fixed-marginal multirater kappa and Randolph's free-marginal multirater kappa (see Randolph, 2005; Warrens, 2010). Brennan and Prediger (1981) suggest using free-marginal kappa when raters are not forced to assign a certain number of cases to each category and using fixed-marginal kappa when they are. Values of kappa can range from -1.0 to 1.0, with -1.0 indicating perfect disagreement below chance, 0.0 indicating agreement equal to chance, and 1.0 indicating perfect agreement above chance. A rule of thumb is that a kappa of .70 or above indicates adequate interrater agreement.
First, specify the number of cases, categories,
and raters then click on "Show table." A table will appear with cases in
rows and categories in columns. Then, in the empty cells, input the number
of raters who agreed that a certain case belongs to a certain category. Do not leave any empty cells; input a zero if no raters agreed that a case belonged to that category.
The sum of each row should equal the number of raters. You can cut and
paste data from most spreadsheet software, but you will have to accept the
digital certificate to do so.
If you are having problems running this application in Java, especially after an update, you might need to change your security settings. In Windows, to do this go to the Start menu, find the Java program, and choose Configure Settings. Choose the Security tab and then click on the Edit Site List button. Add the following URL: http://justusrandolph.net/kappa/ Depending on your browser you might also have to choose to unblock the site and allow the Applet to run. The Online Kappa Calculator does not currently work in Google Chrome.
See the following articles for more information on the formulas used here: