Stats Interrater Reliability#

Source: examples/stats_interrater_reliability.py

Introduction#

Protocol studies often begin with multiple researchers assigning nominal codes to the same design moves. This example estimates agreement beyond chance using Cohen’s kappa, Fleiss’ kappa, and Krippendorff’s alpha.

Technical Implementation#

The coding matrix uses one row per protocol segment and one column per rater. All three estimates use the same explicit nominal labels. A seeded item bootstrap demonstrates the optional uncertainty interval without introducing an external statistics dependency.

 1from __future__ import annotations
 2
 3import design_research_analysis as dran
 4
 5
 6def main() -> None:
 7    """Estimate three nominal agreement coefficients."""
 8    codings = [
 9        ["problem", "problem", "problem"],
10        ["solution", "problem", "problem"],
11        ["evaluation", "evaluation", "evaluation"],
12        ["solution", "solution", "solution"],
13        ["problem", "solution", "solution"],
14        ["evaluation", "evaluation", "evaluation"],
15    ]
16
17    for method in ("cohen_kappa", "fleiss_kappa", "krippendorff_alpha"):
18        method_codings = [row[:2] for row in codings] if method == "cohen_kappa" else codings
19        result: dran.InterraterReliabilityResult = dran.compute_interrater_reliability(
20            method_codings,
21            method=method,
22            n_bootstrap=200,
23            seed=17,
24        )
25        print(
26            method,
27            f"coefficient={result.coefficient:.3f}",
28            f"interval={result.confidence_interval}",
29        )
30
31
32if __name__ == "__main__":
33    main()

Expected Results#

Run Command

PYTHONPATH=src python examples/stats_interrater_reliability.py

The three coefficients are positive because most segments agree, but they are below one because the raters disagree on two segments. Repeated runs produce the same bootstrap intervals.

References#

Cohen (1960), Fleiss (1971), and Krippendorff (2011) define the reliability coefficients demonstrated here.