Analysis: Measure Inter-Rater Reliability#

Researchers often code the same design protocol into categories such as problem analysis, solution generation, and evaluation. This notebook compares the three nominal inter-rater reliability estimators in design-research-analysis, including their treatment of one missing rating.

Setup#

python -m pip install design-research-analysis==0.3.1

Step 1: Represent items by rows and raters by columns#

[1]:

import design_research_analysis as analysis


def format_interval(
    result: analysis.InterraterReliabilityResult,
) -> tuple[float, float] | str:
    """Format an optional bootstrap interval for display."""
    interval = result.confidence_interval
    if interval is None:
        return "not estimated"
    low, high = interval
    return round(low, 3), round(high, 3)


codings: list[list[str | None]] = [
    ["problem", "problem", "problem"],
    ["solution", "problem", "problem"],
    ["evaluation", "evaluation", "evaluation"],
    ["solution", "solution", "solution"],
    ["problem", "solution", "solution"],
    ["evaluation", "evaluation", None],
]
print("Items:", len(codings))
print("Raters:", len(codings[0]))
for index, row in enumerate(codings, start=1):
    print(f"{index}: {row}")

Items: 6
Raters: 3
1: ['problem', 'problem', 'problem']
2: ['solution', 'problem', 'problem']
3: ['evaluation', 'evaluation', 'evaluation']
4: ['solution', 'solution', 'solution']
5: ['problem', 'solution', 'solution']
6: ['evaluation', 'evaluation', None]

Step 2: Use Cohen kappa for two raters#

[2]:

cohen = analysis.compute_interrater_reliability(
    [row[:2] for row in codings],
    method="cohen_kappa",
    n_bootstrap=100,
    seed=17,
)
print("Coefficient:", f"{cohen.coefficient:.3f}")
print("95% interval:", format_interval(cohen))
print("Items used:", f"{cohen.n_items_used}/{cohen.n_items}")

Coefficient: 0.500
95% interval: (-0.048, 1.0)
Items used: 6/6

Step 3: Use Fleiss kappa for complete multi-rater rows#

[3]:

fleiss = analysis.compute_interrater_reliability(
    codings,
    method="fleiss_kappa",
    n_bootstrap=100,
    seed=17,
)
print("Coefficient:", f"{fleiss.coefficient:.3f}")
print("95% interval:", format_interval(fleiss))
print("Items used:", f"{fleiss.n_items_used}/{fleiss.n_items}")
print("Missing ratings:", fleiss.missing_ratings)

Coefficient: 0.583
95% interval: (-0.211, 1.0)
Items used: 5/6
Missing ratings: 1

Step 4: Retain partial rows with Krippendorff alpha#

[4]:

alpha = analysis.compute_interrater_reliability(
    codings,
    method="krippendorff_alpha",
    n_bootstrap=100,
    seed=17,
)
print("Coefficient:", f"{alpha.coefficient:.3f}")
print("95% interval:", format_interval(alpha))
print("Items used:", f"{alpha.n_items_used}/{alpha.n_items}")
print("Missing ratings:", alpha.missing_ratings)

Coefficient: 0.667
95% interval: (0.255, 1.0)
Items used: 6/6
Missing ratings: 1

Step 5: Compare the estimators before choosing one#

[5]:

for result in (cohen, fleiss, alpha):
    print(
        f"{result.method}: coefficient={result.coefficient:.3f}, "
        f"items={result.n_items_used}/{result.n_items}, "
        f"missing={result.missing_ratings}"
    )

cohen_kappa: coefficient=0.500, items=6/6, missing=0
fleiss_kappa: coefficient=0.583, items=5/6, missing=1
krippendorff_alpha: coefficient=0.667, items=6/6, missing=1

Interpretation#

These APIs treat labels as nominal. Choose the estimator from the coding design and missing-data policy, not from whichever coefficient is largest. Audit reliability before treating coded events as outcomes or process states.