Interobserver Agreement (IOA) is a critical measure in behavioral data collection that quantifies the degree to which two or more independent observers agree on the occurrence and non-occurrence of target behaviors. This handbook provides comprehensive guidance on calculating, interpreting, and improving IOA in educational and clinical settings.
IOA represents the extent to which independent observers record the same behavior in the same way. It serves as a quality control measure, ensuring that:
| Term | Definition |
|---|---|
| Primary Observer | The main data collector whose data are used for analysis |
| Secondary Observer | An independent observer who collects data simultaneously for reliability checks |
| Agreement | When both observers record the same outcome for a given observation period |
| Disagreement | When observers record different outcomes for the same observation period |
| Observation Interval | A discrete time period during which behavior is observed and recorded |
Collect IOA data for at least 20-33% of all observation sessions, distributed across different times, settings, and conditions to ensure representative reliability assessment.
High IOA is essential for making informed decisions about student behavior and intervention effectiveness. Without adequate IOA, we cannot be confident that:
Various professional organizations have established minimum IOA standards:
| Organization/Context | Minimum IOA Standard |
|---|---|
| General Research | 80% or higher |
| High-stakes Decisions | 90% or higher |
| Training Phase | 85% before independent data collection |
| Complex Behaviors | 75% may be acceptable with justification |
IOA below 80% suggests potential problems with operational definitions, observer training, or data collection procedures that must be addressed before continuing data collection.
The choice of IOA calculation method depends on the type of data being collected and the recording procedure used. This chapter provides an overview of different IOA methods and when to use each.
When in doubt, choose the more conservative (stringent) IOA calculation method. This provides greater confidence in your data quality and helps identify potential measurement issues earlier.
Total Count IOA is the simplest form of IOA calculation, used when observers are counting discrete behaviors throughout an observation session. It compares the total number of behaviors recorded by each observer.
Scenario: Two observers counting hand raises during a 30-minute class
Observer A: 15 hand raises
Observer B: 18 hand raises
Calculation: (15 ÷ 18) × 100 = 83.3% IOA
Total Count IOA can yield high agreement even when observers are recording completely different instances of behavior. Consider using more stringent methods for important decisions.
Mean Count-per-Interval IOA divides the observation session into smaller intervals and calculates agreement for each interval separately, then averages these agreements. This method is more sensitive to disagreements than Total Count IOA.
5-minute intervals across 15 minutes:
| Interval | Observer A | Observer B | Interval IOA |
|---|---|---|---|
| 1 (0-5 min) | 3 | 4 | (3÷4)×100 = 75% |
| 2 (5-10 min) | 5 | 5 | (5÷5)×100 = 100% |
| 3 (10-15 min) | 2 | 3 | (2÷3)×100 = 66.7% |
Mean IOA: (75 + 100 + 66.7) ÷ 3 = 80.6%
Trial-by-Trial IOA is used when data are collected on discrete learning opportunities or trials, such as in discrete trial training (DTT) or when measuring responses to specific prompts.
10 math problem trials:
| Trial | Observer A | Observer B | Agreement? |
|---|---|---|---|
| 1 | Correct | Correct | Yes ✓ |
| 2 | Incorrect | Correct | No ✗ |
| 3 | Correct | Correct | Yes ✓ |
| 4 | No Response | No Response | Yes ✓ |
| 5-10 | All agreements ✓ | ||
IOA: (9 agreements ÷ 10 trials) × 100 = 90%
Interval-by-Interval IOA is the most commonly used method for partial interval, whole interval, and momentary time sampling data. It examines agreement on the occurrence or non-occurrence of behavior in each interval.
10-second intervals for 2 minutes (12 intervals total):
| Interval | Observer A | Observer B | Agreement? |
|---|---|---|---|
| 1 | + | + | Yes |
| 2 | - | - | Yes |
| 3 | + | - | No |
| 4 | + | + | Yes |
| 5-12 | 7 agreements, 1 disagreement | ||
IOA: (10 agreements ÷ 12 intervals) × 100 = 83.3%
Agreement means both observers marked presence OR absence in the interval
Agreement requires matching on whether behavior occurred throughout entire interval
Agreement based on behavior status at the moment of observation
When measuring how long behaviors last (duration) or how long before they begin (latency), specialized IOA calculations account for the continuous nature of time measurement.
Scenario: Measuring on-task behavior duration
Observer A: 18 minutes 30 seconds (1110 seconds)
Observer B: 19 minutes 45 seconds (1185 seconds)
IOA: (1110 ÷ 1185) × 100 = 93.7%
Scenario: Time to begin task after instruction
Observer A: 12 seconds
Observer B: 14 seconds
IOA: (12 ÷ 14) × 100 = 85.7%
For multiple occurrences, calculate IOA for each occurrence separately, then average:
Use synchronized stopwatches or timing apps to ensure observers start and stop timing at the same moments. Consider using video recording for post-hoc reliability checks when precise timing is critical.
IOA scores provide important information about data quality, but interpretation requires consideration of multiple factors beyond the percentage itself.
| IOA Range | Interpretation | Action Required |
|---|---|---|
| 90-100% | Excellent agreement | Continue current procedures |
| 80-89% | Good agreement | Monitor and maintain |
| 70-79% | Questionable agreement | Review definitions and retrain |
| Below 70% | Poor agreement | Stop data collection and address issues |
High IOA with very low-rate behaviors may be misleading. If behavior occurs in only 2 of 100 intervals, 98% agreement might reflect agreement on non-occurrence rather than true reliability.
When IOA falls below acceptable levels, systematic approaches can identify and address the sources of disagreement.
Regular calibration prevents observer drift:
When IOA is consistently below 80%, use this systematic approach to identify and resolve issues:
| Pattern | Possible Cause | Solution |
|---|---|---|
| Beginning of session | Start time confusion | Use countdown to synchronize |
| Specific intervals | Environmental factors | Reposition observers |
| Particular behaviors | Definition issues | Refine operational definition |
| Gradual decline | Observer fatigue | Shorter observation periods |
| Random | Attention issues | Reduce competing tasks |
Solution: Create a decision rule for connected behaviors. For example: "Separate occurrences must have at least 3 seconds between them."
Solution: Check for observer drift. Both observers may have shifted their criteria together. Recalibrate with original training materials.
| Method | Formula | When to Use |
|---|---|---|
| Total Count | (Smaller ÷ Larger) × 100 | Event recording, quick checks |
| Mean Count-per-Interval | Average of interval IOAs | Event recording, detailed analysis |
| Trial-by-Trial | (Agreements ÷ Trials) × 100 | Discrete trials, structured teaching |
| Interval-by-Interval | (Agreement intervals ÷ Total) × 100 | All interval recording types |
| Duration | (Shorter time ÷ Longer time) × 100 | Duration or latency measures |
Access our comprehensive video training series (coming soon) covering: