Interobserver agreement for IEP data
By the IEP Atlas editorial teamLast updated
Short answer
Interobserver agreement (IOA) is the percent of the time two people recording the same session get the same result. Two observers score the session independently, then you divide agreements by agreements plus disagreements and multiply by 100. The exact formula depends on the data type: count, interval, trial or duration. Low agreement means the definition or the training needs work, not the student.
Interobserver agreement answers one question: if a second adult had taken this data, would they have written down the same thing? When the answer is no, the progress graph is partly a picture of the observer. A goal that looks met on the para's data and unmet on the teacher's data is not a student problem. It is a measurement problem.
This guide gives the standard IOA formulas for each kind of IEP data, with numbers worked through. The method choice itself is covered in the guide on choosing a data collection method, and the full data system in the IEP data collection guide.
What you need before you start
- An operational definition of the behavior, written down, that both observers read beforehand.
- Two copies of the same data sheet with the same intervals, trials or time marks.
- A shared timing cue for interval data, such as a vibrating timer both observers can feel, or a recording on one device.
- A plan for who observes, when, and that neither looks at the other's sheet until the end.
The IOA formulas by data type
| Data type | Formula | Best for |
|---|---|---|
| Total count | Smaller count ÷ larger count × 100 | Quick check on frequency data |
| Mean count-per-interval | Average of (smaller ÷ larger) for each short block | Frequency data where timing matters |
| Exact count-per-interval | Blocks where counts match exactly ÷ total blocks × 100 | Strict check on frequency data |
| Interval-by-interval | Agreements ÷ (agreements + disagreements) × 100 | Partial, whole interval and momentary time sampling |
| Scored-interval | Agreements on scored intervals ÷ intervals either observer scored × 100 | Low-rate behavior on interval sheets |
| Trial-by-trial | Trials scored the same ÷ total trials × 100 | Discrete trials, opportunities, task analysis steps |
| Total duration | Shorter total ÷ longer total × 100 | Duration and latency data |
The total count formula is the easiest and the weakest. Two observers can each count 10 call-outs and still disagree on which 10 they saw. The block-by-block formulas catch that.
Step by step
Step 1: Pick the sessions to check
Choose sessions that are typical, not the calmest period of the week. A check during independent work, when the behavior happens, tells you more than one during a video. For school data, a practical routine is one check per goal each month, plus one whenever a new collector starts.
Step 2: Observe at the same time, apart
Both observers watch the same student for the same time span with the same sheet. They sit where they cannot see each other's marks. If the session uses intervals, both must mark the same interval, so start together on one signal.
Step 3: Compare unit by unit
Lay the sheets side by side. For trials and intervals, compare box by box and mark A for agree or D for disagree. For counts, split the session into short blocks, such as 1-minute blocks, if you plan to use a block formula.
Step 4: Apply the matching formula
Use the table above. Report the formula you used along with the result, because "92% agreement" means different things under total count and exact count.
Step 5: Act on the result
If agreement is high, record it with the date and keep going. If it is low, look at where the disagreements cluster. Many teams treat agreement below about 80% as a sign to retrain or rewrite the definition before trusting the data. That is a working convention, not a legal rule.
Worked example: Kayla's math block
Kayla is a fictional 4th grader with two goals in the math IEP goals area. One is a behavior goal: reduce calling out during math to 2 or fewer per 20-minute block. The other is a math facts goal: answer 18 of 20 multiplication facts correctly within 3 seconds each. Her teacher, Mr. Ortiz, and the school psychologist observe the same Tuesday math block.
Calling out: total count. Mr. Ortiz counts 9 call-outs. The psychologist counts 11. Total count IOA = 9 ÷ 11 × 100 = 82%.
Calling out: mean count-per-interval. They split the 20 minutes into four 5-minute blocks.
| Block | Teacher | Psychologist | Smaller ÷ larger |
|---|---|---|---|
| 1 | 3 | 3 | 1.00 |
| 2 | 1 | 2 | 0.50 |
| 3 | 4 | 4 | 1.00 |
| 4 | 1 | 2 | 0.50 |
| Total | 9 | 11 | Mean 0.75 |
Mean count-per-interval IOA is 75%. Exact count-per-interval is 2 matching blocks of 4, or 50%. The disagreements fall in blocks 2 and 4, both during partner work. Talking to a partner loudly was being scored as calling out by one observer and not the other.
Fix. They add one line to the definition: "Calling out is speaking to the teacher or the whole class without being called on. Talking to an assigned partner does not count." On Thursday the counts are 6 and 6, with every block matching. Exact count-per-interval IOA is 100%.
Math facts: trial-by-trial. Both observers score the same 20 flashcards with a 3-second rule.
| Result | Trials |
|---|---|
| Both scored correct | 14 |
| Both scored incorrect | 3 |
| Teacher correct, psychologist incorrect | 2 |
| Teacher incorrect, psychologist correct | 1 |
Agreements are 14 + 3 = 17. Disagreements are 3. Trial-by-trial IOA = 17 ÷ 20 × 100 = 85%. The 3 disagreements were answers given near the 3-second mark. The team switches from counting in their heads to a visible timer.
Duration example. If the goal had measured time out of seat, and the observers recorded 7 minutes 30 seconds and 8 minutes, total duration IOA = 450 ÷ 480 × 100 = 94%.
Interval data: why the formula changes the answer
For a 30-interval partial interval sheet, suppose the observers agree on 26 intervals and disagree on 4. Interval-by-interval IOA = 26 ÷ 30 × 100 = 87%.
Now suppose the behavior is rare. Each observer scored only 3 intervals, and they agreed on 1 of them. The other 26 agreements are intervals where both saw nothing. Interval-by-interval still looks high, at 26 ÷ 30. Scored-interval IOA tells the truth: intervals either observer scored are 5, they agreed on 1, so 1 ÷ 5 × 100 = 20%. For low-rate behavior, report scored-interval IOA. The interval recording guide explains the sampling methods themselves.
Common mistakes
- Weak: Second observer looks at the first one's sheet halfway through. Better: Sit apart and compare only at the end.
- Weak: Reporting "90% agreement" with no formula. Better: "Exact count-per-interval IOA, 90%, 5-minute blocks."
- Weak: Using total count on frequency data and stopping there. Better: Split into blocks so you see where observers part ways.
- Weak: Checking only on easy days. Better: Check during the activity where the behavior happens.
- Weak: Blaming the para for low agreement. Better: Read the definition first; most disagreements trace back to a gray area in it.
- Weak: Interval-by-interval IOA on a behavior seen in 2 of 40 intervals. Better: Report scored-interval IOA.
Where IOA fits in the IEP record
The IEP must describe how progress toward each goal will be measured.[1] Agreement checks are how a team shows that measurement is consistent across the people taking the data. Write the IOA result in the data file next to the session it belongs to, and mention it in progress reports when a new collector joins. Keep the agreement checks in the same binder or file as the data sheets they belong to. The guide to setting up an IEP data binder shows one way to organize it.
Frequently asked questions
What is a good interobserver agreement percentage?
Many teams use about 80% as the point where they retrain or rewrite the definition, and aim higher for important decisions. Treat it as a working convention. Always state which formula you used, because the same session can give different percentages under different formulas.
How often should we check interobserver agreement?
A practical school routine is one check per goal each month, plus a check whenever a new person starts collecting data or the definition changes. Check during the activity where the behavior happens, not only on calm days.
What is the difference between interval-by-interval and scored-interval IOA?
Interval-by-interval counts every interval where observers agree, including intervals where neither saw the behavior. Scored-interval counts only intervals where at least one observer marked the behavior. For rare behavior, scored-interval gives a more honest number.
Use it in IEP Atlas
- Math IEP goals
- Measurement methods
- Free IEP data sheets
- Operational definition in the IEP glossary
- Interval recording measurement guide
Related guides
- IEP data collection: methods, sheets and schedules that hold up
- How to choose a data collection method for an IEP goal
- How to take trial-by-trial data during instruction
- Partial interval, whole interval and momentary time sampling compared
- Setting up an IEP data binder
- Prompt hierarchy examples for IEP data
Sources
- 34 CFR §300.320: Definition of individualized education program, Legal Information Institute, Cornell Law School
- What Is Data-Based Individualization?, National Center on Intensive Intervention
Written and checked by the IEP Atlas editorial team under our editorial policy. This guide describes general practice and federal rules. It is not legal advice; your state may add requirements, and the IEP team makes the decisions for each student.