This chapter first discusses the use of item response theory (IRT) model for the analysis of agreement between raters in a situation where all raters have supplied dichotomous ratings of the same cases in a sample. Next, the chapter describes two approaches to the quantification of the rater variation in the Rasch model. The approaches are fixed-effect approach, and random effects approach. The models allow raters to have different propensities to score a given set of individuals positively or negatively. Further, the chapter presents a study of umbilical artery Doppler velocimetry used by obstetricians to asses the status of a fetus. Meta analyses of the randomized clinical trials have shown a 49% reduction in perinatal mortality on using the technique on high-risk pregnancies. It is clear from these considerations that the variations in effects of using the ultrasound technique may in part be attributed to the considerable rater variation.