The METRIQ Study team was happy to see a study by several of our authors on the reliability and validity of the ALiEM AIR score published today! This study, now available as an article-in-press [1], examined the use of the ALiEM AIR tool by the team responsible for the ALiEM Approved Instructional Resources (AIR) series. The team found an inter-rater reliability of 0.81 and that 9 raters were required to reliably evaluate a resource. The score had a fair (0.40) correlation with the gestalt quality evaluation of attending physicians.
The Editor’s capsule summary was as follows:
How does this relate to the METRIQ Study?
In two ways:
First, the METRIQ Study is the “further research” needed to “optimize rating systems and compare them with gestalt assessments.” As many of the authors of the ALiEM AIR study are also involved with the METRIQ Study (including Teresa Chan, Andy Grock, Mike Paddock, and Michelle Lin), we had access to this data when designing our study (check out a more complete description of our Research Agenda which already has 2 additional studies in the peer-review pipeline). The METRIQ Study will build on these findings and provide an even more robust analysis of the merits of the ALiEM AIR Score for evaluating quality.
Second, through the use of a usability analysis that has been incorporated as a sub-study of the METRIQ Study, the ALiEM AIR Score will be further refined to increase its inter-rater reliability and ability to quantify quality. While the Annals of EM study described above demonstrates the viability of the ALiEM AIR Score in its current form, we believe that we can make it even better with your feedback.
The bottom line is that we still need your help. If this work or these resources intrigue you at all, please sign up as a collaborator on the METRIQ Study on our homepage. Within 24 hours after you indicate your interest you will be sent a link to begin a study that will help FOAM to become even better.
References
Chan TM, Grock A, Paddock M, Kulasegaram K, Yarris LM, Lin M. Examining Reliability and Validity of an Online Score (ALiEM AIR) for Rating Free Open Access Medical Education Resources. Annals of Emergency Medicine, article in press.
