Unit Lead: Lynsey Martin
This unit focuses on competencies the Clinician Educator (CE) will need to develop and implement a valid system of assessment. These skills are essential in the development of new curricula and in the management of existing curricula.
The Foundations Unit is a prerequisite for this unit
Curriculum is a co-requisite
Competencies Addressed in this Unit
A CE must be able to
- Apply principles of assessment in developing a system of assessment for a program
- Select assessment instruments that are relevant and appropriate for the learning context
- Describe the challenges of self-assessment
By the end of this Unit, the CE will be able to:
1. Demonstrate comprehension of the principles of assessment, by
- recognizing the multiple purposes of assessment in a system of education
- analyzing how a system of assessment fits within the cycle of curricular design
- describing the differences between summative and formative assessment
- applying the concept of “utility” to individual assessment instruments and methods
- identifying and describing the professional responsibilities of educators to ensure that the assessment system meets the needs of trainees and the public
- aligning types of assessment with the corresponding levels of Miller’s pyramid
- comparing and contrasting the terms “assessment” and “evaluation”
- justifying the importance of both self-directed assessment and reflection in the development and maintenance of competence of a clinician
- describing the enablers of and barriers to self-assessment
2. Conduct a survey of important assessment instruments, by
- comparing and contrasting the following methods or instruments of assessment, with an emphasis on their structure, the types of objectives they can assess, the context for which they are best suited (e.g., clinical setting, classroom environment, etc.), and their advantages and disadvantages:
- written exams (constructed response and selective response)
- oral examinations
- OSCEs and their derivatives (e.g., OSATs)
- ITERs and other rating-scale–based expert assessment forms
- instruments for direct observation, such as Daily Encounter Cards, CEX
- Multi-Source Feedback
- chart audits and practice assessments
- selecting and implementing appropriate assessment instruments within a curriculum
3. Demonstrate comprehension of the principles of psychometrics, by
- describing modern validity theory and the concept of reliability and applying these to the evaluation of assessment strategies
- applying the concept of utility and analyzing the implications of the assessment system in the context of the candidate’s jurisdiction
- comparing and contrasting classical test theory and generalizability theory
- defining principles of standard setting and selecting appropriate methods for a given assessment instrument and context
- applying the principles of qualitative research in the context of judging the quality of a portfolio as a composite assessment method (using principles of credibility, consistency, triangulation, transferability, etc.)
4. Demonstrating competence in remediation, by
- applying a remediation framework to an encounter the candidate has had with a learner in difficulty, while emphasizing a step-wise approach
- Schuwirth, Lambert WT, and Cees PM van der Vleuten. "A history of assessment in medical education." Advances in Health Sciences Education5 (2020): 1045-1056.
- Schuwirth LW, van der Vleuten CP. General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach. 2011;33(10):783–97.
- Schuwirth LW, van der Vleuten CP. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 2011;33(6):478–85.
- Van der Vleuten C, Schuwirth LWT. Assessing professional competence: from methods to programmes. Med Educ. 2005; 39: 309-317. (video)
- Schut, Suzanne, et al. "Where the rubber meets the road—An integrative review of programmatic assessment in health care professions education." Perspectives on medical education1 (2021): 6-13.
- Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011;33(3):206–14.
- Watling CJ, Ginsburg S. Assessment, feedback and the alchemy of learning. Medical education. 2018 Aug 2. Online First.
- Govaerts MJB, van de Wiel MWJ, Schuwirth LWT, van der Vleuten CPM, Muijtjens AMM.. Workplace-based assessment: raters’ performance theories and constructs. Adv Health Sci Educ Theory Pract. 2012 May 17.
- Ginsburg S, van der Vleuten CP, Eva KW. The hidden value of narrative comments for assessment: a quantitative reliability analysis of qualitative data. Academic Medicine. 2017 Nov 1;92(11):1617-21.
- Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296(9):1094–1102.
- Downing SM, Yudkowsky R. Assessment in the health professions education. New York: Routledge; 2009. Book: available locally through health sciences library
- Holmboe ES, Hawkins RE. Practical guide to the evaluation of clinical competence. Philadelphia (PA): Mosby; 2008. Book: available locally through the health sciences library
- Turnbull J. What is … normative versus criterion-referenced assessment? Med Teach.
- Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach. 2006;28(2):117–28.
- Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9):S62–67.
- Bandiera G, Sherbino J, Frank JF. The CanMEDS assessment tools handbook: an introductory guide to assessment methods for the CanMEDS competencies. Ottawa (ON): The Royal College of Physicians and Surgeons of Canada; 2006. Free PDF available online: google: CanMEDS Assessment Tools Handbook – The Royal College
- Kogan JR, Holmboe ES, Hauer KR. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302(12):1316–26.
- Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E. Opening the black box of clinical skills assessment via observation: a conceptual model. Med Educ. 2011;45(1):1048–60.
- Gingerich A, Kogan J, Yeates P, Govaerts M, Holmboe E. Seeing the ‘black box’ differently: assessor cognition from three research perspectives. Medical education. 2014 Nov;48(11):1055-68.
- Ginsburg S, Van Der Vleuten CP, Eva KW, Lingard L. Cracking the code: residents’ interpretations of written assessment comments. Medical education. 2017 Apr;51(4):401-10.
- Artino, A. R. , Durning, S. J. & Creel, A. H. (2010). AM Last Page: Reliability and Validity in Educational Measurement. Academic Medicine, 85 (9), 1545. doi: 10.1097/ACM.0b013e3181edface.
- Sargeant J, Armson H, Chesluk B, Dornan T, Eva K, Holmboe E, et al. The processes and dimensions of informed self-assessment: a conceptual model. Acad Med. 2010;85(7):1212–20.
- Gagliardi AR, Brouwers MC, Finelli A, Campbell CM, Marlow BA, Silver IL. Physician self-audit: a scoping review. J Cont Ed Health Prof. 2011;31(4):258–64.
- Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE. Construct Validity of the MiniClinical Evaluation Exercise (MiniCEX). Acad Med. 2003;78(8): 826-830.
- Ramani S, Könings KD, Ginsburg S, van der Vleuten CP. Twelve tips to promote a feedback culture with a growth mind-set: Swinging the feedback pendulum from recipes to relationships. Medical teacher. 2018 Feb 7:1-7.
- Tochel C, Haig A, Hesketh A, Cadzow A, Beggs K, Colthart I, et al. The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No. 12. Med Teach. 2009; 31(4):299–318.
- Eva KW, Regehr G. Rater-based assessments as social judgments: rethinking the etiology of rater errors. Acad Med. 2005;80(10 Suppl):S46–54.
- Hauer KE, Ciccone A, Henzel TR, Katsufrakis P, Miller SH, Norcross WA, et al. Remediation of the deficiencies of physicians across the continuum from medical school to practice: a thematic review of the literature. Acad Med. 2009;84(12):1822–32.
- Sargeant J, Mann K, Manos S, Epstein I, Warren A, Shearer C, Boudreau M. R2C2 in action: testing an evidence-based model to facilitate feedback and coaching in residency. Journal of graduate medical education. 2017 Apr;9(2):165-70.
- LaDonna KA, Ginsburg S, Watling C. Shifting and Sharing: Academic Physicians’ Strategies for Navigating Underperformance and Failure. Academic Medicine
- Harden RM. Ten questions to ask when planning a course or curriculum. Med Educ. 1986;20(4):356–65. Not available online- is available in print through health science library
- Kane MT. An argument-based approach to validity. Psychol Bull. 1992;112(3):527–35.
- van de Ridder JM, Stokking KM, McGaghie WC, ten Cate OT. What is feedback in clinical education? Med Educ. 2008;42(2):189–97.
- Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81–11