Drafted by Dr. Yongchao Shi, Assessment Specialist, & Dr. Joan Forder, Implementation Manager, College of Medicine, University of Saskatchewan
UGME: Undergraduate Medical Education
LCME: Liaison Committee on Medical Education recognized by the U.S. Department of Education as the reliable authority for the accreditation of medical education programs
NBME: National Board of Medical Examiners, developing and managing USMLE, the United States Medical Licensing Examination CanMED roles: the seven roles articulated by Royal College of Physicians and Surgeons of Canada and the College of Family Physicians of Canada that lead to optimal health and health care outcomes: medical expert (central role), communicator, collaborator, manager, health advocate, scholar and professional.
Summative assessment: the assessment that is used to measure students’ achievement after a period of instruction of a section, a chapter, a unit, a module, or a course. Therefore, we may have end-of-section quizzes, end-of chapter assignments, end-of module exams, and end-of-course examinations. Mid-term exams and final exams are the major pieces of summative assessments.
Formative assessment: the assessment that is used formally and informally to monitor and support students’ learning progress during instruction of a course/rotation. It typically involves qualitative or narrative feedback (rather than scores) for both student and teacher to focus on the details of content and performance.
Subjective assessment: a form of questioning which may have more than one correct answer or more than one way of expressing the correct answer such as essays and research papers. Marking of such work is usually less reliable and demands a rubric that describes detailed marking criteria for both instructor and student to follow. This type of assessment is usually compared to objective assessment such as true/false and multiple choice questions, in which fixed answers are expected.
Assessment blueprint: a specification of what should be represented on the assessment. It is usually a table or a matrix that aligns assessment with learning objectives and learning experiences and is a necessary step in producing a valid and reliable exam.
Standard setting procedure: a procedure that is used to determine a defensible cut-off score for a high-stakes examination (i.e., an exam which is relatively difficult to pass and greatly affects promotion and scholarship). The Angoff method of the standard setting procedure (see its template in a separate file for details) requires that a group of subject matter experts are asked to evaluate each item and estimate the proportion of minimally competent examinees that would correctly answer the item.
ITER: In-Training Evaluation Reports, which can be used for both summative and formative purposes.
OSCE: Objective Structured Clinical Examination
This document aims at creation of a student assessment policy to address the requirements of the following LCME accreditation standards within the Undergraduate Medical Education (UGME) program.
ED-26. A medical education program must have a system in place for the assessment of medical student achievement throughout the program that employs a variety of measures of knowledge, skills, behaviors, and attitudes.
9.4 Variety of Measures of Student Achievement / Direct Observation of Core Clinical Skills
A medical school ensures that, throughout its medical education program, there is a centralized system in place that employs a variety of measures (including direct observation) for the assessment of student achievement, including students’ acquisition of the knowledge, core clinical skills (e.g., medical history-taking, physical examination), behaviors, and attitudes specified in medical education program objectives, and that ensures that all medical students achieve the same medical education program objectives.
ED-30. The directors of all courses and clerkships (or, in Canada, clerkship rotations) in a medical education program must design and implement a system of fair and timely formative and summative assessment of medical student achievement in each course and clerkship/clerkship rotation.
9.8 Fair and Timely Summative Assessment
A medical school has in place a system of fair and timely summative assessment of medical student achievement in each course and clerkship of the medical education program. Final grades are available within six weeks of the end of a course or clerkship.
ED-31. Each medical student in a medical education program should be assessed and provided with formal feedback early enough during each required course or clerkship (or, in Canada, clerkship rotation) to allow sufficient time for remediation.
9.7 Formative Assessment and Feedback
A medical school ensures that each medical student is assessed and provided with formal formative feedback early enough during each required course or clerkship four or more weeks in length to allow sufficient time for remediation. Formal feedback typically occurs at least at the midpoint of the course or clerkship. A course or clerkship less than four weeks in length provides alternate means by which a medical student can measure his or her progress in learning.
ED-32. A narrative description of medical student performance in a medical education program, including non-cognitive achievement, should be included as a component of the assessment in each required course and clerkship (or, in Canada, clerkship rotation) whenever teacher-student interaction permits this form of assessment.
9.5 Narrative Assessment
A medical school ensures that a narrative description of a medical student’s performance, including his or her non-cognitive achievement, is included as a component of the assessment in each required course and clerkship of the medical education program whenever teacher-student interaction permits this form of assessment.
In addition, this document is complementary to the University of Saskatchewan Policy (Academic Courses Policy on Class Delivery, Examinations & Assessment of Student Learning), the Comprehensive Curriculum Document approved by the UGME Curriculum Committee, Faculty Council, and the University Council 2013, and the Student Information Guide published by the UGME office.
1. Assessment planning
1.1 Each course/rotation should have an assessment plan which details the methods, schedules, and relative marking weights of formative and summative assessments. No final exam should constitute more than 50% toward the final grade. This plan (see its template in a separate file for formats and details) must be included in the syllabus published before or during the first week of scheduled class.
1.2 Each course/module/rotation will file a blueprint with the Assessment Specialist at the UGME office for each of the major summative assessments (i.e., midterm and final exams) before the course begins to ensure appropriate representation of curricular objectives for courses undergoing major revisions. The Assessment Specialist, working with the course development team, will evaluate, modify, and approve the submitted blueprint. The completed and approved blueprint will guide the development of the related examination.
1.3 Narrative description of medical student performance (i.e., written feedback), including relevant, assessable CanMEDS roles, should be included in the assessment plan for both formative and summative assessments of each required course/rotation wherever teacher-student interaction permits this form of feedback.
1.4 Each student should have opportunities to receive feedback by the mid-point of each required course/rotation to allow sufficient time for remediation. Students should have the opportunity to approach the appropriate course/rotation director for assistance if they have academic difficulties.
1.5 Each course/rotation should provide opportunities for formative assessment, participation of which may account for a maximum of 5% toward the final. In courses/rotations of less than 4 weeks of duration, alternative ways of formative assessment such as self-test may be used to allow students for reflection of their learning progress.
2. Administration of assessments
2.1 Each pre-clerkship course/module will have a bank of questions submitted to the Assessment Specialist at the UGME office before the course/module begins.
2.2 Each pre-clerkship course/module director, in conjunction with the teaching faculty, will provide a final draft of the summative examinations planned for their course/rotation before the course/rotation begins.
2.3 Each pre-clerkship course/module directors, in consultation with the teaching faculty, may revise the question items for their summative assessments during the term before the examination is administered.
2.4 Before administration of a summative assessment, the pre-clerkship course/module director, in consultation with the teaching faculty, will submit to the Assessment Specialist at the UGME office a completed checklist regarding to what extent the blueprint is followed.
2.5 Each course/module/rotation director should ensure the standard setting procedure (see its template in a separate file for details) is followed for summative assessments (either OSCE or written exams) in order to set up a defensible cutoff score. A minimum of 3 faculty should participate in the standard setting of each item.
2.6 The UGME office is responsible for NBME exam (clerkship summative assessment) scheduling, administration, and security.
2.7 Formative in-training evaluation reports (ITER) should be completed by a preceptor at the midpoint of a rotation. Narrative feedback should be provided for the students as to whether or not they have satisfactorily met the learning objectives of the rotation and whether there are any concerns relating to their professional behavior and fitness to practice medicine.
2.8 Summative ITER should be completed by a preceptor at the end of a rotation and feedback should be provided to the students no later than 6 weeks after the end of rotation.
2.9 One or more comprehensive OSCEs (Objective Structured Clinical Examinations) should be administered during clerkship to assess students’ competencies related to the CanMed roles.
3. Scoring and review of assessments
3.1 Each pre-clerkship course/module director, in consultation with teaching faculty, will undertake post-assessment review (i.e., review of students’ feedback on the items and review of item statistical report, see its template in a separate file for details) prior to determination of student scores in order to detect problematic answers and set up a reasonable cut-point. Re-scoring procedure that was employed to address problematic items should be applied to the whole class. A copy of item statistical analysis report should be asked to be sent to the Assessment Specialist at the UGME office.
3.2 Where subjective-type testing (e.g. an essay, a research paper, or oral assessment) is used, special effort (e.g., developing a marking rubric, training the evaluators, employing more than one evaluator for marking the same documents) should be made to ensure reliable results.
3.3 Summative pre-clerkship written assessments (either all the items or just the difficult items) excluding final exams are normally reviewed in the whole class and feedback is given within one week of receiving marks but the review and feedback may be delayed until all students have written the exam. Students who have legitimate reasons for not being able to attend the review or who fail the exam will be given opportunity to review the exam later in person with an instructor.
3.4 During the review of a pre-clerkship summative assessment, students are not allowed to write down or record the review. Exams and answer sheets, whether copies or originals, should be picked up again at the end of the review session if they are distributed to students during the review.
3.5. In the event of a pre-clerkship summative assessment failure, a student may request to review his/her assessment by contacting the appropriate course/module Director or Chair. Students wishing to consider an appeal of an examination result should refer to the appeals process outlined in the Student Information Guide.
4. Evaluation of the assessments
4.1 Each course/module director will report annually on the effectiveness and efficiency of their exams to the Assessment Subcommittee.
4.2 Special attention should be paid to the adverse presentations indicated through course evaluations related to assessment (e.g., below 3.5 out of 5), and review of students’ overall performance (e.g., the mean and standard deviation). Revised assessment plan should be submitted to the Assessment Specialist’s office.
Help from the Assessment Specialist: a link to make an appointment
Templates for implementation of the policy
Other related documents University Student Assessment Policy
College of Medicine Assessment Principles in Comprehensive Curriculum Document
College of Medicine Student Information Guide
These files cover policies related to exam construction, invigilation, grading, remedial, promotion, and appeal procedures which may not be specified in this document.