Assessment
Categories: Policy Remediation Supplemental Assessment clerkship faculty preclerkship UGME
Updated on:
Purpose
The purpose of the Undergraduate Medical Education Student Assessment Policy is to establish student assessment practices within the undergraduate medical education program (UGME) at the University of Saskatchewan.
The policy has been developed by the Assessment Subcommittee, a standing subcommittee of the Curriculum Committee, tasked to “establish, monitor, and update an assessment system throughout the entire medical curriculum.”
Principles
Assessment within the MD program at the College of Medicine will occur in the context of a programmatic curricular alignment such that the assessments are aligned with published student learning objectives and course learning activities. This will include both formative and summative assessments, and will utilize written examinations, (MCQ, SAQ, Essay), performance assessments (OSCE and in-training assessments (ITAR)), assignments (case reports, projects, self-reflection), and portfolios among the assessment tools.
Assessment procedures are to be standardized across courses, where possible, and the Assessment Subcommittee has oversight of all assessment documents.
Definitions
Committee on Accreditation of Canadian Medical Schools (CACMS): Develops “Standards and Elements” which guide the accreditation review process (and replaces the LCME document)
National Board of Medical Examiners (NBME): Develop and manage the United States Medical Licensing Examination (USMLE)
Formative assessment (for learning): An assessment that is designed to promote student learning, and provides specific qualitative feedback to the learner on their progress towards achievement of objectives, and identifies areas for improvement.
Summative assessment (of learning): An assessment used to measure students’ achievement of objectives after a period of instruction, such as a section, , module, rotation or course. May also provide formative information for future learning.
Rubric: A set of criteria against which a performance is judged, with details outlining what would be required to achieve the various grade levels. This is a way of formalizing and making a subjective assessment more objective and reliable across different assessors.
Reliability: A measure of the reproducibility or consistency of the assessment. If a student were assessed on a different day, or by a different examiner, how close the score would be.
Validity: A measure of if the assessment is actually assessing that which it is designed to assess.
Syllabus Assessment Plan: A document outlining an overall plan of how learning within a unit of instruction (course/module/rotation) will be assessed. It will include methods, timing and weighting of the various assessments.
Course/Module/Rotation Objective Assessment Map: A document detailing how each of the learning objectives comprising the unit of instruction (i.e. course, module, rotation) will be assessed. It usually takes the form of a table aligning the objectives with assessment items, and reflects the relative weighting of the individual objectives within the unit.
Examination Blueprint: A document developed for each major assessment, outlining the course objectives it assesses, and mapping them to the relevant component of the assessment, including attention to the adequacy of sampling from the course objectives and representation as guided by the relative importance.
Standard setting procedure: A procedure that is used to determine a defensible cut score for an examination. Various methodologies can be employed to establish a score representing the minimally acceptable performance of a student in achieving the objectives from which the examination samples.
In-Training Assessment Report (ITAR): A rubric outlining performance expectations, often during a clinical rotation.
OSCE: Objective Structured/Standardized Clinical Examination
MCQ: Multiple Choice Question
SAQ: Short Answer Question
Scope of Policy
This policy applies to all undergraduate students applying to or registered in the Doctor of Medicine (MD) program at the University of Saskatchewan irrespective of the geographically distributed site to which they are currently assigned.
This document addresses assessment within the four-year Undergraduate Medical Education Program. Some specific assessment procedures may be outlined in procedural documents.
Related Policies
Policy
1. Assessment Planning
1.1 Each course, module or rotation will develop learning objectives outlining what the student will have learned/be able to do upon completion of the course, module or rotation.
1.2 Each course, module or rotation will develop a plan of organized learning opportunities to assist students in attaining the above learning objectives.
1.3 Each course, module or rotation will complete the “Course, Module or Rotation Objective Assessment Map”, which provides details of how students’ achievement of each of the learning objectives will be assessed. A template is provided.
- This map will include written narrative description of performance where student-instructor interaction permits.
1.4 Each course will develop a “Syllabus Assessment Plan”, which will include information on the methods, timing, and relative contribution to the final mark of all course assessments, both formative and summative. Criteria for passing and remediation must be specified. This plan is included in the syllabus published before or during the first week of scheduled classes. A sample plan is available.
- Assessment methods selected should be appropriate to the modality of the objective(s) assessed: knowledge, skill or attitude.
- In order to ensure multiple perspectives, no single assessment item shall constitute more than 50% of the final grade, other than in the case of a supplemental assessment following a remediation.
- ITARs or similar assessment tools, which are derived from multiple observations, do not represent single assessment items.
- No single selected response question (single answer) should comprise more than 1% of student’s final mark.
- The number of questions and examination timing should reflect guidelines
1.5 Each course, module or rotation will develop an “Examination Blueprint” for each of the major assessments (midterms, finals, and any assessment comprising 20% or more of the final grade for that component). When a course is comprised of multiple assessments less than 20%, the majority does need to be blueprinted. A template for this blueprint for OSCE examinations is available. For MCQ and written examinations, it may be completed as part of the Course or Module Objective Assessment Map.
1.6 The assessment planning documents (objective map, plan and blueprint) will be submitted to the Assessment Specialist, who will work with the course/module/rotation directors and the Director, Assessment to ensure appropriate representation of curricular and program objectives.
1.7. Once finalized, the Assessment Subcommittee will review any items flagged by the assessment specialist and make recommendations on assessment planning documents. which will guide the development of the related assessments.
2.0 Examination Development and Administration
2.1 Each course/module/rotation director, working in conjunction with the teaching faculty, will develop a draft of the major examinations planned for their course/module/rotation and submit the draft to the Assessment Specialist and relevant course administration.
2.2 The submitted examinations will be mapped to the submitted Course or Module Objective Assessment Maps and/or Examination Blueprints to evaluate congruence with and adequacy of sampling from the objectives.
2.3 Each course/module/rotation director, in consultation with the teaching faculty, Director, Assessment and the Assessment Specialist, will revise the question items during the academic term before the examination is administered.
- Submitted questions may be removed/returned for further editing if they do not
address module objectives, are in an inappropriate format, or of poor quality.
2.4 The draft examination shall be finalized no later than two weeks before the date of the assessment, or before the course/module begins if assessment is within 2 weeks of the onset of the course.
2.5 Where students are located in multiple sites for training, they will write the same examinations, and be assessed using the same assessment rubrics, jointly developed by relevant faculty. Multi-site OSCE’s will follow standardized procedures, with joint central coordination of assessors and simulated patients. Where not possible, e.g. deferrals, equivalency of the assessments will be ensured. The one exception to this is students in a Longitudinal Integrated Clerkship (LIC), who will undergo comparable assessments, though timing and structure may vary. For such experiences, students must receive formative feedback at least every six weeks.
2.6 Post examination, student feedback on test items, item analysis and other test statistical reports will be reviewed by the relevant course/module/rotation director and/or teaching faculty, in conjunction with the Assessment Specialist, prior to final determination of student scores. Any modifications made as a result of this analysis will be applied to the entire class. These data will also be used to inform future item development and reuse.
2.7 Summative grades will be provided to students no later than 6 weeks following the end of the course/rotation.
3.0 Remediation and Supplemental Assessment
3.1 Students who are identified as being in academic difficulty may be offered remediation and supplemental assessment. (see UGME Remediation Policy for further details)
3.2 The course/module director will determine the specific type of remediation and supplemental assessment needed for each individual student.
3.3 The supplemental assessment may be in the form of the original assessment or in another form as appropriate for reassessing the student’s area(s) of academic difficulty.
3.4 The supplemental assessment may be cumulative or non-cumulative, and either override a portion of, or the entire course/module mark.
4.0 Standard Setting
4.1 Each course/module/rotation director is responsible for standard setting assessments, in collaboration with the assessment specialist, and will use an appropriate standard setting procedure in order to determine the passing grade or cut score for each major assessment. This score will represent the minimal level of competence deemed acceptable for that assessment at that level of training. When test centered methods are used, participants will include course faculty.
4.2 Cut scores, as determined by appropriate standard setting procedures, may be adjusted to the pass mark, as defined in the course syllabus.
4.3 For testing such as oral exams, case reports, essays, and performance-based assessments, including in-training assessment reports (ITARs), attempts must be made to standardize the grading criteria to improve reliability. Rubrics developed for these purposes must be submitted to the Assessment Specialist, who will work with the course/module/rotation directors to finalize the documents. Once finalized the rubrics will be posted to the current curriculum management system.
5.0 Assessment of Pre-Clinical Years
Outlined throughout the policy document
6.0 Assessment of Clinical Rotations
In addition to information contained in other sections of the policy document:
6.1 Rubrics such as ITARs, or EPA based competency assessments, which incorporate direct observation, must be one of the assessment modalities.
- Narrative feedback must be provided for the students as to their progress in meeting the learning objectives of the rotation.
6.2 Formative ITARs will be completed by preceptor(s) at the midpoint of a rotation, and no less than every six weeks in longer clinical learning experiences. Shorter rotations may utilize alternate forms of formative assessment, as outlined in 7.2.
6.3 Summative ITARs will be completed by preceptor(s) at the end of a rotation, and feedback provided to the students no later than 6 weeks after the end of rotation.
6.4 One or more comprehensive OSCEs will be administered during clerkship to assess students’ clinical competencies.
6.5 Where external examinations such as the NBME, standard set to a pass score lower than that established by the College of Medicine are used, the students’ scores will be adjusted to the pass score defined in the course syllabus.
7.0 Student Feedback on Assessment
7.1 Each course/rotation must provide opportunities for formative assessment.
7.2 Students should receive constructive formative feedback (i.e. feedback beyond a numerical grade value) on their performance during each required course/module in order to allow sufficient time to improve performance based on this feedback. This feedback will be provided and documented by the midpoint of the learning experience for all courses/modules longer than four weeks duration. For courses/modules of less than 4 weeks duration, alternative methods of formative assessment may be used to allow students to reflect on their learning.
- Feedback must include written narrative description of performance whenever teacher-student interaction permits this form of assessment.
- For the purposes of this policy, practice is to provide written narrative feedback to all students in courses which include instruction in small groups, defined as 6 or less students per instructor, and where the student/instructor contact time either continues over several sessions or is at least 3 hours or greater in an individual session.
- The narrative feedback provided by tutors will be monitored and processes to flag tutors will be followed, such that tutors with recurrent flags will not continue to teach in these small groups (see procedure on Narrative Feedback Quality Improvement Process).
7.3 Students with questions or concerns regarding their assessment have the opportunity to approach the appropriate course/rotation director for assistance.
7.4 Any student who fails a major internal summative assessment may request to review his/her assessment by contacting the appropriate course, module or rotation Director. In most cases this will be accommodated, however, in some instances it cannot due to issues of exam security. Students considering an appeal should refer to the Procedure for Academic Appeals.
8.0 Oversight and Review
8.1 The Assessment Subcommittee is responsible for oversight of the submitted assessment documents and will review any items flagged by the Assessment Specialist, Director, Assessment or other course faculty.
8.2 The Assessment Subcommittee will review the performance of any course/module/rotation examination(s), (reliability, adherence to blueprint) and course evaluations as they relate to assessment procedures if flagged.
8.3 Item analysis reports, student feedback on test items, and other statistical reports for all assessments shall also be forwarded to the Assessment Specialist, and Director,Assessment who will bring forward any issues of concern to the Assessment Subcommittee.
8.4 Any adverse assessment performance report, such as low (<70%) course evaluation scores on assessment, high numbers of poorly performing questions, or significant concerns arising from qualitative student feedback, will require a review of the course/module/rotation assessment framework. The subcommittee and Assessment Specialist will work with the course team in revising the assessment framework for any course where problems are identified.
8.5 The subcommittee will report annually to the Curriculum Committee on the performance of the assessment process.
9.0 Support
9.1 Faculty appointed to positions of course or module director will be provided with faculty development and support in the process of developing objectives, course design, assessment planning, developing assessments, and post hoc item analysis.
Responsibilities
The Associate Dean, Undergraduate Medical Education, is responsible for providing oversight to the overall administration of the Undergraduate Medical Education Student Assessment Policy at the University of Saskatchewan.
The Manager, Undergraduate Medical Education, is responsible for the implementation, monitoring, maintenance, and evaluation of the Undergraduate Medical Education Student Assessment Policy at the College of Medicine. This includes the development and stewardship of the standard operating procedures associated with this policy.
The Curriculum Committee with input from the Assessment Sub-Committee is responsible for evaluating, reviewing, and updating this policy every three years.
Non-compliance
Instances or concerns of non-compliance with the Undergraduate Medical Education Student Assessment Policy should be brought to the attention of the Vice-Dean, Education or the Associate Dean, Undergraduate Medical Education, within the College of Medicine.
Procedures
Procedures for this policy will be maintained by the Associate Dean, Undergraduate Medical Education.