Assessment

Student Assessment Form

The purpose of the Undergraduate Medical Education Student Assessment Policy is to establish student assessment practices within the undergraduate medical education program (UGME) at the University of Saskatchewan.

The policy has been developed by the Assessment Subcommittee, a standing subcommittee of the Curriculum Committee, tasked to “establish, monitor, and update an assessment system throughout the entire medical curriculum.”

Assessment within the MD program at the College of Medicine will occur in the context of a programmatic curricular alignment such that the assessments are aligned with published student learning objectives and course learning activities. This will include both formative and summative assessments, and will utilize written examinations, (MCQ, SAQ, Essay), performance assessments (OSCE and in-training assessments (ITAR)), assignments (case reports, projects, self-reflection), and portfolios among the assessment tools.

Assessment procedures are to be standardized across courses, where possible, and the Assessment Subcommittee has oversight of all assessment documents.

Committee on Accreditation of Canadian Medical Schools (CACMS): Develops “Standards and Elements” which guide the accreditation review process (and replaces the LCME document).

National Board of Medical Examiners (NBME): Develop and manage the United States Medical Licensing Examination (USMLE).

Formative assessment (for learning): An assessment that is designed to promote student learning, and provides specific qualitative feedback to the learner on their progress towards achievement of objectives, and identifies areas for improvement. Ideally completed early enough to provide sufficient time for improvement.

Summative assessment (of learning): An assessment used to measure students’ achievement of objectives after a period of instruction, such as a section, module, rotation or course. May also provide formative information for future learning.

Rubric: A set of criteria against which a performance is judged, with details outlining what would be required to achieve the various grade levels. This is a way of formalizing and making a subjective assessment more objective and reliable across different assessors.

Reliability: A measure of the reproducibility or consistency of the assessment. If a student were assessed on a different day, or by a different examiner, how close the score would be.

Validity: A measure of if the assessment is actually assessing that which it is designed to assess.

Syllabus Assessment Plan: A document outlining an overall plan of how learning within a unit of instruction (course/module/rotation) will be assessed. It will include methods, timing and weighting of the various assessments.

Course/Module/Rotation Objective Assessment Map: A document detailing how each of the learning objectives comprising the unit of instruction (i.e. course, module, rotation) will be assessed. It usually takes the form of a table aligning the objectives with assessment items, and reflects the relative weighting of the individual objectives within the unit.

Examination Blueprint: A document developed for each major assessment, outlining the course objectives it assesses, and mapping them to the relevant component of the assessment, including attention to the adequacy of sampling from the course objectives and representation as guided by the relative importance.

Standard setting procedure: A procedure that is used to determine a defensible cut score for an examination. Various methodologies can be employed to establish a score representing the minimally acceptable performance of a student in achieving the objectives from which the examination samples. All methods require use of qualified individuals to participate in the standard setting process.

In-Training Assessment Report (ITAR): A rubric outlining performance expectations, often during a clinical rotation.

OSCE: Objective Structured/Standardized Clinical Examination

MCQ: Multiple Choice Question

SAQ: Short Answer Question

This policy applies to all undergraduate students applying to or registered in the Doctor of Medicine (MD) program at the University of Saskatchewan irrespective of the geographically distributed site to which they are currently assigned.

This document addresses assessment within the four-year Undergraduate Medical Education Program. Some specific assessment procedures may be outlined in procedural documents.

Related Policies

• University of Saskatchewan Academic Courses Policy on Class Delivery, Examinations & Assessment of Student Learning, approved by University Council 2015
• College of Medicine Student Information Guide

UGME Remediation Policy

1. Assessment Planning

1.0 Assessment Planning

1.1 Each course, module, or rotation will develop learning objectives outlining what the student will have learned/be able to do upon completion of the course, module, or rotation.

1.2 Each course, module, or rotation will develop a plan of organized learning opportunities to assist students in attaining the above learning objectives.

1.3 Each course, module or rotation will complete the course, module, or rotation objective assessment map, which provides details of how students’ achievement of the learning objectives will be assessed.

• This map will include written narrative description of performance where student-instructor interaction permits.

1.4 Each course will develop a syllabus assessment plan, which will include information on the methods, timing, and relative contribution to the final mark of all course assessments, both formative and summative. Criteria for passing and information on remediation and supplemental assessment must be specified. This plan is included in the syllabus published before or during the first week of scheduled classes.

• Assessment methods selected should be appropriate to the modality of the objective(s) assessed: knowledge, skills, or attitudes.
• In order to ensure multiple perspectives, no single assessment item shall constitute more than 50% of the final grade, other than in the case of a supplemental assessment following remediation.
• Assessment tools, which are amalgamations of multiple assessment items, do not represent single assessment items. Examples would include some ITARs, weekly quizzes, etc...
• No single selected response question (single best answer) should comprise more than 1% of student’s final mark.
• The number of questions and examination timing should reflect guidelines.
• The Assessment Subcommittee and/or Curriculum Committee will review all syllabus assessment plans.

1.5 Each course, module, or rotation will develop an examination blueprint for each of the major assessments (midterms, finals, and any assessment comprising 20% or more of the final grade for that component). When a course is comprised of multiple assessments less than 20%, the majority needs to be blueprinted. For selected response and written examinations, it may be completed as part of the course or module objective assessment map.

1.6 The assessment planning documents (objective map, plan, and blueprint) will be submitted to the Assessment Team, who will work with the course/module/rotation directors to ensure appropriate representation of curricular and program objectives.

1.7. The Assessment Subcommittee will review any items flagged by the Assessment Team and make recommendations on assessment planning documents which will guide the development of the related assessments.

2.0 Examination Development and Administration

2.1 Each course/module/rotation director, working in conjunction with the teaching faculty, will develop a draft of assessments planned for their course/module/rotation (as outlined in their syllabus assessment plan) and submit the draft to the Assessment Team and relevant course administration.

2.2 The submitted assessment will be mapped to the submitted course or module objective assessment maps and/or examination blueprints to evaluate alignment with and adequacy of sampling from the objectives.

2.3 Following review each course/module/rotation director, in consultation with the teaching faculty and the Assessment Team will revise the assessments for quality and alignment.

2.3.1 Assessments may be removed/returned for further editing if they do not address module objectives, are in an inappropriate format, or of poor quality. Flagged assessments will be brought to the Assessment Subcommittee for review.

2.3.2 Examinations are required to undergo the examination edit process to ensure quality of assessment design and alignment.

2.4 The draft assessments shall be finalized in a timely manner to allow for release to students and the administrative team to ensure accurate application/facilitation.

2.4.1 Exams will be finalized no later than two weeks before the date of the exam, or before the course/module begins if assessment is within two weeks of the onset of the course.

2.4.2 Other assessments will be finalized prior to the beginning of the course/module/rotation.

2.5 Assessment dependant, setting of standards may also be completed at this point (see 5.0).

3.0 Remediation and Supplemental Assessment

3.1 Assessment release dates will be reviewed by the course/module/rotation directors in consultation with the Assessment Team.

3.1.1 Unless otherwise determined and approved by the Assessment Team, non examination assessments will be released on the first day of the course/module/rotation.

3.2 Where students are located in multiple sites for training, they will write the same assessments, and be assessed using the same assessment rubrics. Multi-site OSCEs will follow standardized procedures, with joint central coordination of assessors and simulated patients. Where not possible, e.g. deferrals, equivalency of the assessments will be ensured.

• Exception: the Longitudinal Integrated Clerkship (LIC), who will undergo comparable assessments, though timing and structure may vary.

4.0 Standard Setting

4.1 Post assessment, student feedback and statistical reports will be reviewed by the relevant course/module/rotation director, in conjunction with the Assessment Team, prior to final determination of student scores. Any modifications made as a result of this analysis will be applied to the entire class. These data will also be used to inform future curriculum and assessment development.

4.2 Assessments will undergo standard setting procedures as outlined in 5.0. Standard setting adjustments will be applied prior to release of results.

4.3 The relevant course/module/rotation director holds final responsibility for approval of assessment result release.

4.4 Summative grades will be provided to students no later than six weeks following the end of the course/rotation.

5.0 Assessment of Pre-Clinical Years

5.1 Each course/module/rotation director is responsible for standard setting assessments, in collaboration with the Assessment Team, and will use an appropriate standard setting procedure in order to determine the passing grade or cut score for each major assessment. This score will represent the minimal level of competence deemed acceptable for that assessment at that level of training. When testcentered methods are used, participants will include course faculty.

5.2 Standard setting methods will make use of faculty members with appropriate knowledge and expertise.

5.3 Cut scores, as determined by appropriate standard setting procedures, may be adjusted to the pass mark.

5.4 For testing such as oral exams, case reports, essays, and performance-based assessments, including in-training assessment reports (ITARs), attempts must be made to standardize the grading criteria to improve reliability. Rubrics developed for these purposes must be submitted to the Assessment Team, who will work with the course/module/rotation directors to finalize the documents.

• Once finalized, the rubrics will be posted to the current curriculum management system.

6.0 Assessment of Clinical Rotations

6.1 Each course/rotation must provide opportunities for formative assessment and feedback.

6.2 Students will receive constructive formative feedback (i.e. feedback beyond a numerical grade value) on their performance during each required course/rotation in order to allow sufficient time to improve performance based on this feedback. This feedback will be provided and documented by the midpoint of the learning experience for all courses/rotations longer than four weeks duration. For courses/rotations of less than four weeks duration, alternative methods of formative assessment may be used to allow students to reflect on their learning.

6.2.1 Feedback must include written narrative description of performance whenever teacher-student interaction permits.

6.2.2 For the purposes of this policy, practice is to provide written narrative feedback to all students in courses which include instruction in small groups, defined as six or less students per instructor, and where the student/instructor contact time either continues over multiple sessions or is at least three hours or greater in an individual session.

6.2.3 The narrative feedback provided by tutors will be monitored and processes to flag tutors will be followed, such that tutors with recurrent flags will not continue to teach in these small groups.

7.0 Student Feedback on Assessment

7.1 Any student who fails a major summative assessment may request to review their assessment by contacting the appropriate course, module, or rotation director. In most cases this will be accommodated, however, in some instances it cannot due to issues of exam security. Students considering an appeal should refer to the Academic Appeals  Procedure.

7.1.1 Some assessments may be outlined in the syllabus as special forms of assessment and such will not be available for review.

7.2 Exam review can take different formats and review will be determined to ensure assessment security, meet administrative requirements, and conform with the needs of the course/module/rotation. Review format will be determined by the course/module/rotation director in consultation with the Assessment and Academic Support Teams.

7.3 Exam review and academic appeals may result in changes in marks. Where changes in marks are made they will be applied to the class as a whole. However, if summative marks (i.e. marks indicating success or non-success in a given learning experience) have been released to students, these changes in marks will not negatively impact another student.

8.0 Oversight and Review

8.1 Students who are identified as being in academic difficulty may be offered remediation and supplemental assessment (see UGME Remediation Policy for further details).

8.2 The course/module/rotation director will determine the specific type of remediation and supplemental assessment needed for each individual student in consultation with the Assessment and Academic Support Teams.

8.3 The supplemental assessment may be in the form of the original assessment or in another form as appropriate for reassessing the student’s area(s) of academic difficulty.

8.4 The supplemental assessment may be cumulative or non-cumulative, and either override a portion of, or the entire course/module mark.

9.0 Support

Outlined throughout the policy document.

10. Assessment of Clinical Rotations

In addition to information contained in other sections of the policy document:

10.1 Rubrics such as ITARs, or EPA based competency assessments, which incorporate direct observation, must be one of the assessment modalities in Clinical Rotations.

10.1.1 Narrative feedback must be provided to the students as to their progress in meeting the learning objectives of the rotation.

10.2 Formative ITARs will be completed by preceptor(s) at the midpoint of a rotation, and no less than every six weeks in longer clinical learning experiences. Shorter rotations may utilize alternate forms of formative assessment.

10.3 Summative ITARs will be completed by preceptor(s) at the end of a rotation, and summative grades will be provided to the students no later than six weeks after the end of rotation.

10.4 One or more comprehensive OSCEs will be administered during clerkship to assess students’ clinical competencies.

10.5 Where external examinations that are standard set to a pass score different than that established by the College of Medicine are used, the students’ scores will be adjusted to the pass score defined in the course syllabus.

11. Oversight

11.1 The Assessment Subcommittee is responsible for oversight of the submitted assessment documents and will review any items flagged by the Assessment Team or other course faculty.

11.2 The Assessment Subcommittee will review the performance of any course/module/rotation examination(s), (reliability, adherence to blueprint) and course evaluations as they relate to assessment procedures if flagged.

11.3 Item analysis reports, student feedback on test items, and other statistical reports for all assessments shall also be forwarded to the Assessment Team who will bring forward any issues of concern to the Assessment Subcommittee.

11.4 Any adverse assessment performance report, such as low (<70%) course evaluation scores on assessment, high numbers of poorly performing questions, or significant concerns arising from qualitative student feedback, will require a review of the course/module/rotation assessment framework. The subcommittee and Assessment Team will work with the course team in revising the assessment framework for any course where problems are identified.

11.5 The subcommittee will report annually to the Curriculum Committee on the performance of the assessment process.

12. Support

12.1 Faculty appointed to positions of course/module/rotation director will be provided with faculty development and support related to assessment processes.

The Associate Dean, Undergraduate Medical Education, is responsible for providing oversight to the overall administration of the Undergraduate Medical Education Student Assessment Policy at the University of Saskatchewan.

The Director, Assessment, Undergraduate Medical Education, is responsible for the implementation, monitoring, maintenance, and evaluation of the Undergraduate Medical Education Student Assessment Policy at the College of Medicine. This includes the development and stewardship of the standard operating procedures associated with this policy.

The Curriculum Committee, with input from the Assessment Subcommittee, is responsible for evaluating, reviewing, and updating this policy every two years.

Instances or concerns of non-compliance with the Undergraduate Medical Education Student Assessment Policy should be brought to the attention of the Vice-Dean, Education or the Associate Dean, Undergraduate Medical Education, within the College of Medicine.

Procedures for this policy will be maintained by the Associate Dean, Undergraduate Medical Education.

Joshua Lloyd, Director, Assessment

Ph: (306) 966-8906

Email: joshua.lloyd@usask.ca