As a follow-up post to The Power of Learning Analytics from last spring, guest blogger Jane M. Souza, Ph.D., Assistant Dean of Assessment in the Wegmans School of Pharmacy at St. John Fisher College, shares the strategies used within the school to collect, analyze and continuously improve the Pharmacy program through the use of the computer-based exam management software, ExamSoft and the School of Pharmacy assessment process.
All courses have learning outcomes listed on the syllabi. All faculty members strive to teach learning outcomes in ways that encourage student achievement. These facts are nothing new and are common among any undergraduate or graduate level course, but the way faculty in the School of Pharmacy are looking at student progress on learning outcomes is new. Rather than relying solely on test grades to assess student progress, they are tracking density of coverage on each course outcome along with whole-class as well as individual student achievement within each of those areas. How are they managing that much data? They are testing in an electronic environment using ExamSoft. The implications for change management at the course level are outstanding and only exceeded by the ability to drive quality improvement at the program level and opportunities for self-monitoring at the student level.
So how does this work exactly? It all begins with pulling data out of existing exams maximizing the potential of embedded assessments. Test items, or questions, have been crafted and honed by faculty over years of teaching experience. The items include true/false, fill-in-the-blank, multiple choice and essay questions. Using ExamSoft, each item is electronically tagged or coded to identify the question on multiple levels including the following: learning outcome, program standard, accreditation standard, and level in Bloom’s Taxonomy (e.g. knowledge, application, synthesis). Questions can be tagged several times as needed. For example, one question may be addressing course learning outcomes 1, 3; program outcomes 1e, 2g. 3a; accreditation standards, 16.2, 32.1; and application level.
When students take their exams, data is collected for each test item. Faculty are provided a wealth of information previously unavailable to them. They know how much time, to the second, students spend on each question – both individually and as a class. This is great information for time management and test construction. They know how many questions they asked per exam and per semester on each learning outcome. Most importantly, they know the student achievement on each of these learning outcomes. Consider the two tables to the right and below. The table to the right shows the course-level information available prior to electronic testing. The table below shows the detail currently available and how faculty might use the data.
From the tables above, it is easy to see how faculty can make evidence-based changes to their classes. The same information can drive curricular reform. Consider the old-style curriculum map. It documents that the curriculum has been taught, but does not record what was learned.
Old Style Curriculum Map
Evidence-Based Curriculum Map with Density of Coverage and Student Performance (note the opportunity for improvement)
At the student level, individualized learning analytics help students consider their own strengths and opportunities for improvement. Whereas, in the past students were monitoring their progress through course grades, they are now able to reflect on their strengths and weakness at the learning outcomes level. Previously hidden deficiencies are now revealed and can be addressed promptly.
The student represented in the table to the right appears to be strong in each area, but are there hidden deficiencies?
The student may have a lack of understanding for a single learning outcome, but the deficiency is masked by high achievement in other areas. When students are able to see their performance at the learning outcome level, as shown below, they are able to work on remediation well before they encounter critical assessments such as exit or licensing exams.
The learning analytics made possible through electronic testing have been put to good use in the Wegmans School of Pharmacy. The faculty are employing the data to make course-level changes, the students are addressing their areas of weakness, and curriculum and assessment committees are analyzing the data for continuous improvement at the program level.
Feel free to visit us at Wegmans School of Pharmacy to see this process in action, or stop by and see me. I am always happy to talk assessment.
Jane M. Souza, Ph.D.
Assistant Dean of Assessment, Wegmans School of Pharmacy