Category Archives: Learning Analytics

Digital Badges: What Are They & What Can We Do With Them?

Have you heard of badges? No, I don’t mean those that are worn by police or military officials, or even merit badges collected by boy/girl scouts and displayed so proudly on the child’s uniform, though digital badges are somewhat modeled after these badging systems. According to the MacArthur Foundation, “a digital badge is a validated indicator of accomplishment, skill, quality, or interest that can be earned in many learning environments.” Similar to the physical badges that signify the completion of a task or acquired skill level, digital badges can be used to visually display a wide variety of skills and competencies online, including both physical and virtual skills, as well as hard and soft skills. Digital badges also take some principles from video game design as they can be used as a reward for completion of a task or a means to unlock additional tasks that must be completed in a sequential order.

So, now that we know a little more about what digital badges are, why would we want to use them in an educational setting? In general, digital badges are a relatively new educational technology so the answer to this question is evolving. However, there is a significant amount of research on topics that overlap with the use of digital badges like the effects of student motivation, both intrinsic and extrinsic, on achievement, as well as the use of rewards and recognition within virtual communities. These techniques can be implemented in a variety of ways, but digital badges have become a new option to incorporate these strategies into educational settings. Digital badges really started to get attention in 2011 when the Mozilla Foundation announced the Open Badges Project, which provides a set of standards for creating, credentialing and displaying badges of all kinds. Without this type of underlying infrastructure for badges there would not be the attention from the education market that we see today. Digital badges also overlap with ePortfolio technology, as one of the main components of collecting badges is to display them to others as evidence of your skills or accomplishments on given topics. ePortfolios can be a useful place to display this type of information for academic and professional reasons. Digital badges have also generated discussion around the tracking and credentialing of informal learning. With the increase in MOOCs and other non-credit educational experiences and the issue of how to assess and award credit for these types of learning experiences, the use of digital badges have been suggested as one of many possible solutions.

Please take a few minutes to watch the following video to learn more about digital badges and how they may be used within education.

Source: MacArthur Foundation (

Below are a few examples of those using digital badges in higher education:

University of California, Davis – Sustainable Agriculture and Food Systems Program

“Instead of being built around major requirements and grades in standard three-credit courses, the Davis badge system is based on the sustainable-agriculture program’s core competencies—”systems thinking,” for example. It is designed to organize evidence of both formal and informal learning, from within traditional higher education and without.”

Read the full article here:

Purdue’s Passport Project

“In one early example of Passport’s use, instructors are giving out badges for students who pass an 8-week MOOC-like course in nanotechnology that doesn’t have credit attached. In another example, the provost’s office has created a badge related to intercultural learning that students can earn for their work in different disciplines and departments.”

Read the full article here:

Carnegie Mellon University – Computer Science Program

“Carnegie Mellon University has included badges in the Computer Science Student Network (CS2N), which provides a distributed learning infrastructure for computer science and STEM skills. Learners participate in a scaffold environment and work on achievements ranging from entry-level skills to industry certification. Badge pathways provide a clear view of progress, as learners can clearly see how lower-level competencies lead to higher-level competencies. Creative competitions provide additional motivations and opportunities for peer review and learning from others’ work. Learners can progress to the levels of achievement that tie into industry-accepted certifications and entries to employment.”

Read the full article here:

As well as the above examples, Blackboard has also recently released a tool to allow faculty to create badges and award them to students directly within their courses. The Blackboard Achievements tool connects with the Mozilla Open Backpack project, so that students can display badges acquired within a course on a public page for individual use. Here at Fisher, the Achievements tool was part of the recent upgrade to Blackboard that was completed over the holiday break. If you would like to know more about the Achievements tool, please contact me directly and we can discuss your own badging ideas together.

Would you like to learn more about digital badges and discuss their use in education with other interested faculty at Fisher? On behalf of the Educational Technology Roundtable and the Fisher Geeks Blog, we would like to invite all interested faculty to participate in the following webinar sponsored by Pearson and hosted by The Chronicle of Higher Education, as well as participate in an open discussion following the webinar.

Designing Your Own Open Digital Badge Ecosystem

Thursday, February 6, 2014
 2:00 p.m. in Nursing 209

For a full description of the webinar, please click here.


The Power of Learning Analytics: The Wegmans School of Pharmacy Approach

As a follow-up post to The Power of Learning Analytics from last spring, guest blogger Jane M. Souza, Ph.D., Assistant Dean of Assessment in the Wegmans School of Pharmacy at St. John Fisher College, shares the strategies used within the school to collect, analyze and continuously improve the Pharmacy program through the use of the computer-based exam management software, ExamSoft and the School of Pharmacy assessment process.

All courses have learning outcomes listed on the syllabi. All faculty members strive to teach learning outcomes in ways that encourage student achievement. These facts are nothing new and are common among any undergraduate or graduate level course, but the way faculty in the School of Pharmacy are looking at student progress on learning outcomes is new. Rather than relying solely on test grades to assess student progress, they are tracking density of coverage on each course outcome along with whole-class as well as individual student achievement within each of those areas. How are they managing that much data? They are testing in an electronic environment using ExamSoft. The implications for change management at the course level are outstanding and only exceeded by the ability to drive quality improvement at the program level and opportunities for self-monitoring at the student level.

So how does this work exactly? It all begins with pulling data out of existing exams maximizing the potential of embedded assessments. Test items, or questions, have been crafted and honed by faculty over years of teaching experience. The items include true/false, fill-in-the-blank, multiple choice and essay questions. Using ExamSoft, each item is electronically tagged or coded to identify the question on multiple levels including the following: learning outcome, program standard, accreditation standard, and level in Bloom’s Taxonomy (e.g. knowledge, application, synthesis). Questions can be tagged several times as needed. For example, one question may be addressing course learning outcomes 1, 3; program outcomes 1e, 2g. 3a; accreditation standards, 16.2, 32.1; and application level.

When students take their exams, data is collected for each test item. Faculty are provided a wealth of information previously unavailable to them. They know how much time, to the second, students spend on each question – both individually and as a class.  WSOP_1This is great information for time management and test construction. They know how many questions they asked per exam and per semester on each learning outcome. Most importantly, they know the student achievement on each of these learning outcomes. Consider the two tables to the right and below. The table to the right shows the course-level information available prior to electronic testing. The table below shows the detail currently available and how faculty might use the data.


From the tables above, it is easy to see how faculty can make evidence-based changes to their classes. The same information can drive curricular reform. Consider the old-style curriculum map. It documents that the curriculum has been taught, but does not record what was learned.

Old Style Curriculum Map


Evidence-Based Curriculum Map with Density of Coverage and Student Performance (note the opportunity for improvement)


At the student level, individualized learning analytics help students consider their own strengths and opportunities for improvement. Whereas, in the past students were monitoring their progress through course grades, they are now able to reflect on their strengths and weakness at the learning outcomes level. Previously hidden deficiencies are now revealed and can be addressed promptly.WSOP_5

The student represented in the table to the right appears to be strong in each area, but are there hidden deficiencies?

The student may have a lack of understanding for a single learning outcome, but the deficiency is masked by high achievement in other areas. When students are able to see their performance at the learning outcome level, as shown below, they are able to work on remediation well before they encounter critical assessments such as exit or licensing exams.


The learning analytics made possible through electronic testing have been put to good use in the Wegmans School of Pharmacy. The faculty are employing the data to make course-level changes, the students are addressing their areas of weakness, and curriculum and assessment committees are analyzing the data for continuous improvement at the program level.

Feel free to visit us at Wegmans School of Pharmacy to see this process in action, or stop by and see me. I am always happy to talk assessment.

Jane M. Souza, Ph.D.
Assistant Dean of Assessment, Wegmans School of Pharmacy

Did You Miss Christmas in July? What’s New in Blackboard for the Fall

Thanks to the upgrade to Blackboard around the Fourth of July holiday, there are now a few new tools and features available in your courses. As you begin to prepare for the fall semester, I thought it would be helpful to share a little more about each one.

My Blackboard Notificationsbbnotification

When you log in to Blackboard you will now notice your name in the upper right corner, as well as small number indicating notifications available for your review. Notifications like these are available for faculty and students and include a list of all courses you are enrolled in, a view of discussion posts across all courses, general updates on course activity, as well as retention center notification (which we will discuss in greater detail below) and calendar items across all courses. The notifications area also provides a quick way to navigate from one course to another within the system. Another new feature available through the My Blackboard Notifications area is the ability to add a photo to your Blackboard profile. The profile picture next to your name in the upper right corner is nice, but the real benefit of this feature is that the photo will also be visible through interactions within a course. For example, when a student authors a blog post, discussion board message or a journal entry, you will not only see their name, but their profile photo will be visible as well next to their message. Especially for courses that spend a significant amount of time communicating within Blackboard, this provides a personal touch to the course that was not previously possible.

New Content Editor

The new content editor is an improvement over the previous content editor and will perform much better with the copy and pasting of text from outside sources, like Microsoft Word documents. You will see in the image below all of the same functionality is available in the new editor so hopefully this change won’t be too significant for most, but might relieve a few formatting headaches that we have had in the past.


Improved Calendar

As mentioned above, the calendar will appear as one of the options in the My Blackboard Notifications area. The new calendar view will now display calendar events across all courses and each item will be color-coded based on the course with which it is associated. This view of the calendar is extremely helpful for students who want a single place to view the deadlines in all of their courses at one time and any possible overlaps. The overall look of the calendar is now similar to the look and feel of other modern calendar tools and includes many common features that you are probably used to already, like viewing the calendar by day, week or month and the ability to drag and drop items from one day to another. Faculty and students can also add personal items to their Blackboard calendar that are not associated with any specific course, but private to the individual. It is also now possible to import the Blackboard calendar into other outside calendar tools like Gmail or Outlook if desired.

Improved Discussion Features

In general the discussions tool will look the same as previous versions, but a few new features really change the flow of conversations that are possible. The most significant improvement is that all posts within a discussion thread now appear on one single page. This makes it much easier to read a conversation without having to go back and forth between pages for each message that you are trying to read. As well, it is now even easier to reply to a message of another class member and the message can be written directly in-line with the conversation. Another new feature in the discussion area is the ability to restrict student access to discussion responses from their peers until they have already posted an original response of their own. This feature will cut down on the issue of students only replying to the comments of their classmates instead of posting their own responses to the question directly. The usability enhancements and new features in the discussion tool will make a real difference in courses that rely heavily on the discussion tool for course communication, especially online and hybrid courses where instructors and students spend a significant amount of time within this area of the course.

New Assignment In-line Grading

If you use Blackboard for the electronic submission and grading of student work in your courses, this is the feature where you will most likely notice the biggest difference. Blackboard has incorporated totally new functionality within the Assignment tool. The creation of a Blackboard Assignment is the same and is still done from within a content area of your course. However, once students begin to submit their work and you visit the Grade Center to review it, you will quickly notice the changes. All documents submitted by students will be immediately available for viewing from the View Attempt window directly in the browser without downloading the files to your own computer. Comments and annotations to the student’s work can also be done directly on this screen. This method of providing feedback to students on their written work is an alternative to using the track changes functionality available within Microsoft Word. However the track changes method is still available for those that prefer that process. Both the original student submission and the document including annotations can be downloaded to your own computer for archiving. This new functionality should make the process of grading and providing feedback on student papers faster since you will no longer need to download and upload documents for each student.

New Retention Centerretentioncenter

The new Retention Center features are an improvement to what was previously available in the early warning system within Blackboard and is available under Evaluation in the Control Panel of every course (shown to the right). The Retention Center is only available to instructors and allows for the identification and communication with students who may need further assistance in the course. A number of rules are added by default to identify at risk students, including students who have not accessed the course for at least five days, students whose overall course grade is at least 25% below the class average, students who have missed assignment deadlines and students who are at least 20% less active overall in the course than their peers. These rules are completely customizable and additional rules may also be added.

Click here for the BB_WhatsNew PDF,  a short summary handout on all of the new features.

Please remember to visit the Educational Technology website for more information on Blackboard, including documentation and tutorials for faculty and students, as well as a list of upcoming events and workshops. Also, I am always available for 1-1 training sessions on any specific tool in Blackboard you would like to learn more about.

Best of luck with your fall semester classes!

The Power of Learning Analytics

In the digital world we live in today, we all know the endless amount of data that is collected throughout the many interactions that occur in our daily lives. You can see the evidence of this type of data in a variety of ways, including Amazon’s recommendation feature that suggests what products you might find interesting based on your previous purchases and what similar customers also purchased, as well as the Netflix movie recommendations that are created based on your ratings of previous movies you’ve watched. You probably won’t be surprised to know that even one of the large, local companies (where I know I spend too much time and money) uses this technology as well. Wegmans uses the data from Shoppers Club Cards for a variety of purposes, including sending out coupons and making sure the right products are stocked based on store locations. In at least one case, Wegmans even contacted shoppers who purchased a certain brand of soup to let them know about a recall on the product. We all create these kind of data points every day without even noticing it. Companies have found a way to utilize this type of data for marketing, advertising and to create better products and services based on what they have learned about their customers. This strategy of data mining for business purposes has been going on for many years.

With the increased use of technology and web-based tools in education, students have also left data footprints as they travel through their educational activities online. This data often goes untouched by many educators; however, there is a growing trend to make better use of this data to increase student learning. Many educational technology companies are beginning to focus on better utilization of this data and enhanced reporting to allow educators to use the data in more effective ways.

When we hear “Khan Academy”, most people immediately think of a short video with a black background, annotated colorful text and someone talking through a complex math theory. This is a large part of what Khan Academy offers, but a key component of their overall design is the tracking of student progress through a series of these short videos and accompanying activities over time. This data is collected while the student watches videos and completes short practice questions and concept quizzes during and after each skill checkpoint. The data collected includes not only the answer to the quiz questions, but data on how many times a student has viewed a certain video, how long they spent on it, what time of day it was viewed and more. Check out this Khan Academy data reports page for a full list of the reports they offer to educators to track the progress of students.

Knewton is another company innovating new ways to utilize the vast amount of data created by students throughout their learning experiences online to create tailored and more personalized learning environments for each student. Check out the below video from Knewton’s website.

Many publishers, including Pearson, Houghton Mifflin and Wiley, partner with Knewton in order to apply their adaptive learning technology to the large amount of content that publishers have created over the years. The combination of resources, on one side focused on the analytics and the other on high quality content, create innovative, data driven tools that can quickly be used by students and instructors in a variety of course and content areas.

Now that we have discussed the type of data that is tracked and how it has been used in a few examples to create personal learning environments for students and valuable reports for instructors, you might be wondering, “How can I apply this idea to my own courses?”

First of all, both examples listed above, Khan Academy and Knewton, provide content that utilize this type of data collection and reporting to instructors in many disciplines. Check to see what might be available for you. Also, check with your publishers if there are other resources that might give you this type of information. If you are using other technologies in your courses, check to see what type of statistics the tool is tracking already on student usage and if the reports of this data might be useful to you.

If you use Blackboard in any way, even just to post a syllabus document, there is already data available at your fingertips. You can quickly see the last time a student has logged into your course, as well as if they clicked on the file that you uploaded for them to read before the next class. The amount of data available for you to view in reports increases with your use of the variety of Blackboard tools. For example, you can view how many times a user has clicked on a specific content area, as well as other tools like activity in discussion forums and groups. These reports provide the specifics on each student, as well as the common days and times that content is most used in your course overall. Combine this data with the results of a short quiz activity and you would be able to acquire similar data that is made available to users of Khan Academy resources.

Blackboard also provides tools for early warning messages based on predetermined rules. If a student meets a certain criteria identified by the instructor as “at risk” on a certain assignment or overall in the course, the instructor is notified and is prompted to send a notification to the student directly. Blackboard also provides adaptive release tools that allow instructors to control the release of certain course material based on the successful completion of previous items.

A variety of other commonly used tools at Fisher provide similar types of data and reporting features to instructors. Echo360, for example provides instructors with a easy to view report that includes data on how many times a video recording has been viewed by unique viewers and overall, as well as a heat map of the video indicating points that have been viewed most frequently by all students. These points may indicate areas of confusion where content was viewed multiple times. These statistics can be very handy if you are teaching with the flipped classroom model, which we discussed in a previous post, in order to know who is viewing each recording prior to class and to identify areas that might need further clarification in the next face-to-face class session with the larger group. Below shows an example of the Echo360 statistics.

Echo360 stats

The data provided by a variety of educational tools can be used by instructors to paint a more accurate picture of where students are spending time, where they may be struggling with content and students who may not be fully engaged with the course. The ability to access and utilize this data can help all instructors make more informed decisions in communications with individual students, as well as for course development and planning overall.