eLearning Weekly

Using Metrics That Matter October 24, 2008

When it comes to measuring the impact of training events (ex. instructor-led classes and online courses), there are two main types of metrics: transactional data and user data. Transactional data is all about numbers. For example, it tells you how many people completed a class in a given time period. While this data is sometimes mildly interesting, it doesn’t tell you if your students learned anything. That’s where user data comes in. User data allows you to dig a level deeper to see the true impact of your training event. This is where we get into Kirkpatrick’s levels of evaluation, the Success Case Evaluation Method (PDF), and other classification systems. User data is more likely to provide us with information to help us make decisions. User data is actionable data. It shows us where we are succeeding and where we are failing. It helps us realize what we need to change. The tough part is that it’s difficult to collect accurate user data.

Over the past few months, I’ve been researching a product from KnowledgeAdvisors called Metrics That Matter®. To put it simply, Metrics That Matter (from here on, MTM) is a system that integrates with your LMS to automate the evaluation process for training events. It records the evaluation data and then it gives you several ways to slice, dice, benchmark, and review the data once it is in the system. (The back-end of MTM provides a customizable dashboard, scorecards, and 100+ reports.)

In the past, I’ve used paper and online surveys to collect evaluation information, but I will admit that it never felt right. I wasn’t getting the information I needed, which made it tough to make decisions. This is where MTM steps up to the plate: KnowledgeAdvisors has done a great job of not only building a system that can survey learners and collect evaluation data, but they can help you develop sound evaluation instruments that you can rely on to provide actionable data. Forget "smile sheets" – they have created some serious evaluation sheets that collect great data. We haven’t bought or implemented this system, but I really like what I’ve seen so far.

Here’s how MTM works, in a nutshell:

1. An individual takes an online course or attends a live class.
2. (That night, some magic voodoo takes place between your LMS and MTM to communicate the names of the learners that completed classes.)
3. The next day, MTM sends an email to the person requesting that they fill out an evaluation.
4. The person fills out the evaluation. (MTM can be configured to send out reminder emails to the learner if they don’t fill out the evaluation.)
5. The system collects the evaluation data from the person.
6. You are immediately able to view this information using the dashboard, scorecards, and reports. You can also do blind comparisons of other organizations (benchmarking) to see how your organization stacks up.
7. Follow-up surveys can automatically be sent at whatever interval you prefer (ex. 30 days, 90 days, etc.) to collect additional evaluation information.

And I know I’m leaving off many of the features of MTM, because I’m not that familiar with everything it can do (yet). Surf on over to the KnowledgeAdvisors site to learn more.

Additional MTM information:

• MTM walkthrough
• MTM Product Sheet (PDF)