jump to navigation

Using Metrics That Matter October 24, 2008

Posted by B.J. Schone in eLearning.
Tags: , , , , , , , , , , ,
trackback

When it comes to measuring the impact of training events (ex. instructor-led classes and online courses), there are two main types of metrics: transactional data and user data. Transactional data is all about numbers. For example, it tells you how many people completed a class in a given time period. While this data is sometimes mildly interesting, it doesn’t tell you if your students learned anything. That’s where user data comes in. User data allows you to dig a level deeper to see the true impact of your training event. This is where we get into Kirkpatrick’s levels of evaluation, the Success Case Evaluation Method (PDF), and other classification systems. User data is more likely to provide us with information to help us make decisions. User data is actionable data. It shows us where we are succeeding and where we are failing. It helps us realize what we need to change. The tough part is that it’s difficult to collect accurate user data.

Over the past few months, I’ve been researching a product from KnowledgeAdvisors called Metrics That Matter®. To put it simply, Metrics That Matter (from here on, MTM) is a system that integrates with your LMS to automate the evaluation process for training events. It records the evaluation data and then it gives you several ways to slice, dice, benchmark, and review the data once it is in the system. (The back-end of MTM provides a customizable dashboard, scorecards, and 100+ reports.)

In the past, I’ve used paper and online surveys to collect evaluation information, but I will admit that it never felt right. I wasn’t getting the information I needed, which made it tough to make decisions. This is where MTM steps up to the plate: KnowledgeAdvisors has done a great job of not only building a system that can survey learners and collect evaluation data, but they can help you develop sound evaluation instruments that you can rely on to provide actionable data. Forget "smile sheets" – they have created some serious evaluation sheets that collect great data. We haven’t bought or implemented this system, but I really like what I’ve seen so far.

Here’s how MTM works, in a nutshell:

  1. An individual takes an online course or attends a live class.
  2. (That night, some magic voodoo takes place between your LMS and MTM to communicate the names of the learners that completed classes.)
  3. The next day, MTM sends an email to the person requesting that they fill out an evaluation.
  4. The person fills out the evaluation. (MTM can be configured to send out reminder emails to the learner if they don’t fill out the evaluation.)
  5. The system collects the evaluation data from the person.
  6. You are immediately able to view this information using the dashboard, scorecards, and reports. You can also do blind comparisons of other organizations (benchmarking) to see how your organization stacks up.
  7. Follow-up surveys can automatically be sent at whatever interval you prefer (ex. 30 days, 90 days, etc.) to collect additional evaluation information.

And I know I’m leaving off many of the features of MTM, because I’m not that familiar with everything it can do (yet). Surf on over to the KnowledgeAdvisors site to learn more.

Additional MTM information:

Comments»

1. Measuring the impacts of training: transactional and user data | Workplace Learning Today - October 29, 2008

[…] Using Metrics That Matter | eLearning Weekly | B.J. Schone | 24 October 2008 […]

2. Bill - March 17, 2009

as a client of metrics that matter, I can say it is the most powerful system I use to do three things: 1) reduce wasted learning, 2) reduce administrative time (i used to use paper evaluations, too🙂, and 3) provide better training to my division. We are better able to defend out learning budget since we have implemented the system, even in these critical economic times.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: