Theory v. Reality: There’s Room for Both

learning mapThis is a post from a Boise State student (I am an alum) in response to a webinar I participated in during 2010.  I welcome the conversation and have inserted my comments in blue.

“I feel like I should preface my write-up below by explaining that while I was listening to the webinar and reviewing the transcripts, I really was looking for an aha moment when I could apply what I had learned in class with what was being done in the field. While I had “my aha moment”, in that I was able to make this connection, it was not in the manner I was expecting. I hope I don’t seem like I am being too critical, as there was a lot of good information presented in the webinar. I am choosing to share this as my “aha” moment, because I think it makes a good correlation between the material learned in class and how it may be applied in the field. Also, not sure if I needed to add references, but since I can use the practice, I made an attempt. 

Elliot Rosenberg touched on several topics during his webinar with Dr. Steve Villachica on October 11, 2010. These topics included discussions on cross generational trainings and the fact that in 2020 there will be five generations in the workplace. However, the part that captured my attention was his discussion on training evaluations. Typically I find it nice to see how class material applies to the everyday world, but I was really confused and disappointed in what Rosenberg had to say about training evaluations. Here is the excerpt I am referring to in which Rosenberg describes the use of training evaluations (as copied from the webinar transcript): 

If Level Ones is a measure, I’m right there. Even Level Twos—but honestly, I don’t really care about level ones and twos—you have to understand it to see that if your project is at least on target, what really matters is the level threes and fours. And let’s not put all the stock into Kirkpatrick’s Levels of Evaluation; it’s possible lots of variable could impact it. The bottom line is, for me, are we making more sales? Are we reducing our employee turnover, and are we increasing customer satisfaction? Those are the three measures that guide the success of our efforts. It has nothing to do with levels, quite frankly—it’s all about the business. 

I feel like on one hand he is explaining how he uses training evaluations in his organization which follow Kirkpatrick’s theory, but then he makes statements like, “I don’t really care about level ones and twos..” “Let’s not put all the stock into Kirkpatrick’s levels of evaluation…” He identifies three questions he wants to answer at the end of a training session, but then follows up with the statement that they have nothing to do with levels. This is where I disagree. Rosenberg describes his bottom line as answering the following 3 questions: 

  1. Are we making more sales? 
  2. Are we reducing employee turnover? 
  3. Are we increasing customer satisfaction? 

I would argue that by answering the three questions above he is in fact completing a Level 4 results evaluation. Chyung (2008) describes a Level 4 evaluation as a look at results on the organization and examines if the training resulted in a behavior change that positively affected the organization. If the training allowed the sales managers to make more sales, reduced employee turnover and increased customer satisfaction, I would think that the training positively affected the organization. On the other hand if you had “No” answers to any of the questions – you might think there was a negative impact on the organization. Either way, the results from these three questions still represent a Level 4 Results evaluation since they identified how the organization was impacted by the training.”

Kirkpatrick’s Levels of Evaluation provide the framework from which ISD professionals can measure their work and, hopefully, impact.  However, without context for those measurements, they will mean nothing for your client.  

Now, in my case, my department is not funded by HR but, rather, by the sales organization.  While HR might find it OK to say that participants liked the training (L1) and they were able to pass a knowledge test after the class (L2), for a sales organization there is no link to their objective which is to make more sales.  And, if the learning organization can’t impact those results, why have them around?

If the implication is that I actually achieve a L4 result if I can impact all three questions, that might not work with someone who is skilled in ISD.  A nearly impossible task of isolating all of the variables that could impact L4 results is critical.  However, if I could imply that training had something to do with increased sales, reduced turnover, and improved customer satisfaction then we could have that discussion.  If you believe the 70/20/10 rule for formal vs informal learning (only 10% of learning is accomplished via a formal setting (ILT, for example), then it is only reasonable to assume that the other 90% is accomplished outside of the formal setting (20% from coaching and observations and the other 70% from simply gaining experience by doing the job), then you can see the diminished role that Kirkpatrick’s concepts have on learning, over the long term.  Even if you applied this line of thinking to the academic world, you should draw the same conclusions (IMHO).  

You have to start with Kirkpatrick but you must not end there.  Thoughts?