The Kirkpatrick Four Levels: A Fresh Look After 50 Years 1959

A Kirkpatrick White Paper

April 20April 200909

The Kirkpatrick Four Levels:

A Fresh Look After 50 Years

1959 - 2009

By Jim Kirkpatrick, PhD

and Wendy Kayser Kirkpatrick

April 2009

The Kirkpatrick Four Levels: A Fresh Look After 50 Years

A Golden Anniversary Celebration

The Kirkpatrick Four Levels are turning 50 this year! November

1959 marked the first time Don Kirkpatrick published his

thoughts on training evaluation with a series of four articles in the

Journal of the ASTD. Don will tell you he did not call them

levels, and he did not coin the phrase Kirkpatrick Four Level

Evaluation Model. He did, however, use the legendary four

words that comprise the foremost training evaluation model

today: Reaction, Learning, Behavior, and Results.

On the anniversary of the four levels, we thought it would be

appropriate to first honor Don and his work, and second to share

with you how the four levels have both evolved and remained

relevant over the last half century.

The Day of Reckoning Has Arrived

When we conduct workshops on the Kirkpatrick Model all over

the world, we find a surprising number of misconceptions about

it. After 50 years, we figured it was time to set the record straight.

Rather than attempt to clarify the many misunderstandings about

the four levels, we will focus on the salient elements of the

model, and show you how to leverage its power at a time when

the day of reckoning has arrived for the entire learning industry.

Don’s incredible vision is evident in a quote he obtained from a

colleague back in 1959:

“Managers, needless to say, expect their manufacturing

and sales departments to yield a good return and will go

to great lengths to find out whether they have done so...

likewise, training directors might be well advised to take

the initiative and evaluate their programs before the day

of reckoning arrives.”

“The Experimental Evaluation of Management Training:

Principles and Practice,” Daniel M. Goodacre III, The

B.F. Goodrich Company, Personnel, May 1957

This statement represents what the ultimate intent of the four

levels of evaluation was then, and is today: to show the business

value and worth of training.

Donald L. Kirkpatrick, PhD

The ultimate intent of

the Kirkpatrick Four

Levels was then, and is

today:

To show the business

value and worth of

training.

The Kirkpatrick Four Levels: A Fresh Look After 50 Years

Before unveiling our new Kirkpatrick Model visual, we will

present what we consider to be the five foundational principles.

The following are the cornerstones of this model as a

differentiator from others, and the keys to successful

implementation of training programs. Some will likely be a

surprise to you.

Five Foundational Principles

1. The end is the beginning

Most learning professionals have heard of the four levels, and

many can recite them. But relatively few know how to effectively

get beyond Level 2. When presenting these concepts to groups

of professionals, we often refer to current evaluation practice as

“smile sheets (L1), pre and posttests (L2), and hope for the best

(L3 and L4).” Most of our workshop participants know exactly

what we are saying.

In 1993, Don wrote his first book on the Four Levels to explain

the entire model (which is, and has always been, more

comprehensive than four simple levels). On page 26 of

Evaluating Training Programs: The Four Levels (1

Edition,

Berrett-Koehler, 1993), Don writes:

“Trainers must begin with desired results and then

determine what behavior is needed to accomplish them.

Then trainers must determine the attitudes, knowledge,

and skills that are necessary to bring about the desired

behavior(s). The final challenge is to present the training

program in a way that enables the participants not only

to learn what they need to know but also to react

favorably to the program.”

It is unfortunate that the message above has been missed by

many learning professionals. For decades, practitioners have

attempted to apply the four levels after a program has been

developed and delivered. It is difficult, if not impossible, to create

significant training value that way.

All this is said to distinguish the development of the plan to build

effective programs and evaluation methodology (which starts

with Results, as shown to the right) from the actual data

collection, which does begin with Level 1 and works forward

through Level 4.

Kirkpatrick Four Levels

Level 4:

Results

To what degree

targeted outcomes

occur, as a result of

the learning event(s)

and subsequent

reinforcement.

Level 3:

Behavior

To what degree

participants apply

what they learned

during training when

they are back on the

job.

Level 2:

Learning

To what degree

participants acquire

the intended

knowledge, skills, and

attitudes based on

their participation in

the learning event.

Level 1:

Reaction

To what degree

participants react

favorably to the

learning event.

The Kirkpatrick Four Levels: A Fresh Look After 50 Years

2. Return on Expectations (ROE) is the ultimate indicator

of value

What normally happens when executives ask for new training?

Despite what they might say, most learning professionals jump to

the task by all-too-quickly retreating to their departments and

commencing to design and develop suitable programs. While a

cursory needs assessment may be conducted, it is rarely taken

to the point that expectations of the training contribution are

completely clear.

Much of the training and consulting that the Kirkpatricks do

involves helping learning professionals negotiate key business

stakeholder expectations. This is a process where learning

professionals ask questions to clarify and refine the expectations

of the key business stakeholder so that they are satisfying to the

latter and realistically achievable to the former. Learning

professionals then need to convert those rather generic

expectations into observable, measurable success outcomes by

asking the question, “What will success look like to you?” Those

success indicators then become the Level 4 outcomes – the

targets – to which you can sharply focus your collective efforts to

accomplish return on stakeholder expectations.

3. Business partnership is necessary to bring about

positive ROE

Note in the last sentence, we wrote, “collective efforts.” Contrary

to training myth and deep tradition, we do not believe that

training events in and of themselves deliver positive, bottom line

outcomes. Much has to happen before and after formal training.

Learning professionals need help to do it, and that help needs to

come from our business partners. Historically, the comfort zone

performance processes.

Before training, learning professionals need to partner with

supervisors and managers to prepare participants for training.

Supervisors should explain to their direct reports what they will

arn in training, why it’s importa

of learning professionals has been Levels 1 and 2. We believe

this is one of the major reasons why many learning professionals

spend almost all of their time there. But the actual execution of

learning programs and overall corporate strategy occurs

primarily at Level 3.

2008 research by Bersin and Associates shows us that as much

as 70% of employee learning occurs on-the-job in many different

ways. Thus, we need to reinvent our roles by becoming experts

in and deeply involved with learning that extends beyond our

normal comfort level, or we run the risk of losing our positions.

Therefore, not only do we need to call upon our business

partners to help us identify what success will look like, but we will

need a cooperative effort throughout the learning and

nt, expectations for actions

during and after the training event, and the kind of support they

will receive throughout the process.

Return On

Expectations (ROE):

Focusing training and

reinforcement efforts on

the stated bottom-line

expectations of

business partners, with

the goal of delivering

the key outcomes /

results they expect.

Josh Bersin and Associates, 20

Where Learning Takes Place

Prior to

being

"trained":

20%

During

"training":

10%

On the job:

70%

The Kirkpatrick Four Levels: A Fresh Look After 50 Years

Even more critical is the role of the supervisor or manag

the training. They are the

er after

key people to reinforce newly learned

knowledge and skills through support and accountability. The

degree to which this reinforcement and coaching happens

directly correlates to improved performance and positive

outcomes.

It is often said that a picture is worth a thousand words. To the

right is one of our signature pictures that we believe accurately

illustrates the concept of business partnership – with the

business on the left, training on the right, and the bridge as our

way of crossing over.

This is no easy sell to many executives and managers, because

we have all been in cahoots to perpetuate the myth that good

training leads to positive results on its own. Going forward, we

must make a business case to our key stakeholders that we

need them to work with us at different points along the learning

and performance continuum. Business partnership, not the

delivery of training programs, is the secret to positive outcomes.

4. Value must be created before it can be demonstrated

A colleague of ours, Sandy Almeida, MD, MPH, recently sent us

a summary of her research that identified statistical correlations

between the four levels. There is good correlation between

Levels 1 and 2, in that positive learner engagement led to a

higher degree of learning. Similarly, the correlation between

Levels 3 and 4 was significant – if employees consistently

perform critical on-the-job behaviors, individual and overall

productivity increased. There was not, however, a significant

correlation between Levels 2 and 3. In short, Sandy states that

even providing excellent training does not lead to significant

transfer of learning to behavior and subsequent results without a

good deal of deliberate and consistent reinforcement.

Consider those findings in combination with a study conducted

by Dr. Brent Peterson at Columbia University in 2004. He

compared the amount of time that is spent developing training

and related activities, and what actually contributes to learning

effectiveness. He found that the typical organization invests 85%

of its resources in the training event, yet those events only

contributed 24% to the learning effectiveness of the participants.

The activities that led to the most learning effectiveness were

e training event.

we are putting most of our time into

g training (L1 and L2) and

ter of the benefit. And we are

e follow-up activities that

nge and subsequent results

ng programs to deliver.

Activities Contributing to

Learning Effectiveness

Typical Learning Investment

Dr. Brent Peterson,

Columbia University, 2004

Pre-

Work

26%

Learning

Event

24%

Follow-

Up 50%

Pre-

Work

10%

Learning

Event

85%

Follow-

Up 5%

follow-up activities that occurred after th

What does this mean? That

designing, developing and deliverin

only getting about one-quar

spending virtually none of our time on th

translate into positive behavior cha

(L3 and L4) that we intend our traini

The Kirkpatrick Four Levels: A Fresh Look After 50 Years

were

rned

h, and an ineffective culture of follow-up and

That is one of the biggest reasons why a comprehensive ASTD

study in 2006 identified the fact that more than 70% of training

failure comes after training is completed. The major factors

that employees didn’t get a chance to apply what they lea

oon enougs

coaching existed. Bottom line? It is essential that learning

professionals redefine their roles and extend their expertise, their

involvement, their influence, their impact, and their value into

Levels 3 and 4 before the day of reckoning arrives.

5. A compelling chain of evidence demonstrates your

bottom line value

What do we mean by “corporate jury”? We believe that each of

us – individuals, departments, our entire industry – is on trial,

accused of “not bringing enough value to the bottom line to

justify our costs.” And in all cases, there is either an individu

group of business leaders who are sitting in judgment of our

al or

erformance and our impact to the business.

tribution

e last

Our friend and colleague, Nick DeNardo from Edward Jones,

refers to data collected at Levels 1 and 2 as “consumptive

metrics” and data at Levels 3 and 4 as “impact metrics.” If we

are only providing our jury members with numbers of program

and attendees (L1 and L2 data) we are basically shouting, “Lo

how much this is costing you!” and, thus, have a bulls eye

painted on our backs. If, on the other hand, we work our wa

backwards from Level 4 and include L3 and L4 metrics,

particularly for mission critical programs, we are saying just as

loudly, “Look how much value we are bringing.”

A chain of evidence consists of data and information that

sequentially connects the four levels and shows the con

Consumptive metrics:

Level 1 and Level 2

data indicating how

much time and how

many resources have

been invested in

training

Impact metrics:

Level 3 and Level 4

and tangible results that

training has delivered

data indicating the valu

learning has made to the business. We recently wrote a book

entitled Training On Trial (AMACOM, 2009) that details these

and other principles of what we call the Kirkpatrick Business

Partnership Model. We talk about how we do not believe in

attempting to isolate the impact of training in order to prove our

value. Instead, we advocate presenting a chain of evidence that

illustrates the value of the entire business partnership effort.

The “New” Kirkpatrick Model

It is now time to pull the wraps off not only the true Kirkpatrick

model, but also the complete model. While it contains the same

ideas that Don (and we) have been communicating over th

50 years, we felt a more complete illustration of the model would

help both learning professionals and their business partners to

create more training value together.

The Kirkpatrick Four Levels: A Fresh Look After 50 Years

Training Needs Analysis: The End is the Beginning

a formal connection between the

resolving a business problem, or taking

nd is the beginning because you are

mine what success will look like (L4), and then

. We identify “critical behaviors” as the few,

ir jobs in order to bring about the

e the actions and processes that are

programs, etc.) that are designed and

heir critical behaviors.

for participants to be able to perform

address the learning environment and

conditions that will support learning effectiveness and enjoyment (L1). We define “environment” as the

onditions” as prerequisites for ultimate success (i.e., corporate

ulture, participant readiness, etc.).

ctive, it is time to get your instructional

een identified as necessary to bring

is is als the time to determine the best

easurement plan you will use. Next

cted earlier. This is also

el 1 and Level 2. This is probably familiar territory for most

arning professionals.

Start in the upper left corner of the model. Note that it starts with

business and learning, whereby a request is made to work toward

advantage of a market opportunity. As mentioned already, the e

working from L4 to L1 in the first four steps. You will deter

subsequent critical behaviors and organizational drivers (L3)

targeted behaviors that program graduates must apply back at the

identified outcomes. “Organizational drivers”, on the other hand, ar

carried out by others (e.g., coaches, mentors, peers, software

implemented to reinforce the participants to perform t

Next, we deal with the required KSAs – or competencies – needed

their new on-the-job behaviors (L2). Finally, we subsequently

venue and modality for training, and “c

Once the four levels have been dealt with from a planning perspe

designers working on the actual program or programs that have b

about the targeted outcomes. Importantly (and often forgotten), th

way to measure each of the four levels, and build the tools and m

comes the delivery of the program(s), in whatever modality and venue you sele

hen formal evaluating begins with Lev

The Kirkpatrick Four Levels: A Fresh Look After 50 Years

Entering “New Territory” For Many Learning

Professionals

drivers, and preliminary outcomes (L4). Use the data to make

sure that behaviors and drivers are meeting standards to ensure

positive outcomes, and provide your stakeholders – jury

members – with a periodic dashboard to show them initiative

progress.

During this step is also the time to partner with managers and

supervisors who are responsible for ongoing coaching and

reinforcement. Offer any assistance you can and follow up with

them frequently. While this step in the process looks like one

small box, it actually represents months or even years of effort.

This is where you will see 50% of learning effectiveness occur.

It’s where you can prevent 70% of the potential learning failures

if done properly. In short, we believe “ongoing reinforcement and

The diagram (now moving back from right to left) starts to move

into new territory for many learning professionals at this point.

The next step is called “initiate ongoing reinforcement and

monitoring.” This has everything to do with what happens after

learning events, and deals with what we talked about under the

principle of creating value. It is true that what gets measured,

gets done, and the best suggestion we have here is to monitor

and measure the critical behaviors (L3), the organizational

monitoring” is among the most critical steps in the entire model!

The gray box on the left side of the diagram is a reminder to

analyze your findings along the way, and make positive

adjustments where you can to eliminate snags at Levels 1, 2, or

3. If you determine that conditions are limiting the ability of

associates to learn or apply the information, now is the time to

call attention to this and create a backup plan. There is no value

in going through the entire process to find out at the end that

intended results aren’t achieved because of something that could

have been corrected months earlier!

Building Your Chain of Evidence

pefully) this value-filled and

and compelling presentation.

When you are satisfied with your preliminary findings at each of

the levels, it’s time to prepare final data and testimonials that will

make up your “closing argument” to the jury. Presentation of

your chain of evidence goes from right to left, starting with Level

1 and moving step by step to create your value story through

Level 4, Results. Ultimately (and ho

evidence-based story will lead to a positive verdict: one where

your stakeholders say that beyond a reasonable doubt, the

overall learning initiative has successfully contributed to the

intended results. For mission critical cases (programs), be sure

to practice how you will make your oral presentation to your jury.

It is always sad for us to see training departments that do great

work fall short of impressing their stakeholders because they

didn’t make an effective

Ongoing

reinforcement and

monitoring

• May reflect months

or years of effort

• 50% of learning

effectiveness occurs

here

• 70% of potential

learning failures occur

here if not done

properly

The Kirkpatrick Four Levels: A Fresh Look After 50 Years

The Four Levels Are Not Just For Training Evaluation

t the Four Levels are used only to

be used for this purpose (and most

ally an integral part of every phase of a

show that programs begin by

en each of the other thre

odel. In fact, all fo

Another commonly held misconception we hope to debunk is tha

measure the impact of training after it is complete. While they can

typically are by learning professionals), the Four Levels are actu

training initiative, from beginning to end.

We have illustrated this by shading the new Kirkpatrick Model to

considering what Results the business wants to accomplish. Th

considered. And this pattern continues all the way through the m

e levels is

ur levels are addressed

at almost every step in the process! To reflect this reality, we are

Kirkpatrick Four Level Evaluation Model, as this is, to some degre

name of the model is simply the Kirkpatrick Model.

no longer referri

e, a misnomer. As yo ew

ng to the model as the

u can see, the n

The Kirkpatrick Four Levels: A Fresh Look After 50 Years

The Window Washers

y taxi-

atching post (but I did snap his picture as I walked away).

The next day, I was in the country of Brunei, conducting another

workshop. During a break, I wandered over to this young man

and asked him,

“What is your job here at the resort?”

Chai (who later introduced himself to me by name), stopped

what he was doing, walked over to me, looked me in the eye and

said;

“I am part of the team that creates exceptional

experiences for our guests.”

After I picked my jaw off the ground, we carried on quite a

conversation, which included Chai asking me about my stay, if I

had everything I needed, what I thought of his grounds, and what

I thought of his country.

On my way back to the U.S., I thought about Window Washer #1

and Chai. They were both about the same age, so why were

their answers to the same question so different? Of course, my

thoughts went to training and reinforcement. I never found out

what kind of training Window Washer #1 received (he probably

took a job down the street when he was offered a modest raise).

Chai, on the other hand, told me about his orientation, training,

and the coaching and encouragement he received from his

supervisor. He was taught that he was in training in order to

learn, perform, enhance his career possibilities, and ultimately

serve as an ambassador for his resort and his country.

short, Chai received training and reinforcement that modeled

hat we have presented here. While his windows were no

s, he knew that the purpose of

ose were tied to the bigger

ows. He made me

to return.

Window Washer #1

Chai

et’s put some real faces on what we have talked about. In the L

right margin there are photos of two men who seemingly have

the same job. The first is a man that I (Jim) saw in front of a hotel

in Asia. While I was waiting for a taxi, I went up to him and asked

him,

“What is your job here at the hotel?”

Without looking at me, he answered,

“I wash windows.”

ince we had little else to talk about, I went back to mS

cleaner than Window Washer #1’

aining and development and his purp

icture. So he did more than just clean wind

feel like a welcome guest so I would want

The Kirkpatrick Four Levels: A Fresh Look After 50 Years

An Urgent Call to Action

o there you have it. A new visual for the Kirkpatrick Model that

ith the recent changes in the economy and the world, there is

no bette the learning

dustry. Allow me to explain with a personal story. We have a

f our home. As I (Jim)

as walking up the path shortly after we were married, I noticed

the form the footprints of (I assume) their

mall child, and carved his or her initials into the sidewalk.

use,

sked Wendy where a screwdriver was, and proceeded back

You, ho rk in the learning

dustry. During this time of economic challenge, and when our

–

ble challenges. You have the unique opportunity to

rovide significant answers for them – and to carve out a new

learning tners.

Fortuna ’t last forever. This

reates a limited time frame and urgency for you to take action

e says it all:

ning

ng us

we hope better conveys the entire process and essence of what

Don Kirkpatrick designed 50 years ago. The sound, timeless

principles upon which it is based. And now, a challenge and

opportunity for you.

r time to create a new standard within

cement walk leading to the front door o

er owners depressed

Upon seeing this “legacy” left for all to see, I went into the ho

outside to add my own mark. Well, obviously the cement had

dried since 1995. I was too late to leave my mark on the front

walk.

wever, are not too late to leave your ma

industry is under fire, the good news is that business leaders

our jury members – are looking everywhere for solutions to their

incredi

legacy by becoming true strategic business par

tely, these economic times won

before the cement dries (or the rock bridge collapses).

Following one of our training sessions, I received an e-mail from

one of the participants. Sh

“This was one of the most insightful and worthwhile

training events that I’ve taken in awhile. I look forward

reading more of your work – your father and you are

very inspiring. Too often we think “training design and

development” and we don’t spend nearly enough time in

evaluation at the beginning. This training was so eye-

opening – you made a “Brunei Window Washer” out of

me.”

Kat Siedlecki, Booz Allen Hamilton, OS Lear

Team

hanks, Don, for a very useful 50 years. Thanks for showiT

a path to creating true value to the business world, and the

people we serve. Now it is up to all of us to heed your warning,

and effectively apply what you and others have taught us.

The Kirkpatrick Four Levels: A Fresh Look After 50 Years

most sought after speakers in the

w.linkedin.com/in/donaldkirkpatrick

About Don Kirkpatrick

Don Kirkpatrick, PhD, is professor emeritus at the University of

Wisconsin. He is the creator of the Kirkpatrick Four Level

Evaluation Model, the foremost training evaluation tool in the

world. He is one of the

workplace learning and performance field.

Don is the author of numerous books on training evaluation and

HR topics.

Don is a past president of the American Society for Training and

Development (ASTD). He has been honored with the Lifetime

Achievement in Workplace Learning and Performance awa

from ASTD, and is a member of the HRD Hall of Fame of

Training Magazine.