Category: Evaluate Training
Last night I had the honor of speaking at the ISPI Atlanta chapter with Denise Traicoff from the Centers for Disease Control . I did my usual speech on my Predictive Evaluation Model (PE) and Denise followed on how she and her team had implemented PE on one of their programs. She told the story of how her team had been given the assignment to teach leadership & management skills to groups of volunteers from around the world to go in a eradicate polio in countries. It was a compelling story that kept all in attendance transfixed on her talk.
Sandi Brown, Dave Basarab, and Denise Traicoff
She shared how on her flight to the World Health Organization to begin the needs analysis she was reading my book . She thought that due to nature of her course “the world deserved more than just happy sheets”. As she read, she got more and more excited about PE and shared that the could provide more rigor to her course measurement. What made me proud was that she implemented Intention and Adoption evaluation from just reading the book. What ever author wants.
Denise and her team, including Sandi Brown (pictured), have now successfully implemented PE on three sessions and will continue to use it. But PE is not the story here. It is Denise and her team – what they have done. Even more fun, was that last night was the first time we had met face-to-face even though we both live in Atlanta. I am honored to be associated with these talented people. So in a small way sometime we do make a difference. I had my moment last night. Thank you so much Denise.
When you embark on creating a training evaluation plan one of the first steps is to determine the process or how the evaluation will be performed. Whatever evaluation method you elect to use (Predictive Evaluation, The Kirkpatrick Model, The Success Case Model), the process to implement needs to be defined and accepted. The process I use is:
The Plan phase is the starting point of the training evaluation. Planning an evaluation is specifying what and how you’re going to evaluate by reviewing the program, developing the project’s mission, creating the evaluation objectives & deliverables, developing the questions the evaluation will answer, recording assumptions, documenting boundaries, identifying stakeholders, developing the strategy, estimated costs & timelines, and performing risk assessment. From this a detailed evaluation project plan with milestones is created.
- Developing evaluation procedures involves taking the approved evaluation plan from phase 1 and drafting the techniques, procedures, and instruments necessary to implement the plan.
- This phase of the training evaluation of entails ( collect phase) gathering data to address the evaluation questions created in the evaluation plan.
- In the analyze phase the meaning of collected data are interpreted. This phase uses the collected data to determine the conclusions for which the data supports and the amount of support the data supplies for, or against, those conclusions. Quantitative & qualitative data analysis procedures are employed.
- The purpose of this phase ( report ) is to create the vehicles to communicate training evaluation information to stakeholders so that they can make use of the evaluation information for decision-making or purposes of accountability. This phase will generate an evaluation report and an executive dashboard detailing the results of the evaluation and the recommendations for the city to maximize the program’s impact.
As always, I look forward to you comments and ways to improve this approach.
Failure to do proper training evaluation planning causes poor efforts that usually include rework, the project being over budget, and delays. The Plan phase is the starting point of the training evaluation. Planning an evaluation is specifying what and how you’re going to evaluate by reviewing the BOC-1 program, developing the project’s mission, creating the evaluation objectives & deliverables, developing the questions the evaluation will answer, recording assumptions, documenting boundaries, identifying stakeholders, developing the strategy, estimated costs & timelines, and performing risk assessment. From this a detailed evaluation project plan with milestones is created.
I begin all my evaluations with with a detailed plan in which I include a steering committee to build it. The steps I use with the steering committee are:
- Determine the evaluation methodology
- Understand the course being evaluated
- Mission of the evaluation
- Objectives & deliverables
- Questions the evaluation will answer
- Assumptions
- Boundaries
- Stakeholders
- On-the-job behaviors
- Beliefs
- Strategies (data collection, data analysis, reporting)
- Timeline (project plan & key milestones)
Over the next few weeks I will post how to do each step. I hope you find them useful and encourage you to give me your thoughts.
Many times I have T&D professionals ask me the steps I take when evaluating programs. Here are the steps I follow:
But this is only the framework for conducting an evaluation, you should create an Evaluation Project Plan . This is a formal, approved document used to guide both project execution and project control. The primary uses of the Evaluation Project Plan are to document planning assumptions and decisions, facilitate communication among stakeholders, and document approved scope, cost, and schedule baselines.
Currently I am about to begin Level 3 Evaluation project for a client. Here is the project plan we just agreed to:
This gives you an idea of the time per task/milestone and the activities (tasks) for each step in the framework.
With the evaluation plan in place and approval to proceed, implement the evaluation by using the following sequence.
- Choose the course you wish to evaluate ; it can be a new course design or a course that already exists. Note: this should have been done when you developed the evaluation plan.
-
Review the course to understand fully what it is, what it is supposed to teach, how participants learn the material, what business issues it addresses, who is to attend, how they attend, what pre-course preparations are in place, what post-course support mechanisms exist, who are the sponsors, what sponsors see as the purpose for the course.
- Form a committee to sponsor the evaluation and provide oversight while the evaluation is under way.
- Create evaluation protocols, develop the data collection instruments, and create models to analyze collected data for the evaluation. Developing evaluation procedures involves taking the approved evaluation plan and drafting the techniques, procedures, and instruments necessary to implement the plan.
- Collect quantitative & qualitative evaluation data as prescribed in the plan and using the data collection instruments. This phase of the training evaluation of entails gathering data to address the evaluation questions created in the evaluation plan. The goal is to obtain the right kind at the lowest-cost to answer the evaluation questions.
- Analyze the evaluation data and draw truthful conclusions and recommendations . In the analyze phase the meaning of collected data are interpreted. This phase uses the collected data to determine the conclusions for which the data supports and the amount of support the data supplies for, or against, those conclusions. Quantitative & qualitative data analysis procedures are employed.
- Create an Evaluation Report and Executive Dashboard. The purpose of this phase is to create the vehicles to communicate training evaluation information to stakeholders so that they can make use of the evaluation information for decision-making or purposes of accountability. This phase will generate an evaluation report and an executive dashboard detailing the results of the evaluation and the recommendations for to maximize the program’s impact.
Previous posts in this series:
Questions Evaluation will Answer
Elements in an Evaluation Plan
Evaluation Deliverables & Schedule
Coming up in this series of posts:
- Training evaluation and continuous improvement
- Training evaluation organizational readiness
As always, I would enjoy hearing your thoughts on this topic.
I am honored to have an new article just published in
In the article I discuss the elements of my Predictive Evaluation model . Watch this short video about the PE model.
To learn more about the model, take a look at my latest book:
In addition to the evaluation questions , other elements of the evaluation plan arise from stakeholder interviews and your research into the course. Each element that you should consider as part of the plan is shown below.
Course to be evaluated. List the title and purpose for the course. Provide a history evaluated (business reason) for the course and what it is intended to achieve. For new courses (ones yet to be developed), these data can be found in the needs analysis and/or instructional design documents. For existing courses,use the current training materials.
Purpose of the evaluation. Documents why the evaluation is being undertaken and how the information will be used (comes from stakeholder requirements). Example: Our approach is based on moving away from the typical measurement of activity or level-based measures in favor of focusing on value-driven continuous improvement efforts.This ensures that the training investment creates value and produces the behaviors and results that lead to the desired business outcomes.
Questions the evaluation will answer. These are the basic building blocks for the training evaluation. They are the questions that key stakeholders would like to see answered. Examples: Are participant intentions and beliefs upon program completion aligned with desired values? Are participants using (adopting) the skills on the job? Is the training program contributing to achievement of the business outcomes? If yes, to what degree? If no, why not?
Key Stakeholders. These are the individuals who have a vested interest in the evaluation information.
Once this is completed, it’s time to move on and look at the evaluation deliverables (next week’s post).
Previous posts in this series:
Questions Evaluation will Answer
Coming up in this series of posts:
- Evaluation deliverables
- Evaluation schedule
- Performing the evaluation
- Training evaluation and continuous improvement
- Training evaluation organizational readiness
As always, I would enjoy hearing your thoughts on this topic.
A training evaluation is successful when the needs of the stakeholders have been met. A stakeholder is anybody directly or indirectly impacted by the evaluation. As a first step, it is important to identify the stakeholders for your evaluation. It is not always easy to identify the stakeholders of an evaluation, particularly those impacted indirectly. Examples of stakeholders for training evaluation are as follows:
- The Head of the Training and Development organization
- Key executives from various business units or staff functions (note: these may be represented by a Steering Committee or other such body)
- Instructional designers, course developers, and instructors (internal or external)
- External training suppliers (to a lesser extent)
Once you understand who the stakeholders are, the next step is to establish their needs. The best way to do this is by conducting stakeholder interviews. The questions that you ask stakeholders can depend on (1) who you are talking to and (2) how they fit into the overall picture. When formulating stakeholder questions, ask yourself the following:
- What do you want to find out?
- How many people can you interview?
- What will you do with the data to help guide the evaluation?
- What is the underlying reason for asking a specific question?
Some questions that you may wish to use with the stakeholders are:
- What business issue(s) is this training going to help in solving?
- Why does this issue exist?
- Why do you want an evaluation?
- What kind of information would you like to see from the evaluation?
- What decisions will you make based on this evaluation information?
- What is your timetable for getting evaluation information?
- How would you like to see it and receive it?
- Who else should I talk to?
Take time during the interviews to draw out the true needs that create real benefits. Often stakeholders talk about needs that aren’t relevant and don’t deliver benefits. These can be recorded and set as a low priority. Once you have conducted all the stakeholder interviews and have a comprehensive list of needs—is to prioritize them. From the prioritized list, create a set of questions the evaluation will answer.
Previous posts in this series:
Coming up in this series of posts:
- Elements in an Evaluation Plan
- Evaluation deliverables
- Evaluation schedule
- Performing the evaluation
- Training evaluation and continuous improvement
- Training evaluation organizational readiness
As always, I would enjoy hearing your thoughts on this topic.
So you’ve decided that you’re going to do a training evaluation, and, yikes, it is your first one!
First, answer these questions:
- Will the evaluation be used in deciding whether or not to move forward with training?
- Why is the evaluation being done; that is, what decisions does the company want to make during the evaluation?
- Who are the key stakeholders who should receive evaluation reports? What do they want from the evaluation?
- What elements of a evaluation will provide the data to aid in the decisions that you and/or stakeholders want to make?
- When is the information needed?
- What resources are available to conduct the evaluation? Can a Steering Committee be formed to aid with the evaluation? Will internal resources be used or will you employ external evaluation resources?
- Is funding available? What projects will be delayed or not done if the evaluation is implemented?
- What existing evaluation practices are in place? Can they be leveraged and used in the evaluation? Will this evaluation replace some of these practices?
The answers to the questions attempt to get at the purpose of the evaluation and what is prompting it. Implementing your first training evaluation can be a bit intimidating, but, as with any effort, it gets easier and quicker as you do it. First, you need a plan of action—the Evaluation Plan . The key to a successful training evaluation is in the planning. Often, evaluation planning is ignored in favor of getting on with the evaluation work. However, many people fail to realize the value of an evaluation plan in saving time and money and preventing problems.
Coming up in this series of posts:
- Questions training evaluation answer
- Elements in an Evaluation Plan
- Evaluation deliverables
- Evaluation schedule
- Performing the evaluation
- Training evaluation and continuous improvement
- Training evaluation organizational readiness
As always, I would enjoy hearing your thoughts on this topic.
PE Intention Goals are the planned actions that we want participants to author during training as their personal transfer plan. If training is working as designed, we want participants writing actions that drive desired adoptive behaviors back into their everyday work patterns. Developing Intention Goals is relatively easy if the course has a set of well-written, performance-based learning objectives. They refer to training outcomes that specify the information and/or skills to be mastered and what participants do to demonstrate mastery. Performance-based learning objectives manifest themselves in training in the forms of content modules and skill-practice exercises. Therefore, the path to developing Intention Goals is performance-based learning objectives, which lead to course content/learning, which leads to Intention Goals. To create Intention Goals, you review the performance-based learning objectives and the course content/learning and answer two questions: