What makes PE different?
Predictive Evaluation (PE) is the only training evaluation approach that adds the element of forecasting (prediction). Using the PE Model companies can successfully predict training’s results, value, intention, adoption and impact, allowing them to make smarter, more strategic training and evaluation investments.
The Predictive Evaluation Model is a practical step-by-step training evaluation process – it’s unique because it: adds the element of prediction to training evaluations. Requires student participation, making associates feel invested and engaged throughout the process. Provides recommendations for continuous improvement. Predicts the quantifiable impact of training to forecast training investments, results, and ROI. While performing all the traditional training evaluation activities.
How PE Differs from Other Approaches
By doing all the previous actions in creating an evaluation plan (mission, objectives & deliverables, questions the evaluation will answer, assumptions, boundaries, and stakeholders), you can now develop the strategies you will use to perform the evaluation. An evaluation strategy describes how the ends (goals) will be achieved by the means (resources). This details the actions you will perform during the evaluation.
Items I usually include in a training evaluation strategy are the methods to, collect data, analyze it, and report the findings. Here is an example of recent evaluation plan strategy I did for a Level 3 evaluation:
Validate the strategies with your evaluation stakeholders and client, make changes as needed, and move to the next step.
Provide evaluation data that is exactly what the business needs, that is actionable by key decision-makers, and that has been interpreted and analyzed, with solid business recommendations. My latest article was published in Training Magazine on using evaluation to build strong relationships with your business partners.
You can read the article here . I would enjoy hearing your reaction and additions.
Evaluation Stakeholder management is critical to the success of every evaluation. By engaging the right people in the right way in your evaluation during the planning, it can make a big difference to the evaluation success.
A stakeholder is a person or organization that:
- Is actively involved in the evaluation.
- Has interests that may be positively or negatively affected by the performance or completion of the evaluation.
- May exert influence over the evaluation, its deliverables, or its team members.
Evaluation stakeholder management is an important discipline that successful evaluators use to win support from others. It helps ensure that their evaluation succeed where others fail. Stakeholder Analysis is the technique used to identify the key people who have to be won over. You then use Stakeholder Planning to build the support that helps you succeed. The benefits of using a stakeholder-based approach are that:
- You can use the opinions of the most powerful stakeholders to shape your evaluation at this early stage. Not only does this make it more likely that they will support you, their input can also improve the quality of your evaluation.
- Gaining support from powerful stakeholders can help you to win more resources – this makes it more likely that your evaluation will be successful
- By communicating with stakeholders early and frequently, you can ensure that they fully understand what you are doing and understand the benefits of your evaluation– this means they can support you actively when necessary
- You can anticipate what people’s reaction to your evaluation may be, and build into your plan the actions that will win people’s support.
The steps to engage evaluation stakeholders are:
Step 1. Identify your stakeholders.
The first step is to brainstorm who your stakeholders are. As part of this, think of all the people who are affected by your evaluation, who have influence or power over it, or have an interest in its successful or unsuccessful effort.
Step 2. Prioritize Your stakeholders.
Map out your stakeholders and classify them by their power over your work and by their interest in your work. The top people are your key stakeholders.
Step 3. Gather key stakeholder requirements.
You now need to know more about your key stakeholders. You need to know how they are likely to feel about and react to your evaluation. You also need to know how best to engage them and how best to communicate with them. Key questions that can help you understand your stakeholders are:
- What financial or emotional interest do they have in the outcome of the evaluation? Is it positive or negative?
- What information do they want from you? What questions do they want answered from the evaluation?
- How do they want to receive information from you? What is the best way of communicating your message to them?
A very good way of answering these questions is to talk to your stakeholders directly – people are often quite open about their views, and asking people’s opinions is often the first step in building a successful relationship with them.
Once you have completed your stakeholder analysis, you can continue with creating your evaluation plan.
The boundaries of an evaluation generally identify what is included within the evaluation. To identify and state the boundaries for an evaluation, it is required to clearly define items that are inside and outside of the work. All this is about defining the limits and exclusions for the evaluation. Stakeholders should review a drafted boundaries to decide whether a particular aspect should be included in the evaluation or not.
Answer this question: What are the specific items that are not within scope of the evaluation?
Example of evaluation boundaries:
- We will not include non-XYZ course participants.
- We will not include participants who did not complete the course.
- We are not doing a Level 4 Results/Impact evaluation.
- We will not redesign the program or evaluate instructors.
Your Quality Check
- Defines evaluation scope
- Must be aligned with evaluation objectives
- Are they complete?
Few evaluations begin with absolute certainty. If we had to wait for absolute certainty, most evaluations would never get off the ground. As evaluations are planned and executed, some facts and issues are known, others must be estimated. Estimation is an art, with many fine points to finesse between certainty and wishful thinking. You can’t just hope you have the resources you need to do the evaluation, and you can’t wait until every resource is available to begin. You have to manage and mitigate using informed assumptions.
Assumptions fill in the gaps between known proven facts and total guesswork. Each assumption is an “educated guess”, a likely condition, circumstance or event, presumed known and true in the absence of absolute certainty. Once identified, these assumptions and shape our evaluation plan.
Consider this example:
- A defined budget is a fact. i.e. $10,000 has been allocated to complete a given evaluation.
- The belief that the budget is sufficient to complete the project on time and as required is an assumption. This assumption should not be a guess. It should be the result of a planned, verified budget estimate.
From initiation to closure, assumptions set the stage for evaluation planning and execution. As the evaluation is planned, assumptions are used to define and shape tasks, schedules, resource assignments and budget allocations. As such, each is used to manage an otherwise uncertain future, laying out a roadmap for how the evaluation will proceed.
So assume carefully…but document those assumptions as backup.
Answer this question: What assumptions have already been made about the evaluation?
What assumptions have you made regarding a training evaluation? I would enjoy hearing your thoughts.
Now that you understand the course you are evaluating, have an agreed upon mission, objectives, and deliverables you can turn to the questions this evaluation will answer. These are the questions that we will answer after we complete the evaluation.
They drive the evaluation strategy and activity and must be aligned with the mission, objectives, and deliverables.
Some examples for a transfer evaluation:
- What behaviors have been successfully transferred to the job?
- What is the rate of transfer? What should it be?
- What is the profile of a graduate who has successful transfer?
By creating these questions it keeps your evaluation focused on the things that matter and avoids scope creep that may occur.
Once a mission statement (see this blog ) has been developed, you can write your evaluation objectives. Note that objectives are much more specific than the mission statement itself and defines results that must be achieved in order for the overall mission to be accomplished. Also, an objective defines the desired end result.
Deliverables are the items that will be produced by the training evaluation. Items such as reports, dashboards, executive presentations, etc. are products typically named here.
As example of objectives and deliverables in an evaluation plan:
The following two questions are useful both in setting objectives and monitoring progress:
- What is the desired outcome? This is called the outcome frame. It helps you keep focused on the result we are trying to achieve, rather than on the effort expended to get there.
- How will we know when we achieve it? This is the evidence question. This question is very useful for establishing exit criteria for objectives that cannot be quantified.
What are your thoughts regarding objectives and deliverables with respect to training evaluation efforts?
The mission of an evaluation documents
- What we going to do?
- For whom are we going to do it?
The mission itself defines what you are doing – how you’re going to do it is Evaluation Plan. An example of an evaluation mission is…
As the word “mission” implies, the mission statement clearly states the evaluation’s goal. With this goal in mind, all stakeholders have an understanding of the evaluation direction and purpose. A mission statement is one of the simplest and most basic necessities of a training evaluation, yet it is frequently overlooked by evaluators.
Last night I had the honor of speaking at the ISPI Atlanta chapter with Denise Traicoff from the Centers for Disease Control . I did my usual speech on my Predictive Evaluation Model (PE) and Denise followed on how she and her team had implemented PE on one of their programs. She told the story of how her team had been given the assignment to teach leadership & management skills to groups of volunteers from around the world to go in a eradicate polio in countries. It was a compelling story that kept all in attendance transfixed on her talk.
Sandi Brown, Dave Basarab, and Denise Traicoff
She shared how on her flight to the World Health Organization to begin the needs analysis she was reading my book . She thought that due to nature of her course “the world deserved more than just happy sheets”. As she read, she got more and more excited about PE and shared that the could provide more rigor to her course measurement. What made me proud was that she implemented Intention and Adoption evaluation from just reading the book. What ever author wants.
Denise and her team, including Sandi Brown (pictured), have now successfully implemented PE on three sessions and will continue to use it. But PE is not the story here. It is Denise and her team – what they have done. Even more fun, was that last night was the first time we had met face-to-face even though we both live in Atlanta. I am honored to be associated with these talented people. So in a small way sometime we do make a difference. I had my moment last night. Thank you so much Denise.