Category: Training Evaluation
Provide evaluation data that is exactly what the business needs, that is actionable by key decision-makers, and that has been interpreted and analyzed, with solid business recommendations. My latest article was published in Training Magazine on using evaluation to build strong relationships with your business partners.
You can read the article here . I would enjoy hearing your reaction and additions.
Once a mission statement (see this blog ) has been developed, you can write your evaluation objectives. Note that objectives are much more specific than the mission statement itself and defines results that must be achieved in order for the overall mission to be accomplished. Also, an objective defines the desired end result.
Deliverables are the items that will be produced by the training evaluation. Items such as reports, dashboards, executive presentations, etc. are products typically named here.
As example of objectives and deliverables in an evaluation plan:
The following two questions are useful both in setting objectives and monitoring progress:
- What is the desired outcome? This is called the outcome frame. It helps you keep focused on the result we are trying to achieve, rather than on the effort expended to get there.
- How will we know when we achieve it? This is the evidence question. This question is very useful for establishing exit criteria for objectives that cannot be quantified.
What are your thoughts regarding objectives and deliverables with respect to training evaluation efforts?
The mission of an evaluation documents
- What we going to do?
- For whom are we going to do it?
The mission itself defines what you are doing – how you’re going to do it is Evaluation Plan. An example of an evaluation mission is…
As the word “mission” implies, the mission statement clearly states the evaluation’s goal. With this goal in mind, all stakeholders have an understanding of the evaluation direction and purpose. A mission statement is one of the simplest and most basic necessities of a training evaluation, yet it is frequently overlooked by evaluators.
Failure to do proper training evaluation planning causes poor efforts that usually include rework, the project being over budget, and delays. The Plan phase is the starting point of the training evaluation. Planning an evaluation is specifying what and how you’re going to evaluate by reviewing the BOC-1 program, developing the project’s mission, creating the evaluation objectives & deliverables, developing the questions the evaluation will answer, recording assumptions, documenting boundaries, identifying stakeholders, developing the strategy, estimated costs & timelines, and performing risk assessment. From this a detailed evaluation project plan with milestones is created.
I begin all my evaluations with with a detailed plan in which I include a steering committee to build it. The steps I use with the steering committee are:
- Determine the evaluation methodology
- Understand the course being evaluated
- Mission of the evaluation
- Objectives & deliverables
- Questions the evaluation will answer
- Assumptions
- Boundaries
- Stakeholders
- On-the-job behaviors
- Beliefs
- Strategies (data collection, data analysis, reporting)
- Timeline (project plan & key milestones)
Over the next few weeks I will post how to do each step. I hope you find them useful and encourage you to give me your thoughts.
Many times I have T&D professionals ask me the steps I take when evaluating programs. Here are the steps I follow:
But this is only the framework for conducting an evaluation, you should create an Evaluation Project Plan . This is a formal, approved document used to guide both project execution and project control. The primary uses of the Evaluation Project Plan are to document planning assumptions and decisions, facilitate communication among stakeholders, and document approved scope, cost, and schedule baselines.
Currently I am about to begin Level 3 Evaluation project for a client. Here is the project plan we just agreed to:
This gives you an idea of the time per task/milestone and the activities (tasks) for each step in the framework.
With the evaluation plan in place and approval to proceed, implement the evaluation by using the following sequence.
- Choose the course you wish to evaluate ; it can be a new course design or a course that already exists. Note: this should have been done when you developed the evaluation plan.
-
Review the course to understand fully what it is, what it is supposed to teach, how participants learn the material, what business issues it addresses, who is to attend, how they attend, what pre-course preparations are in place, what post-course support mechanisms exist, who are the sponsors, what sponsors see as the purpose for the course.
- Form a committee to sponsor the evaluation and provide oversight while the evaluation is under way.
- Create evaluation protocols, develop the data collection instruments, and create models to analyze collected data for the evaluation. Developing evaluation procedures involves taking the approved evaluation plan and drafting the techniques, procedures, and instruments necessary to implement the plan.
- Collect quantitative & qualitative evaluation data as prescribed in the plan and using the data collection instruments. This phase of the training evaluation of entails gathering data to address the evaluation questions created in the evaluation plan. The goal is to obtain the right kind at the lowest-cost to answer the evaluation questions.
- Analyze the evaluation data and draw truthful conclusions and recommendations . In the analyze phase the meaning of collected data are interpreted. This phase uses the collected data to determine the conclusions for which the data supports and the amount of support the data supplies for, or against, those conclusions. Quantitative & qualitative data analysis procedures are employed.
- Create an Evaluation Report and Executive Dashboard. The purpose of this phase is to create the vehicles to communicate training evaluation information to stakeholders so that they can make use of the evaluation information for decision-making or purposes of accountability. This phase will generate an evaluation report and an executive dashboard detailing the results of the evaluation and the recommendations for to maximize the program’s impact.
Previous posts in this series:
Questions Evaluation will Answer
Elements in an Evaluation Plan
Evaluation Deliverables & Schedule
Coming up in this series of posts:
- Training evaluation and continuous improvement
- Training evaluation organizational readiness
As always, I would enjoy hearing your thoughts on this topic.
Evaluation Deliverables
Using the evaluation questions and other planning elements you have defined earlier, create a list of information that the evaluation needs to deliver in order to answer the evaluation questions. Specify when and how each item must be delivered. These are the elements of the evaluation. Add the deliverables to the evaluation plan, with an estimated delivery date. More accurate delivery dates can be established during the scheduling phase, which is next.
The Evaluation Schedule
Create a list of tasks that need to be carried out for each deliverable. For each task, identify the following:
- The amount of effort (hours or days) required to complete the task
- The resource that will carry out the task
Once you have established the amount of effort for each task, you can work out the effort required for each deliverable and determine an accurate delivery date. Update your deliverables section with the more accurate delivery dates. A common problem discovered at this point occurs when an evaluation has an imposed delivery deadline from the sponsor or steering committee that is not realistic based on your estimates. If you discover that this is the case, you must contact the sponsor/committee members immediately. The options you have in this situation are the following:
- Renegotiate the due dates
- Employ additional resources
- Reduce the scope of the evaluation
Congratulations! Having followed all the preceding steps, you should have a good evaluation plan. Remember to update your plan as the evaluation progresses and measure progress against the plan.
Previous posts in this series
Questions Evaluation will Answer
Elements in an Evaluation Plan
Coming up in this series of posts:
- Performing the evaluation
- Training evaluation and continuous improvement
- Training evaluation organizational readiness
As always, I would enjoy hearing your thoughts on this topic.
You should now have a clear picture as to the impact that the training is providing and whether or not it is as predicted. An issue that exists with the current impact data is that it is self-reported via the Impact Survey and somewhat validated with the Impact Interviews. This is good and, many times, sufficient for you to produce the Impact Dashboard and draw your conclusions on the value of training. If you choose, you can further build your case by collecting data from existing company records. This type of data is known as extant data.
Extant data are existing data, information, and observations that have been collected or that are available for use in the assessment and evaluation processes. Basically, you analyze records and files that the company already has, rather than set out to gather additional information. Where and what files you look at depend on the impact being claimed by participants (e.g., sales figures, attendance figures, call-backs for repair, etc.). Customer e-mails, tech support log files, marketing and business plans, business requirements, documents, and vision statements are just a few of the types of information that help you validate the impact.
The quickest way to gather extant data is to ask High-Impact interview participants to provide the documentation that proves their case (assuming they have permission to share it with you). This typically comes in the form of presentations, spreadsheets, sales records, financial statements, manufacturing reports, time and motion studies, etc. When you receive the documentation, review it and ensure that you have sufficient information to validate claimed impact.
At times, the participant no longer has the backup documentation or it simply does not exist. This does not mean that you do not include the impact in your Impact Dashboard, but you may need to add a disclaimer to the report stating so.
Bringing Meaning to Impact Results
It is now time to bring meaning to the Impact results that you have collected. Do this by:
- Analyzing results and draw conclusions
- Is the training program contributing to achievement of the business outcomes?
- Collect training expense data
- Compare results & ROI to predictions in the Impact Matrix
Previous Blogs in the Impact Evaluation Series
Collect Detailed Impact Data from Successfully Adopted Participants
Interview Successful and Unsuccessful Participants (part 1 of 2)
Interview Successful and Unsuccessful Participants (part 2 of 2)
One element of data collection for an Impact Evaluation involves interviews with participants in order to find examples of the best results and to identify those employees who have no or very little impact. You need only interview a subset of the graduates in order to generalize conclusions to the entire population. From Adoption Evaluation, review the list of participants that you classified as Unsuccessful Adoption: this is the list of potential individuals to interview.
Randomly select a sample from both groups. You interview enough people in the High-Impact category to find approximately ten best examples of impact. This may involve interviewing fifteen to twenty graduates or more. Note: the more examples found, the stronger is the correlation from this sample to the entire population. However, these numbers are based on the sample size (the number of graduates). With fifty graduates, it is likely that ten to fifteen are in the High-Impact category, which requires a lower number of interviews.
The critical factors affecting interviewing are the time and the budget to conduct the interviews. The more interviews that you can conduct the better, but it needs to be within time and budget constraints. For estimation purposes, a single High-Impact interview (conducting the interview and summarizing the result) takes approximately one hour.
Impact Interview Protocol
Follow the protocol below when conducting Impact interviews:
Inviting High Impact Participants to the Interview
Invite participants to the Impact interview via email. A sample email is:
Inviting Low/No Impact Participants to the Interview
High Impact Interview Template
Low/No Impact Interview Template
Previous Blogs in the Impact Evaluation Series
Collect Detailed Impact Data from Successfully Adopted Participants
In order to predict training value, you need to forecast how the Intention Goals will be distributed among participants. In other words, if twenty participants attend training, what percentage will author goal 1, goal 2, goal 3, etc.?