Category: Training Value
Conducting an Effective Impact Evaluation Interview
Here are some tips for conducting an effective Impact Evaluation interview:
- Listen effectively. Be a good listener; know how to redirect a conversation to get the critical information as quickly as possible. Listen without judging.
- Prepare. Review the employee’s Adoption and Impact Survey data prior to the interview.
- Ease on in. Start with easy to respond to questions—just follow the interview protocol.
- Set the tone. Let the interviewee know how the data will be used and that everything said will be held in strictest confidence. A standard phrase is this: “Mary, I am an external consultant who has been hired by the company to evaluate the effectiveness of the XYZ Training Course. You have been selected from the people who completed the course survey last week. I am interested in how you have applied what you learned in class and what impact it has had on you, your team, or the company. What you tell me will be held in strictest confidence. I may, however, include your story in a summary report without your name. Are you okay to proceed?” Also, express your appreciation for the time that participants have taken to talk to you. Explain how the interview will proceed and then try to follow that format as closely as possible.
- Use the script. Follow the interview protocol.
- Listen to your instincts. As you hear things regarding Adoption and Impact, continue to probe until you feel you have uncovered everything.
- Know what you want. If you’ve set aside thirty minutes for the interview, do your best to stick with that schedule. But be prepared to cut the interview short and jump to the concluding questions. Don’t waste your time or the interviewee’s if you are not talking about Impact data. The most common things that interviewees bring up that side-track you are discussing how well they liked the class, complaining about the company in general, etc.
- Write it down. Forget about remembering everything that transpires during an interview. You’ll want to take notes so that you can review at a later time. This is especially important if you’re interviewing many people.
- Probe. As you uncover areas that you feel are promising, investigate further by asking probing questions. Some common probing questions are as follows: Tell me more about that. When you did that, what was the result? How did you do that? What did you do specifically? What was the result? You mentioned [insert his/her result]; can you place a metric on that result? What is the metric?
Create Impact Profiles
Impact profiles show in a short, salient manner the profile of successfully adoption that has significant impact. To create these profiles:
Print the interview summaries for your review. Read each one, and sort them into two piles (the ones that are the best examples of Impact as aligned with the Impact Matrix and those that are not as powerful). Use the following items as criteria for placing an interview summary in the Impact Profile stack:
- Direct impact on and value to the business can be traced to training via the participant’s Intention Goals and Adoptive Behavior.
- Adoption of training has made a measurable difference.
- Results validated in the interview support the data from the Impact Survey.Results are in alignment with the predicted Impact identified in the Impact Matrix.
The interviews that pass the preceding test become your Impact Profiles—examples of the best results that the training has produced. These validate the Impact data reported in the Impact Survey and are further proof of Impact created. The Impact Profile is a one-page synopsis of the participant’s story. Review these profiles, and answer these questions: Do the profiles support the answers to the questions you determined from the Impact Survey data analysis? Those questions are the following:
- What impact has the company received from training?
- How does that impact compare to predicted impact?
Sample Impact Profile
Previous Blogs in the Impact Evaluation Series
Collect Detailed Impact Data from Successfully Adopted Participants
Interview Successful and Unsuccessful Participants (part 1 of 2)
Many times evaluators struggle with the balance between using quantitative and qualitative data in training evaluations. I like to use both. However, the element that determines what type of data I use always comes back to the purpose of the evaluation and the questions that evaluation will answer. These two things determine what types of data are needed and how to collect it. It is rare that I do not use both data types.
I view quantitative data as the “business end” of the evaluation. Senior executives are used to see data in charts and graphs and like Impact Data in those forms. I use an Impact Dashboard exclusively for these types of data. I always use qualitative data with Impact Evaluation – I view these as the “heart and soul” of the Impact and have found Seniors Executives gravitate more to these than the charts and graphs. What criteria do you use to determine what type of data are used?
Quantitative Data Analysis – Relative Comparison
Qualitative Data Analysis
- Read through the data Raw data are ordered and organized so that useful information can be extracted from them.
- What data do and do not contain. Make notes of the patterns and themes that emerge from the data.
- Answer these questions: What are the data telling you? What is missing? What are the central themes contained in the data? Are you uncovering something unexpected? What is it? Is it pertinent to the evaluation?
Quantitative Data Analysis – Measures of Central Tendency and Variability
Quantitative Data Analysis – Frequency Distribution
Analyzing Qualitative Data
Qualitative data analysis involves the identification, examination, and interpretation of patterns and themes in Impact data and determines how these patterns and themes help answer the question of how much impact has been realized. It is important to note that qualitative data analysis is an ongoing, fluid, and cyclical process that happens throughout the data collection stage of your evaluation project and carries over to the data entry and analysis stages. Although the steps listed below are somewhat sequential they do not always (and sometimes should not) happen in isolation of each other.
Drawing Conclusions to the Entire Population
For each Adoptive Behavior Look at your data and answer this question: What percentage of the sample has successfully adopted? For example, if 50 percent of the sample has been coded as Successful Adoption, you can draw the conclusion that 50 percent of all participants (the population) have successfully adopted. So if 600 participants is the population, 300 would be assumed as Successful Adoption.
Next answer: What statistic best describes the results of the sample? Mean, median, or mode? For example, if you judge that the $5,000 for an Adoptive Behavior is representative, choose that value. You then can conclude that the mean value applies to the population (all participants).
Calculate the total impact for the Adoptive Behavior by multiplying the statistic chosen (in this example, the $5,000 mean) times the Successful Adoption participants from the population (in this example, 300). Therefore, the total impact for this Adoptive Behavior is $5,000 × 300 = $1,500,000. Note: For a sample to be used as a guide to an entire population, it is important that it be truly a representative of that overall population. Representative sampling assured, inferences and conclusions can be safely extended from the sample to the population as a whole. A major problem lies in determining the extent to which the sample chosen is actually representative. Statistics offers methods to estimate and correct for any random trending within the sample and data collection procedures. There are also methods for designing experiments that can lessen these issues at the outset of a study, strengthening its capability to discern truths about the population. Statisticians describe stronger methods as more “robust.” If you have concerns about this, seek assistance from a statistician.
Answering the Impact Questions
Answer two questions and then (validate your initial findings later, with participant interviews and investigating company records).
What impact has the company received from training? Sum all the value being reported and discount by the external contribution factor. You may also view it as an average per participant if that is warranted.
How does that impact compare to predicted impact? Look at what you have found and compare it to the course’s Impact Matrix . Are the results ahead, behind, or on track with the predicted impact? Are you seeing unforeseen impact? Are participants doing things related to training and producing impact that was not predicted?
Summary
Interpretation is the process of giving meaning to the result of the analysis and deciding the significance and implications of what the data show. Answering the evaluation question means interpreting the analyzed data (quantitative and qualitative) to see if results support or do not support the answers to the evaluation questions.
Previous Blogs in the Impact Evaluation Series
Collect Detailed Impact Data from Successfully Adopted Participants
Creating value requires an investment in future returns. In our case the cost to design, develop, deliver, maintain, and evaluate the course. Since value is usually generated over time you need to calculate the lifetime cost of the training.
Training has a life cycle similar to a product life cycle used in marketing. It describes the stages a course goes through from when it was first thought of until it finally is removed from the being offered. In these stages, costs are incurred – I call these the Lifetime Training Costs.
The Steering Committee has accomplished a good deal to get to this point. To recap, you have the following predictive elements in place: Beliefs, Intention Goals, Adoptive Behaviors, Distribution of Goals, Adoption Rate, Adoption Value, and External Contribution Factors.
These are all the elements needed to calculate annual impact per participant using the Impact Matrix. The Matrix uses simple formulas to calculate training’s predicted impact.
Results that employees realize may be attributed to training, but other organizational forces could affect them also. These forces may include elements directly controlled by the company (internal forces) and those external to the company. Examples of internal forces are new product introduction, price changes, new Human Resources processes, change in strategy/annual operating plans, new leadership, mergers and acquisitions, changes in compensation plans, etc.
Examples of external forces are new competition, government regulations, local/global economic conditions, etc. For example, if the Steering Committee predicts a $10,000 value from an adopted behavior, internal and external forces have contributed to that amount to some degree. This needs to be recognized and accounted for so that a more accurate prediction of training value is created.
Adoptive the value is the annual result of an adopted behavior being performed by one participant in monetary terms. Each result was worth something to the company. If it isn’t, why include in the course? This is where expert performers, subject matter expert, human resource professionals, and individuals from the finance organization can provide input into the value. The committee answers this question: What results are realized when a participant successfully implements the adopted behavior.
When capturing the adopted values I used the following worksheet:
Now that we have predicted the Adoption rate we need to begin forecasting the return that this course can provide. To do this you estimate the number students to be trained on an annual basis. Then estimate the course life in years. These form the basis for the value and costing elements in the model. See the chart below for the Business Acumen Training course.
When organizations implement training, it’s usually driven by a bigger-picture issue, like launching new business processes. However, companies often struggle to justify training’s effort and expense because they can’t predict definitive business outcomes.
In lieu of a crystal ball, trainers need some way to prove that training’s worth the investment. By adding the element of prediction, they can demonstrate – with high confidence – the additional value that training will deliver, including increased revenue, sales, savings, and/or profit. Essentially, it creates a business forecast for training.
Not every participant will adopt his or her goals. The question is what is an acceptable percentage of those who adopt? That is what the adoption rate attempts to do. It is the percentage of participants that we want to perform the work (Adoptive Behaviors) and is the prediction of successful transfer from training to the workplace. The Steering Committee needs to answer this question for each Intention Goal: What percentage of participants with this Intention Goal will successfully perform the Adoptive Behavior?
So far we have predicted training’s value by predicting the intention goals and each goal’s associated distribution rate. We also have predicted the beliefs the course should create. The next step is to predict the adoptive behaviors . Here we ask our steering committee to document, by intention goal, of the expect the goal to be performed on the job.
The Committee members must tackle the challenge of predicting the Adoptive Behaviors that they want participants to perform on-the-job. An Adoptive Behavior is the unit of work (what the participant does on-the-job, based on the intended goals, beliefs, and course design).
The Adoptive Behaviors flow from the course design and are the manifestation of Intention Goals and Beliefs.