Category: Predictive Evaluation
I am honored to have an new article just published in
In the article I discuss the elements of my Predictive Evaluation model . Watch this short video about the PE model.
To learn more about the model, take a look at my latest book:
Prior to sending the survey, it is a good idea to build the data analysis model that used to summarize the data. It usually takes the form of a spreadsheet or database. In a spreadsheet, the data structure in each row is one participant response with the columns being the data.
When to collect data is a factor of the number of course deliveries and volume of participants trained. Other guidelines on the timing or frequency of data collection include the following:
- Intention Evaluation : collect at the conclusion of each course. For classroom courses, collect a goal sheet from each participant. For e-learning courses, embed the goal sheet into the end-of-course programming and submit it to the data analysis model.
- Adoption Evaluation : collect data when a sufficient number of participants have graduated and have had the opportunity to transfer their skills to the workplace. If you have a sufficient population size, you do not have to collect adoption data from all participants. Collect enough data to ensure a reasonable return rate and sample to obtain a high degree of confidence.
- Impact Evaluation : collect data immediately after the Adoption data have been analyzed and reported.
Survey Administration
Draft an email from a company executive soliciting survey completion and watch the response rate (the ratio of number of people who answered the survey divided by the number of people in the sample, usually expressed in the form of a percentage).
A reminder email is sent requesting completion of the survey. You want to get the highest return rate as possible, because obtaining a high response rate can bolster statistical power, reduce sampling error, and enhance the generalizability of the results to the population (participants).
Data Scrubbing
When collection has ceased, you need to scrub the Impact data. Data scrubbing, sometimes, called data cleansing, is the process of detecting and removing or correcting any information in a database (or spreadsheet) that has some sort of error. This error can be because the data are wrong, incomplete, formatted incorrectly, or are a duplicate copy of another entry (the participant responded multiple times). Simply review the data and make the necessary corrections.
As always, please send me your thoughts on this method. Next blog: Analyze Impact Survey Data.
Previous Blogs in the Impact Evaluation Series
Impact Evaluation
Evaluation is identifying the direct impact on – and value to – the business that can be traced to training and is part of my Predictive Evaluation model . It assesses in quantifiable terms the value of the training by assessing which Adoptive Behaviors have made a measurable difference. Impact evaluation goes beyond assessing the degree to which participants are using what was learned; it provides a reliable and valid measure of the results of the training to the organization.
When participants report a positive impact from training, this approach allows you to articulate how and why training is impacting the business, leveraging this information to enhance the organizational impact of future deliveries. Conversely, when little or no impact is found, this evaluation method uncovers why and determines what can be done to remedy that undesirable outcome. Impact evaluation seeks value to the business. This is done by collecting data on actual business results that participants attribute to successful adoption of their Intention Goal(s). In their Adoption Evaluation survey, participants told us what they did, and we classified them as Successful Adoption or Unsuccessful Adoption. From the successful participants, collect additional impact data via three methods:
- Completion of an Impact Survey,
- Interviewing participants, and
- Examination of company records to confirm findings.
The Impact Hunt
It is impractical to follow every participant and determine impact. So I use sampling with a subset of participants to estimate the impact of the whole population (all participants.) I refer to this as the “Impact Hunt.”
The method is:
- Start with all participants and survey them via the Adoption evaluation survey technique.
- Narrow the potential pool of impact analysis participants to only those who were judged as successfully adopted.
- Send everyone who is successfully adopted a detailed impact you valuation survey.
- Using the Impact survey data, identify people who have the highest likelihood or self-reporting impact. This becomes your Impact pool.
- From this pool randomly sample participants to conducting in-depth impact interview with and evaluate their results.
Graphically the Impact Evaluation Process is:
As always, please send me your thoughts on this method. Next blog: Develop the Impact Survey.
Creating value requires an investment in future returns. In our case the cost to design, develop, deliver, maintain, and evaluate the course. Since value is usually generated over time you need to calculate the lifetime cost of the training.
Training has a life cycle similar to a product life cycle used in marketing. It describes the stages a course goes through from when it was first thought of until it finally is removed from the being offered. In these stages, costs are incurred – I call these the Lifetime Training Costs.
The Steering Committee has accomplished a good deal to get to this point. To recap, you have the following predictive elements in place: Beliefs, Intention Goals, Adoptive Behaviors, Distribution of Goals, Adoption Rate, Adoption Value, and External Contribution Factors.
These are all the elements needed to calculate annual impact per participant using the Impact Matrix. The Matrix uses simple formulas to calculate training’s predicted impact.
Results that employees realize may be attributed to training, but other organizational forces could affect them also. These forces may include elements directly controlled by the company (internal forces) and those external to the company. Examples of internal forces are new product introduction, price changes, new Human Resources processes, change in strategy/annual operating plans, new leadership, mergers and acquisitions, changes in compensation plans, etc.
Examples of external forces are new competition, government regulations, local/global economic conditions, etc. For example, if the Steering Committee predicts a $10,000 value from an adopted behavior, internal and external forces have contributed to that amount to some degree. This needs to be recognized and accounted for so that a more accurate prediction of training value is created.
Adoptive the value is the annual result of an adopted behavior being performed by one participant in monetary terms. Each result was worth something to the company. If it isn’t, why include in the course? This is where expert performers, subject matter expert, human resource professionals, and individuals from the finance organization can provide input into the value. The committee answers this question: What results are realized when a participant successfully implements the adopted behavior.
When capturing the adopted values I used the following worksheet:
When organizations implement training, it’s usually driven by a bigger-picture issue, like launching new business processes. However, companies often struggle to justify training’s effort and expense because they can’t predict definitive business outcomes.
In lieu of a crystal ball, trainers need some way to prove that training’s worth the investment. By adding the element of prediction, they can demonstrate – with high confidence – the additional value that training will deliver, including increased revenue, sales, savings, and/or profit. Essentially, it creates a business forecast for training.
Beliefs —what do you want the participants to believe in when they complete the course?
When participants believe that new skills and knowledge mastered in training will help them and/or their organization, the likelihood of adoption is increased. When participants change their beliefs about themselves and their personal performance, they apply the new skills and even take on new challenges. Beliefs manifest themselves in words and actions. Therefore training should instill the set of beliefs that support participants in trying, practicing, and finally transferring these skills.
You do not teach beliefs in training, but the design should develop the belief structure during the learning experience. The instructional design needs to make sure that everything that is said and done in the course is consistent with the training’s belief statements. A few techniques include guest speakers sharing their success stories using the new skills, the language that instructors use when teaching, posters (signage) in the classroom, etc.
Predictive Evaluation: Ensuring Training Delivers Business and Organizational Results
Thursday, April 19th, 2012 2:00-3:00pm EST
Please join me for the second of my webinar series Making the Most of Corporate Training Dollars
- Do you struggle to define training’s success?
- Are you fighting to justify the training’s value within your organization?
- Does your organization view training as an expense versus an investment with predicted return?
- Do you need a method of predicting (forecasting) the training’s value to help decide whether to train?
- Are your current evaluation efforts always “after the fact”?
- Do you want to measure success using leading indicators that drive continuous improvement?
Then consider Predictive Evaluation