Category: Evaluation of Training
Just had a new article published with one of my best clients: Michael Yarter in Chief Learning Officer titled: How to Set Up an Internal Training Evaluation Department . Michael and I have both set up internal evaluation departments and collaborated on what we call the “imperatives” for doing this properly to write this article.
I hope you find it useful and would enjoy hearing your comments and experiences on how you have done this.
One aspect of training evaluation is to seek small improvements in course design, delivery, and transfer with the objective of increasing impact. Continuous improvement should be one of the tools that underpins the philosophies of training evaluation. Through constant study and revision of evaluation results, better training occurs, resulting in increased impact (value to the business). Continuous Improvement is a set of activities designed to bring gradual but continual improvement to training results through constant review.
The principles that underlie continuous improvement using training evaluation are the following:
- Focus on the stakeholder. Training is designed, developed, and delivered to meet stakeholder requirements and needs. To accomplish this, stakeholders were solicited in the planning phase and their requirements were built into the evaluation. A decision is then made by stakeholders as to proceed with training, based on the attractiveness of benefits (value) to be gained. Training results are then measured from the perspective of the stakeholder.
-
A preventive approach. “Do it right the first time” instead of correcting through evaluation (inspection) if possible. This requires a systematic instructional design process (such as ADDIE [Analysis Design Development Implementation Evaluation]) to ensure that the training will deliver the results that stakeholders require.
- Management by data. Fact rather than supposition drives continuous improvement actions. Evaluation data communicated in the dashboards against respective Success Gates—direct decision in improvements and outcomes.
-
Commit to ongoing improvement. Each time that Success Gates are reached, consider setting new and improved success measures. Training Evaluation is not a project or a task with a definite end point, but an ongoing commitment to seek out opportunities to deliver better results.
- Cross-functional problem solving. Training problems do not usually fall neatly into a single function such as design, development, delivery, management support, etc. They usually call for solutions that cut horizontally across organizational functions. People with different responsibilities and talents need to develop habits of working together to improve training and its results.
-
Constancy of leadership commitment. A commitment to using evaluation as continuous improvement must be a way of life, built into the fabric of the company. In most organizations, this represents a fundamental change in culture and values. Tenacious and visible commitment of top leadership to evaluation is required if employees inside and outside of the training function are to be motivated to make the investment to change.
Previous posts in this series:
Questions Evaluation will Answer
Elements in an Evaluation Plan
Evaluation Deliverables & Schedule
Coming up in this series of posts:
Training evaluation organizational readiness
As always, I would enjoy hearing your thoughts on this topic.
PE Intention Goals are the planned actions that we want participants to author during training as their personal transfer plan. If training is working as designed, we want participants writing actions that drive desired adoptive behaviors back into their everyday work patterns. Developing Intention Goals is relatively easy if the course has a set of well-written, performance-based learning objectives. They refer to training outcomes that specify the information and/or skills to be mastered and what participants do to demonstrate mastery. Performance-based learning objectives manifest themselves in training in the forms of content modules and skill-practice exercises. Therefore, the path to developing Intention Goals is performance-based learning objectives, which lead to course content/learning, which leads to Intention Goals. To create Intention Goals, you review the performance-based learning objectives and the course content/learning and answer two questions:
Predicting via the Impact Matrix: The Impact Matrix is a rectangular array of Intention goals & beliefs, Adoptive behavior examples, business results, and external contribution factors that, when combined using certain rules, predict the Impact from training.
It’s usually created by a Steering Committee of subject matter experts and key decision makers, drawing a direct link from training to business impact.
As soon as you have adequate information to describe the course, establish a Steering Committee of eight to twelve people to develop the predictions. Ideally, this group is made up of individuals from the learning function along with subject matter experts from business units and functional units; it may also include external consultants. Ideally, they are people who represent their function or business unit and have the expertise to provide valuable insight during the prediction process.
One of the key elements in my Predictive Evaluation (PE) model is the activity to Predict Training’s Value. A few weeks ago I had the honor to present PE to the ASTD chapter in New York City and the chapter president, Patrick Dial, suggested that I create a blog series to show how to do this. It is a wonderful idea. So, for the next few months I will post the steps and techniques for Predict Training’s Value – I hope you enjoy them.
The steps to Predict Training’s Value
I am honored that the ASTD New York City Book Club SIG is spending time this month reading and discussing my Predictive Evaluation book . I had a great time last week presenting Predictive Evaluation (PE) at the February chapter meeting – see the picture below. They were a wonderful group full of questions and made me feel right at home.
In Predictive Evaluation (PE), I share my evaluation findings with stakeholders using the dashboard concept. The dashboard is a report to senior management providing an at-a-glance perspective on the current status of a course in the context of predetermined (predicted) metrics for that program (Intention, Adoption, or Impact). PE dashboards are simple, short, and do not require interpretation. Click here for a sample of these types of
Predictive Evalution
dashboards.
Recently, some of my clients have needed year-end reports that share evaluation findings ( Intention , Adoption , and Impact ) in a single dashboard. I thought that sharing this new year-end dashboard with you was an excellent topic for my final blog of 2011. When designing the new dashboard I set the following requirements:
Recently one of my clients took what I consider the next evolution in Action Planning (
Intention Goals
). They have implemented I am calling “Modular Journaling and Action Panning”. At the end of each major course module (approximately 4 hours of learning), the participants take a few moments and answer a series of questions specific to the skills they just learned and draft the associated Intention Goal (Action Plan).
This is a form of journaling to drive the proper Intention Goals – what makes this unique is that the questions they journal about are specific to that section of the course. The question typically include:
Recently one of my largest clients put together criteria for when Predictive Evaluation (PE) could be used with their courses. I thought it was a wonderful approach and have dedicated this blog to it. The questions asked below are operationalized in order with a yes or no response. To be considered a viable candidate for PE, they are looking for yes answers.
In our training system we have created a 3-5 year strategy for our learning activities, next we decided when to use custom courses to implement the strategy, to ensure our courses will deliver promised business impact we predicted the value of training, and finally we built training transfer into everything we do.
Why evaluate your learning strategy and your course deliveries? Peter Drucker said it best: “ You can’t manage what you can’t measure “. There are many reasons companies evaluate/measure their training system – here are two that I feel are important: