Category: Evaluation Survey
Training Evaluation work processes need to be integrated with the following existing training and development processes:
- Instructional Design
- Course Development
- Course Delivery
At times we use emails and surveys to collect evaluation data (especially Adoption and Impact data). We also want a sufficient return rate so that the conclusions we make are reliable. Here are a few tips to help you increase that rate when using email and and on-line survey.
Tips to improve the open rate of your email:
- Make your ‘From’ name easily recognizable by including your company name, or the name of an individual.
- Keep subject lines short – 35 characters or less.
- Avoid large graphics at the top of your email as it wastes valuable space. Include your most important content instead.
Everyone wants to have the best end of course evaluation form for their courses. Too often, people copy other organizations forms because they like the questions without really thinking through the data the instrument needs to capture. Before developing your training / course evaluation form, answer these questions:
- How will the data be used? Understand the purpose of the training / course evaluation form. It must provide data that supports the purpose of the evaluation.
- How will the data be analyzed? Training / course evaluation form data is collected, tabulated, summarized, and reported to others. Design the training / course evaluation form so that data can be easily summarized and analyzed.
- Who will use the information? Identify the audience and how they plan to use the information from the training / course evaluation form.
- What facts are needed? Identify the specific data the training / course evaluation form needs to capture so that provides useful information.
- What is success? This is defining the Quality Gate or Q-Gate of successful results such as a mean value of 4.0 or greater on a 5-point scale.
Recently I commented via LinkedIn to a discussion on how many people you need when conducting a training evaluation, so I thought I would provide an explanation here.
In terms of sample size for evaluation I have used the rule of thumb of a minimum of 30 (the properties of normal distribution) tend to come into play at this minimum amount. The key construct that I follow is that you need sufficient data to draw conclusions so that you can report your findings and defend them. On large scale evaluations, the minimum depends upon the size of the population of participants. You will need to be concerned with population size, sample size, confidence level, and confidence interval. An excellent site (free) to help you determine all of this is: Sample Size Calculator .
So what does all of this mean in say an Adoption Evaluation? An example will demonstrate. A survey is emailed to 1,000 participants and the evaluator wants enough responses to be at the 95% confidence level. What is the number of completed surveys (return rate) to be at the 95% confidence level? Answer: 278. That is a 27.8% response rate. If there are less than 278 responses, the evaluator will not be able to generalize from the data and draw conclusions about the entire population of participants (see Figure 18). This chart shows how the response rate varies based upon the population/sample size and is based upon response rate charts found in many statistics books.
The other day a colleague asked me to review a new end-of-course “Level 1 Reaction” survey he was about to launch for a series of on-line leadership programs. He also wanted to know how to use Predictive Evaluation (PE) techniques with the survey. Cool question – see my answer below.