Conducting an Effective Impact Evaluation Interview

Here are some tips for conducting an effective Impact Evaluation interview:

• Listen effectively. Be a good listener; know how to redirect a conversation to get the critical information as quickly as possible. Listen without judging.
• Prepare. Review the employee’s Adoption and Impact Survey data prior to the interview.
• Ease on in. Start with easy to respond to questions—just follow the interview protocol.
• Set the tone. Let the interviewee know how the data will be used and that everything said will be held in strictest confidence. A standard phrase is this: “Mary, I am an external consultant who has been hired by the company to evaluate the effectiveness of the XYZ Training Course. You have been selected from the people who completed the course survey last week. I am interested in how you have applied what you learned in class and what impact it has had on you, your team, or the company. What you tell me will be held in strictest confidence. I may, however, include your story in a summary report without your name. Are you okay to proceed?” Also, express your appreciation for the time that participants have taken to talk to you. Explain how the interview will proceed and then try to follow that format as closely as possible.
• Use the script. Follow the interview protocol.
• Listen to your instincts. As you hear things regarding Adoption and Impact, continue to probe until you feel you have uncovered everything.
• Know what you want. If you’ve set aside thirty minutes for the interview, do your best to stick with that schedule. But be prepared to cut the interview short and jump to the concluding questions. Don’t waste your time or the interviewee’s if you are not talking about Impact data. The most common things that interviewees bring up that side-track you are discussing how well they liked the class, complaining about the company in general, etc.
• Write it down. Forget about remembering everything that transpires during an interview. You’ll want to take notes so that you can review at a later time. This is especially important if you’re interviewing many people.
• Probe. As you uncover areas that you feel are promising, investigate further by asking probing questions. Some common probing questions are as follows: Tell me more about that.  When you did that, what was the result?  How did you do that? What did you do specifically?  What was the result?  You mentioned [insert his/her result]; can you place a metric on that result? What is the metric?

Create Impact Profiles

Impact profiles show in a short, salient manner the profile of successfully adoption that has significant impact.  To create these profiles:

Print the interview summaries for your review. Read each one, and sort them into two piles (the ones that are the best examples of Impact as aligned with the Impact Matrix and those that are not as powerful). Use the following items as criteria for placing an interview summary in the Impact Profile stack:

• Direct impact on and value to the business can be traced to training via the participant’s Intention Goals and Adoptive Behavior.
• Adoption of training has made a measurable difference.
• Results validated in the interview support the data from the Impact Survey.Results are in alignment with the predicted Impact identified in the Impact Matrix.

The interviews that pass the preceding test become your Impact Profiles—examples of the best results that the training has produced. These validate the Impact data reported in the Impact Survey and are further proof of Impact created.   The Impact Profile is a one-page synopsis of the participant’s story. Review these profiles, and answer these questions: Do the profiles support the answers to the questions you determined from the Impact Survey data analysis? Those questions are the following:

• What impact has the company received from training?
• How does that impact compare to predicted impact?

Sample Impact Profile

Previous Blogs in the Impact Evaluation Series

The Impact Hunt

Develop Impact Survey

Collect Detailed Impact Data from Successfully Adopted Participants

Analyze Impact Survey Data

Interview Successful and Unsuccessful Participants (part 1 of 2)