Measuring the impact of learning and development on business results doesn't need to be a daunting task. By asking a few key questions and tapping into existing metrics, we can demonstrate credible return on investment in learning initiatives.
by Cushing Anderson
October 25, 2009
Few learning and development professionals would argue that measuring the business results of training just makes good sense. And yet, most keep putting off the pursuit of a return-on-investment (ROI) study.
Why? Typically, the hurdle is a fear of the unknown — a fear that makes organizations shy away from doing an ROI study based entirely on the assumption that the process will be too time consuming and cumbersome.
But it doesn’t require a Herculean effort to come up with a good, accurate ROI, nor does it need to be a major initiative. An ROI study can be conducted without jeopardizing validity and still minimize the time commitment of the training department. Well-planned and well-designed training interventions can create significant gains, which is where it always starts. Or does it?
Identifying the Need
The start of a good ROI, as well as a good intervention, is identifying the real business need. Why, for example, is this unit’s or department’s work important and how do its actions and work habits impact the business? How could a change in those actions and habits result in a gain for the organization? Knowing the business reason, both before undertaking any training intervention as well as during an ROI study, is crucial to ensuring that your organization will achieve the change in work habits desired and will address the business need you have identified.
The time to do this and to think about measuring ROI is at the beginning of the project. In doing an ROI study, you also need to measure Levels 1 to 3 first. Planning upfront actions, like pre-tests, post-tests and how to get behavioral examples, will make them easier to gather when the time comes. The first time we measured the ROI of a leadership program, we did pre-tests before each of three modules or classes. The second time, we surveyed all the pre-class measures in one document instead of three. That was far less intrusive on both participant and manager time.
Another rich source of data for determining the underlying business need is data that the organization has gathered in employee engagement surveys or client surveys. One of our clients, a financial services company, surveyed its internal clients for input about the role they would like their internal business partners to play. The data revealed an overwhelming need for the internal business partners to improve their consulting skills. A follow-up focus group with the internal business partners uncovered their desire to learn the tools that external consulting firms, such as McKinsey, use in their client engagements. Their desire was not to replace the need for external consultants — external consultants’ expertise is well-regarded in the organization — but to match the rigor of the data and problem analyses that external consultants bring to the table. Armed with this information, we built root cause analysis and data analysis tools into the curriculum.
When we conducted the ROI study, we included the analyses tools and skills in the survey and were able to get an accurate read of how effective the internal business partners were in using the skills. The ROI measured the change in behavior of the internal business partners at six weeks following the training and again at nine months post-training. We chose to only survey the internal business clients and the managers of the internal business partners to safeguard the ROI from the perception that the results were skewed by self-reporting — a knock on many ROIs when participants are included in the study.
The results confirmed what trainers have always suspected: Given the opportunity to use skills learned in the classroom and reinforced over time, participants’ effectiveness in using the skills improved considerably. This reversed the downward trend on the learning curve that trainers have come to expect. We were able to demonstrate that the internal business partners improved their effectiveness in using the skills in the nine months following the program, in some cases from 29 to 55 percent. Note the learning curve trended sharply upward, an achievement and byproduct discovered only through the measurement process.
Slimmed-Down ROI: Step by Step
There are many ways to conduct a study of your return on investment. But bigger, more time consuming, painful or expensive doesn’t always mean better.
Try slimmed-down ROI. This approach incorporates seven critical steps, all of which focus on exactly what you need to know and do — no more and no less.
Step 1: Identify and isolate the key skill or competency that most directly relates to the key business driver. With so many variables, it’s important to identify the one or two that are truly key to improvement. By isolating the key skill or competency, you are narrowing both your focus and your ROI study, and as important, narrowing them to something meaningful to the organization. As with all the steps outlined here, it is the organization that must articulate the key skill. You don’t have to measure, or try to measure, every skill or competency affected by your training program. Concentrate on what matters most.
Step 2: Tie the key skill or competency to one or two key metrics. What is a true measure of success? There are so many metrics, and not all metrics are created equal. By narrowing your focus, you can expend more effort on ways to determine the change, rather than chasing after spreadsheets full of meaningless numbers. Seek out a formula that appears to be most realistic. You can find suggestions through various online resources such as ASTD, the U.S. Bureau of Labor Statistics and the Corporate Leadership Council, among others.
What metrics does your company analyze, and what do senior leaders truly value or look for? If a number or formula exists, it can simplify your work greatly. If not, create a formula that isolates these skills as much as possible, where data is readily available to measure and benchmark with other organizations. Some of these metrics include turnover, customer or employee retention, increase in sales or lower costs. The real benefit of working with either an existing number or a simple calculation for the results you are seeking is that it keeps you from measuring less valuable or altogether irrelevant factors. Focus on what’s most important and actionable to your senior management.
Step 3: Create an ROI survey to get the data from participants, their managers and clients quickly and simply. It’s important to ensure that you’re getting the results from the end customer. Keep the survey short, focused and to the point. Isolating the key skill and key metrics is crucial, so don’t blow it by including a whole lot of other data on the survey. By keeping it short and focused, you’ll have greater participation and increased completion rates. Be sure to use a data analysis tool to distribute and calculate results. This could be your learning management system (LMS) or another tool, such as Survey Monkey or Zoomerang. We’ve seen survey participation rates from 58 to 90 percent, well above typical survey participation rates. Shorter, more focused surveys take less time to complete and to analyze.
Step 4: Implement the survey. Communicate ahead of time about the reasons for the survey, what the organization hopes to learn from it and why participation is important to success. Use a recognizable communication tool so your e-mail is less likely to be seen as spam. Create a reasonable time frame for the survey — one to two weeks is typically good — with a last-chance reminder for those who have not completed it a couple of days before closing the survey. The ideal time frame depends upon your project and your employee culture; keep in mind that there are situations in which extending the length of time that the survey is open can result in diminishing returns beyond the original end date.
Step 5: Calculate the costs of the program, as fully loaded as possible to preserve validity of the research. These costs include the design and development time, training delivery costs, travel and participant salaries. Use averages wherever you can, such as average participant salary based on salary grades. This is another way to reduce the amount of time spent gathering and analyzing individual salaries. Financial analysts often work with averages or means in determining costs when predicting payback periods for investments, so you need not fear that average costs will weaken the validity of your ROI. The savings? By using averages where appropriate, you can save a lot of effort and hard work while still gaining a good approximation of the numbers you are seeking.
Step 6: Calculate the results from the survey. When asking survey participants for specific costs, always ask for their degree of confidence in the numbers. Then multiply their number by the degree of confidence. Always use the most conservative figure to gain credibility with senior management. In the first ROI study we conducted, we asked salespeople to isolate a specific negotiation skill they learned in class and estimate the revenue they generated from using the skill. One participant felt he could attribute $300,000 in revenue to his use of the negotiating skill and gave it a 100 percent degree of confidence. We didn’t believe the number could be correct. We checked what revenue his manager had entered on his survey, and found that $200,000 was more likely. The manager gave it a 50 percent degree of confidence. In the end, we reported $100,000, which was still a remarkable achievement.
What’s the slimming aspect of Step 6? Greater accuracy and agreement and being conservative in your reporting increase your credibility. Credibility buys you a great deal of time saved in trying to explain extraordinary figures.
Step 7: Calculate ROI by dividing program costs by the key metric measured. In virtually every case, efforts will pay off with a positive ROI, and you can create an effective report and summary for key executives with numbers that aren’t contrived and are valued by your organization. In the end, it’s perhaps most important that you have a defensible ROI that focuses on how learning contributed to the bottom line.
By using the seven steps for a slimmed-down ROI, you can provide data that is compelling, clear, accurate and based on key metrics that the organization has determined will contribute to the bottom line. And perhaps just as important in these busy times, this ROI methodology will require less input from the organization than ever before, while providing greater credibility than the work that you have done previously. Most learning and development professionals know they make a real difference for employees. Here’s to making that difference measurable.