In some instances, success is the achievement of something desired, planned or attempted. For example, for a struggling high school football team, winning a game or two might be a reasonable accomplishment at the end of the season. However, for a professi
by David Vance
November 30, 2005
In some instances, success is the achievement of something desired, planned or attempted. For example, for a struggling high school football team, winning a game or two might be a reasonable accomplishment at the end of the season. However, for a professional football team, losing just one game may be considered a failure, especially if that game is the Super Bowl. With the stakes for the players and the team so high, a professional coach is likely to need a more sophisticated measurement process to track the team’s performance throughout the season, identify areas for improvement and make adjustments to enable long-term success.
Measuring the success of learning is like playing in the big leagues: Organizations need an equally professional learning measurement method to assess how effective learning is in enabling business success. In addition, although measurement has typically been viewed as something that happens after the fact, learning measurement should be more predictive so that it can be a critical component of planning and enhancing learning interventions
Introducing Learning Effectiveness Measurement
Learning effectiveness measurement addresses the weaknesses in current approaches to learning measurement by providing a highly credible framework for measuring the quantitative and qualitative value of learning and guiding sound investment decisions. Using learning effectiveness measurement also helps training managers, designers, developers and evaluators plan, deliver and enhance training and learning initiatives to provide verifiable business value for their organizations.
While most will agree that there must be a causal relationship between learning performance and business results, few have been able to find it. Quarter-century-old models give sparse attention and provide little insight into on-the-job performance and business results measurement. Too many organizations have been satisfied with increasing the efficiency of learning, rather than its effectiveness at achieving its objectives and contributing substantively to business success. Indeed, the quixotic search for ROI is often misguided, undermining collaboration and actually devaluing learning’s highest contributions.
In contrast to traditional learning measurement, the rationale for learning effectiveness measurement is the well-established principle: What you measure is what you get. If you measure low-level outcomes alone, that’s what you will get. But learning effectiveness measurement does more than track effectiveness—it actually guides the design of more valuable and powerful interventions. It does this by better aligning learning with business results up front, not just measuring learning effectiveness after the fact.
Indeed, the major differentiator of learning effectiveness measurement is alignment. Without alignment, it is impossible to connect learning initiatives to performance and then to business results. Ideally, all learning measurement should be done top-down: An organization must first examine its strategy and begin to understand how its learning initiatives can drive it toward its desired goals.
The Five-Phase Process
Unlike most other approaches, learning effectiveness measurement implements learning measurement before the program is designed and continues throughout the program’s lifecycle. It consists of five phases:
- Predictive measurement.
- Formative measurement.
- Baseline measurement.
- In-process measurement.
- Retrospective measurement.
Predictive measurement starts early in the learning planning cycle and helps decision-makers target the highest-leverage performance improvement opportunities, develop the measurement plan and identify the critical success factors for optimizing the effectiveness of the program design.
During this phase, a causal chain is developed to help both learning professionals and business stakeholders understand the linkage (alignment) between learning and business results. The causal chain is a powerful tool for building more high-impact interventions and also identifies the key learning, behavioral, performance and business indicators that you can use to monitor the success of the learning intervention and make sure it is doing its job. It also provides the vehicle for identifying learning content that will change behavior and drive key performance measures that in turn will contribute to business impact.
The development of a causal chain will involve key stakeholders using a common language in a common search for learning impact and business results, rather than just deploying learning content. An added benefit is that causal chains can be reused, adapted and provide invaluable intellectual capital for the organization.
Once the causal chain is completed, the indicators at each level of the causal chain can be operationally defined into “measures” and become the basis of a powerful results-oriented plan for measuring all levels of learning effectiveness: learning, behavior change, performance and results. A word of advice: When selecting your performance measures, you should leverage those that are already being used in the organization, rather than inventing your own. You will find a surprisingly large amount of data already available for you to tap into as you measure the impact of your learning programs. The predictive measurement phase of learning effectiveness measurement culminates with the identification of the critical success factors—learning and non-learning—needed to drive learning effectiveness through the chain of causality and ultimately to business results.
Formative measurement is a qualitative assessment conducted during the program design reviews and pilot (if any) to determine whether the program’s content and implementation plan is adequate to achieve the goals identified in the predictive measurement phase. You will find that using a results-oriented learning measurement process dramatically changes your approach to design. Focusing on behavior change, performance improvement and business results, as well as learning measures, significantly changes the rules of the game for learning interventions. Designers not only need to address more than the standard learning enablers and inhibitors, but also must consider how behavior change is going to be transferred to the job and translated into sustainable performance improvement. Intervention design is no longer just a matter of disseminating knowledge and developing skills or designing attention-getting cues, appropriate visuals and relevant practice exercises. The challenge provoked by learning effectiveness measurement is about developing effective instruction that is wrapped in a package that might include non-learning components—such as process enhancement and management reinforcement—so that business goals can be met. You are likely to find that the mere fact that serious measurement is occurring might be the most powerful non-learning component of all.
Baseline measurement, too often the forgotten phase of learning measurement, provides the “before” data to compare the before-and-after differences. Baseline measurement provides the comparatives by which all progress can be determined. And of course, at the end of the program, it helps demonstrate the program’s ultimate impact and, if desired, ROI. Indeed, it is impossible to monitor progress or demonstrate effectiveness if there is no baseline data. Baseline measurement can be done at any time before the learning program is launched. When done early enough, baseline data also enables the establishment of target values for each measure, which can be useful input for program design since they indicate how large the gap is between current and desired performance.
In-process measurement should begin soon after the program is launched and continue throughout the remainder of the lifecycle of the learning program. It tracks program effectiveness during deployment and enables corrective actions to be taken in a timely manner. Just because an intervention has been carefully designed to meet learning and performance requirements doesn’t mean that it will be as effective as anticipated once it is deployed. In-process measurement is high-leverage because it typically requires little additional effort and can provide extremely valuable, ongoing feedback. Once the learning and performance measures have been identified, ongoing data collection should be relatively routine. By the time you get to in-process measurement, the measurement activity should be pretty well systematized and involve little more than tapping into the data that is already being collected. Multiple measurement points are important, but the actual number of in-process measurements depends on how critical your learning program is, and how much feedback you need to make sure your program is on track to achieving its learning, performance and business goals.
Retrospective measurement is the final phase of the process and is typically conducted when the program is fully deployed at the end of a learning initiative’s lifecycle, or at a point of program maturity. This measurement is done to collect post-program data and provide input for final evaluative decision-making. It might be helpful to think of retrospective measurement as the final in-process measurement point. While this phase cannot change the impact of the current learning program, it can provide valuable data that can inform decisions about future programs. At the end of the program, if the learning results show a major improvement over the baseline and are consistent with the performance and business objectives, then the learning executive has evidence for management that the organization is playing in prime time and playing to win.
Enabling Business Success
By implementing all five phases of the learning effectiveness measurement process, an organization can:
- Provide a common conceptual model and language for discussing learning relative to business issues.
- Focus learning programs on strategic business, rather than tactical issues, contributing to the organization’s financial success.
- Assist in prioritizing learning opportunities.
- Help develop more powerful learning and performance improvement solutions.
- Achieve greater organizational and financial support for performance improvement efforts.
Overall, learning effectiveness measurement is more than just a measurement methodology. It is an approach to managing the learning and performance improvement process to achieve the desired organizational impact. Its real power lies in its ability to drive the learning, performance and business outcomes, not just assess the extent to which outcomes were or were not achieved.
For example, recently a major mortgage financing company experienced the benefits of learning effectiveness measurement firsthand. Executives in the information systems division at this company decided that a key business objective was to turn learning into a strategic driver of competitive advantage. They believed that if employees could more quickly and effectively implement new IT projects, the company could be more responsive to market needs, ultimately driving more revenue and profit. Learning was vital to realizing this vision, but the company discovered that many of its existing learning programs were resulting in little more than knowledge dissemination. Existing measurements only showed how much training was delivered, how it was received by the participants and what the overall cost was. Recognizing that it did nothing to help define what training was needed, and could not be linked to business results, the company implemented a process to help address the effectiveness of the learning programs currently in place and what needed to change in order to achieve the desired business goals.
By implementing the methodology as part of its learning strategy, the development and analysis of a causal chain immediately revealed serious disconnects between the training programs, employee behavior and desired business impact. By leveraging these findings, the company predicted the potential impact of its learning program, documented the impact of past plans and identified the best ways to improve performance. These powerful insights led to significant changes in the company’s approach to training. For example, from its first introduction to learning effectiveness measurement, the training staff enthusiastically replaced training logic with business logic and identified the critical success factors for driving business impact from training.
One of the keys to success was forging closer partnerships between the training organization and internal clients, and emphasizing the pivotal role of line management in behavior change. Adoption of learning effectiveness measurement transformed high-priority training programs into business-critical training processes, with an increase in job-relevant activities. In addition, training was tracked with line-of-sight in-process measures to ensure that stakeholders could clearly see progress toward behavior change and business impact targets, and make real-time changes if the measures were not on-track.
As a result, learning measurement is no longer simply synonymous with satisfaction at this company, but involves the collection and use of many types of data to help stakeholders make more timely and relevant decisions—before, during and after training. It also involves senior management support and a partnership between all business units and internal clients, which is critical to the success of its learning program. Also, the company’s training team is confident that it has developed the fundamental knowledge and skills needed to implement learning effectiveness measurement in future programs.
This company and others that have adopted this approach know precisely what they need to step up their game and implement professional learning measurement methods to better align learning with business results up front, not just to measure learning effectiveness after the fact. This methodology ensures that issues can be identified early enough for learning executives to take action to improve how the game is played while they can still do something about it. That’s the way to contend for the learning and performance Super Bowl—and maybe even win it.
Dean Spitzer, Ph.D., is a consultant and innovator in the area of performance improvement, high-impact training, learning measurement and organizational change, with more than 25 years of experience. He can be reached at dspitzer@clomedia.com.