Measuring Impact and ROI in Executive Development Programs

By Scott D. Saslow, The Institute of Executive Development

The following article is an abbreviated version of “Measuring Impact and ROI in Executive Development Programs,” originally published in February 2006 with The Institute of Executive Development. In the original research, several executives representing multiple functional areas of responsibility were interviewed and their perspectives and business needs were explored in depth.

Today two-thirds of respondents stated that they “Rarely” or “Never” calculate the ROI on executive development (see table below, from market study Leadership Development: Challenges & Best Practices), although there are several factors that are leading to increased activity in tracking the impact of executive development programs within the corporations they operate. In some cases, beyond simply monitoring the impact, certain methodologies which track the return on investment (ROI) of developmental activity, including both an examination of the benefits as well as the costs, are deployed. The conventional wisdom is that like any corporate spending, executive development investments need to be fully understood and ranked against other company projects to ensure proper allocation of scarce organizational resources.

There are multiple factors that drive the need to track executive developmental investments:

• Increased accountability for all functional areas within the organization, driven by initiatives such as “Total Quality Management” and other process improvement programs. In areas such as Training & Development, which are known to produce a mix of tangible and intangible benefits, there is an increased desire by senior management to measure performance and outcomes.
  
• Increased spending on executive development programs in many organizations over the past several years, making developmental activity more visible across the organization, therefore requiring greater accountability.
  
• Increased outsourcing for all areas of corporate training and development, recognizing that many components of executive development are already outsourced. This trend has put additional pressure on the need to evaluate, quantify, and justify in-house executive development programs.
  
• Increase in Human Resource Development (HRD) professionals who are viewing their departments as stand-alone business units and are utilizing department profit and loss statements as one measure of performance. Increasingly, professionals from non-HR departments are joining HR teams and bringing a “bottom-line” perspective and discipline.
  
• Increased awareness by both executive management as well as HRD professionals on the availability of tools and techniques to quantify impact of executive development programs.

Question: Does your organisation attempt to calculate an ROI on executive development?
10%	Always	Each programme is evaluated independently
7%	Frequently	Most of our programmes are evaluated
15%	Sometimes	As appropriate we will evaluate some of our programmes
21%	Rarely	Only a small number of programmes are evaluated
46%	Never	We never measure ROI for leadership development
From “Leadership Development: Challenges & Best Practices” www.execsight.com/benchmarks

The factors that drive positive impact & ROI
Based on interviews and research, four key elements were identified which positively had an impact on the business benefits and ROI of executive development programs: Process, Instructors, Environment, and Learner Orientation.

Process In terms of the developmental process, all activities must be based on the strategic goals and needs of the organization. For high level executives with busy schedules and many pressures to perform, none can afford to attend training that is “nice to have”- it must be imperative and help executive’s get their jobs done. It is important to utilize relevant material, meaning generic case studies are less common, and real life data about the organization are more the norm. It is also important that leaders learn about themselves (and not just about different learning models and styles): “What are their specific leadership styles? How do they best assimilate new information and perspectives? What do they realize about their current inventory of skills, competencies?” This involves benchmarking and third party feedback, which helps to complete the picture. Also important is to hold developmental sessions that best suit their schedules- which may necessitate multiple formats (i.e. online based) to ensure training happens when and where the executives are.

Instruction Given that much learning happens from observing others, it is important to use existing managers/executives as instructors. They have credibility which is imperative with this audience. Also it is important to set up mentoring systems, since a great deal of informal learning happens this way, and to ensure that the culture is supportive of creating scenarios when junior executives can work alongside and watch the senior executives in action. Buddy systems allow learning to be discussed among peers, and ultimately the best way to solidify the learning is to have the participant’s facilitate upcoming programs.

Environment In terms of the learning environment, customization plays a significant role. It is important to find a good match between the individuals and the specific learning programs so that the design is relevant for them. Also, all developmental programs must have explicit support of the participant’s supervisors. They will play an important role in providing pre-program input on what the skill gaps are, support during the program (especially in cases when the participants need to be out of office), and opportunities to utilize new skills and management styles immediately to help reinforce and solidify the learning.

Learner Orientation The biggest beneficiaries will be those participants who are action-oriented learners;they are honest with their own shortfalls, constantly seek to better themselves, and are very “hands-on” throughout the process. They are the ones always asking questions in the discussions and are active contributors. Managers can play a role in encouraging this behavior and orientation by holding the subjects accountable for their developmental process. Participants need to report to their superiors what they intend to gain from given programs, and most importantly they need to report back after the programs what they learned and how they plan to integrate the knowledge into their work. Developmental activities should be part of the performance review (what gets measured gets managed).

Implications for Management Development
What will this trend toward increased usage and sophistication of techniques in measurement mean for management development?

One can expect a change in both the quantity and type of management development programs in some organizations. Already business schools are experiencing a dramatic shift in enrollments from their open enrollment programs to custom programs, as the latter are much more measurable and impactful on near-term performance. Additionally, custom programs are integrating action learning programs, those which blend on-the-job experiences with current organizational challenges as the classroom is extended well beyond the training center’s facilities. Some organizations have chosen to eliminate open enrollment programs for their executives and have stopped programs which bring in outside specialists for “generic” management development programs. However, one must question if the pendulum is swinging too far. Are organizations only developing leaders to deal with current issues and not broadening their managerial palette so that they may continue to contribute as well-rounded managers?

The theory is, especially with senior management, that they have made it into the executive ranks due to a demonstrated baseline of skills and competencies, therefore such generic developmental training is redundant and wasteful. In reality, some executives ascend through the ranks with one or many “general manager” deficiencies and thus such training, though difficult to quantify on near term financial statements, would be of value. It is too early to see whether the increased need for quantifying executive development impact will infact continue to shift the types of developmental activities and lead to a short-term and short-sighted approach to executive development.

For practitioners: it is important that they become well versed on common measurement techniques. They need to understand the pros and cons of each, and understand what is involved from a time and resource standpoint to conduct the measurement. Most importantly, they need to understand what is relevant in their organization. What metrics matter to whom? How often do they need to see the data? Ultimately, the developmental professional needs to keep track of the metrics they believe are best suited to determine the effectiveness of the programs they run, even if no one else in the organization is requiring them currently. This will help them deliver better programs and proving impact will likely be requested, given the increased awareness and activity in measurement.

For the service providers (covering the spectrum from business school programs, to consultants to leadership development companies): they must think about measurement from the start. They should expect to include this, if they are not already doing so regularly, as a standard part of any proposal. They can expect to be asked for case studies with metrics for past client engagements. Ultimately, development professionals should take the lead in finding out what is important to the organization they are serving (understanding its business and key drivers) and then work hand in hand with the organization to determine the development impact metrics that matter.

Best Practices in Measuring and Maximizing ROI in Executive Development Programs

Practice	Description
Pre-Program
Link Development to Strategy	Start with the organization’s key strategies, conduct a “gap analysis” to determine the shortfall in needed talent to execute the strategy, and from this create the developmental plan for executives.
Gain Stakeholder Support	Ensure that the managers of the participants are fully aware and supportive of the developmental programs.
Pre-testing	Conduct pre-program evaluations to determine the baseline prior to developmental activities.
Expectation Setting	Ensure expectations are accurately set with the participant’s managers, peers, and subordinates in terms of the time, effort, and expected outcome of the developmental program. Managers in particular need to realize their role in holding participants responsible for developmental activity.
During Program
Instructor Credibility	Ensure that program instructors have credibility in the eyes of the participants- they either come from within the organization and are viewed as successful executives, or they are outside specialists with proven success in the topics at hand.
Real time Feedback and Modifications	Ensure to capture feedback during the program and have the ability to act on it, especially relevant for week-long programs which represent a significant time and cost investment.
Custom, Action Learning	Designing program content based on the specific organization’s current challenges, and allowing opportunities for integrating the developmental activity into the ongoing work of the participants.
Post-Training
Data Collection	Ensure to collect data from multiple parties such as program instructors; managers, subordinates, and peers of participants; and outside specialists. Also collect data from multiple time frames (3 month, 6 month, etc.).
Compare to Focus group	When possible, use a focus group which has not yet gone through the program to establish a point of comparison for the program effectiveness.
Audit Impact	Occasionally utilize outside specialists to track program effectiveness to confirm internal findings. The credibility and new processes of outside specialists can help produce novel findings.

About the author
Scott Saslow, Executive Director, The Institute of Executive Development
Scott Saslow is the Executive Director of The Institute of Executive Development, a company that delivers a unique service to executive development professionals by providing an unbiased source of information on the best practices and innovative ideas in executive development. Clients of The Institute include professionals from global companies who manage their organization's executive development programs, as well as recognized experts who advise, publish, and speak on industry topics. The Institute is the largest in its field and most dynamic in terms of its services. There are currently over 4,000 individuals from 85 countries who use the Institute. As head of the organization, Mr. Saslow is responsible for all strategy and operations. Mr. Saslow also manages the design and delivery of multiple market studies in executive development such as "Transforming Corporate Leadership: Best Practices in Executive Education" and "Leadership Development in European Organisations". In all, Mr. Saslow has authored over 15 articles and in-depth research reports on the field of executive development for publications such as CLO Magazine, Strategic HR Review, Leadership in Action, and ASTD's Leadership / OD Newsletter.

Mr. Saslow has been interviewed by and quoted in Harvard Business Review, Forbes, Business 2.0, Workforce, Chief Learning Officer, Training, Globe & Mail (Canada), and Business Management. He also serves on the Editorial Board for The Academy of Management's Learning & Education Journal, and is a contributor to Leadership In Action, The Center for Creative Leadership's Management Development publication. A frequent speaker on issues relating to executive development, Mr. Saslow has spoken at The Global Forum on Executive Development and The Chief Learning Officer Symposium among other events.

More information
> Please send us a mail with your comments to this article.

> Read more about corporate leadership development

For more information about this survey, please visit www.execsight.com

Shortcut to this article:
http://www.mannaz.com/Mail.asp?MailID=118&TopicID=1374