Level 4 – Business Results:
This level is to measure if the coaching has had any effect on the final business results. The level includes any outcome that most people would agree is “good for the business.” The outcomes are either tangible or intangible business results. Also in this phase it is recommended by Kirkpatrick (1994) to have both pre- and post-test as well as a control group.
Few empirical research have been evaluating coaching from level 1 following through to level 4. One of the most cited is Olivero and Bane and Kopelman (1997). 31 people went through training and after that 8 of them were trained to become coaches, who coached the remaining 23 participants.
- The measurement of level 1 showed that the coaches had a favorable reaction to the coaching process.
- The measurement of level 2-3 showed a 20% increase in knowledge after the coaching, but the sample size of both pre and post tests were too small to give any statistical inferences.
- The last part of the research of level 4 showed a 65% increase in productivity after training and coaching compared to 22% after the training alone.
- These results from the research by Olivero et al (1997) strongly suggest coaching having a positive effect on productivity.
Return On Investments (ROI):
ROI measures the relationship between the monetary value of the business result and the cost of the coaching.
The most cited research which has gone through all four levels of Kirkpatrick plus measuring ROI was done by McGovern, Lindemann, Vergara, Murphy, Barker and Warrenfeltz (2001). The study is often referred to as ‘The Manchester Study’ (published in The Manchester review, 2001, Vol 6, no 1). There were 100 participants in the research, who all held positions of vice president or above. The coaching duration ranged from 6-12 months and included personal instruments, multi-rater surveys and interviews with members of the multi-rater survey sample.
- The measurements in level 1 showed 86% of participants were ‘very satisfied’ or ‘extremely satisfied’.
- The level 2/3 was measurement by degree of achievements of their goals. 73% considered that they had achieved their goals ‘very effectively’ or ‘extremely effectively’.
- The measurement of level 4 and ROI was estimated in a most conservative manner, and the calculations showed an average ROI for the participants of $100,000 in average or a return on 5.7 times the initial investments in coaching.
- According to Phillips (1997) a training program should generate at least 25% return on investments the first year. Therefore the authors of the Manchester Study are concluding the positive effect of coaching very confidently.
Besides the coaching, many other factors will influence the business result of the company using coaching – the situation of the overall line of business, the organizational context, the circumstances of the coachee etc. One of the most difficult things in the measurement of the effect of executive coaching is to isolate the coaching effect from all other factors. Therefore a control group, who is not receiving coaching, needs to be part of the research design to produce the most reliable measurements on the effect of executive coaching. Only one of the above most cited research has included a control group in the research design: Smither et al. (2003).
Future measurements of effectiveness of executive coaching are recommended to follow the four levels by Kirkpatrick (1994) plus ROI, and most importantly also include control group (receiving no coaching) to ensure that the effect of executive coaching is isolated from all other factors influencing the overall business results of the company.
The ideal situation would be not only to prove that executive coaching is effective but also to document under what circumstances is executive coaching most effective. In order to identify the best possible circumstances for using coaching the future research should look into the effect of coaching related to the following external factors:
- Purpose of using executive coaching (e.g. support to 360 degree feedback, support new leadership job, support career advancement, overcome behavior problems etc.)
- Business environment of the company (e.g. declining, growing, innovative etc.)
- Organizational context (e.g. change management, support from supervisor and HR etc.)
- Not only the external circumstances are interesting to get documented but also the actual coaching setup is important. Future research of executive coaching should look at the effect of coaching related to the following internal factors:
- Coaches (internal/external, differences in experience, background, style and method)
- Coachee motivation and reason for coaching (top/bottom performers, High/low EQ, extrovert/introvert etc.)
- Duration of the executive coaching (frequency and length of coaching)
- Role of internal ‘stakeholders’ or ‘collaborators’ like boss and HR (support, provide ongoing feedback, image of coaching program (e.g. for selected high performers) etc.)
Based on the research findings it is possible to prove a positive effect of executive coaching but it is not possible to make unambiguous conclusions to what extent. Too few research have been including pre-test which makes it difficult to quantify the effect. Adding to the problem is that most research have not been including a control group, which has not been receiving coaching. This is a must in order to isolate the effect of coaching from all other factors influencing the business result of the company.
Another issue is that at this point of time there is not much evidence to support a clear guideline to when it is recommended to use coaching; as it cannot be documented under what circumstances coaching will be the best choice for the company.
Even though the research paper can conclude a positive effect of executive coaching, more empirical research is needed to quantify the effect. Overall the above mentioned issues with the measurements of executive coaching give a “black box” feel to executive coaching.
Feldman D. C. & Lankau M. L. (2005). Executive Coaching: A Review and Agenda for Future. Research. Journal of Management 2005; 31; 829
Fillery-Travis A. & Lane D. (2006) Does coaching work or are we asking the wrong question? International Coaching Psychology Review Vol. 1 No. 1
Graddick M. M. & Lane P. (1998). Evaluating executive performance. In Smither J. W. (Ed.). Performance Appraisal: State-of-the-art in practice. San Francisco, Jossey-Bass
Hill G. (2010). Executive Coaching: Perspectives of Effectiveness from Executives and Coaches. Queensland University of Technology.
International Coaching Federation (2012). 2012 ICF Global Coaching Study. www.coachfederation.org/coachingstudy2012
JOO B.K. (2005). Executive Coaching: A Conceptual Framework From an Integrative Review of Practice and Research. Human Resource Development Review Vol. 4, No. 4 December 2005
Kampa-Kokesch S. & Anderson M. Z. (2001). Executive Coaching. A Comprehensive Review of the Literature. Consulting Psychology Journal: Practice and Research. Vol 53, no 4.
Kirkpatrick D. L. & Kirkpatrick J.D. (2009). Implementing the Four Levels: A Practical Guide for Effective Evaluation of Training Programs. Berrett-Koehler Publishers. (Original work published 1994).
Mcgovern J., Kindemann M., Vergara M., Murphy S., Barker L. & Warrenfeltz R. (2001). Maximizing the impact of Executive Coaching: Behavior Change, Organizational Outcome and Return on Investments. The Manchester Review, Volume 6. No 1.
SALING N. E. (2005). An Empirical Study Comparing the Effect of Feedback, Training, and Executive Coaching on Leadership Behavior Change. Graduate Faculty of North Carolina State University in partial fulfillment of the requirements for the Degree of Doctor of Education
Olivero G., Bane K. D. & Kopelman R. E. (1997). Executive coaching as a transfer of training tool: Effects on Productivity in Public Agency. Public Personnel Management, Volume 26, No 4.
Passmore J. & Gibbes C. (2007). The state of executive coaching research: What does the current literature tell us and what’s next for coaching research? International Coaching Psychology Review.Volume 2 No. 2 July 2007
Phillips J. J. & Phillips P.P. (2005). Measuring ROI in Executive Coaching. International Journal of Coaching in Organizations, 2005, Volume 3, Issue 1
Smither J. W., London M., Flautt R., Vargas Y & Kucine I. (2003). Can Working with an Executive Coach Improve on Multisource Feedback Ratings over time? A Quasi-Experimental Field Study. Personnel Psychology; Spring 2003; 56, 1
Thach E. C. (2002). The impact of executive coaching and 360 feedback on leadership effectiveness. Sonoma State University, Rohnert Park, California, USA
The Sherpa Executive Coaching Survey (2012). 7th Annual report. www.sherpacoaching.com/survey.html
Wasylyshyn K. M. (2003). Executive Coaching: An Outcome Study. Consulting Psychology Journal: Practice and Research, Vol. 55, No. 2