Research Paper By Lene Brendstrup
(Executive Coach, DENMARK)

Introduction

The coaching industry is growing and the companies’ spending on executive coaching is increasing. Therefore the need for measuring the effect of executive coaching is getting increasingly important. The paper is answering the question ‘Is it possible to document an effect of executive coaching?’ by looking into what researches exist to document the effect. The purpose of this paper is not to provide an overview of all related research, but to seek clarity on what is possible to conclude in relation to the question of the paper. The included empirical sources are selected based on high level of citation by others and to what degree they are contributing to answering the question of this paper.

Definition of executive coaching

International Coaching Federation (ICF) defined in 2003 ‘Professional Coaching is an ongoing professional relationship that helps people produce extraordinary results in their lives, careers, businesses or organizations. Through the process of coaching, clients deepen their learning, improve their performance, and enhance their quality of life’. There are several definitions of executive coaching, and the main expansion of the ICA definition is that the client/coachee is a manager/executive with a set of goals to improve his/her professional performance.

Importance of Measurement

The use of external executive coaches is an increasingly popular trend in corporations (Graddick & Lane, 1998). Many of the world’s notable corporations and government organizations utilize executive coaching, and the interest in executive coaching continues to escalate (Hill, 2010).

The popularity in usage of coaching has also attracted many practitioners. The 2012 ICF Global Coaching Study estimates 47,500 professional coaches worldwide in 2011 compared to 30,000 in 2007. The 2012 ICF Global Coaching Study also estimates the professional coaches generated close to 2 billion USD annual revenue/income.

With the growing usage of executive coaching it has become more important to be able to measure the effect of executive coaching in order to justify the increasing spending. Also among the coaches in the 2012 ICF Global Coaching Study 30% mentioned ‘confusion in the market place’ as a key issue facing the coaching industry.

Adding to the need for measuring the effect of executive coaching is the fact ‘Coach’ is not a protected title. Anybody can label him/herself coach as coaching is not registered as a profession. The International Coaching Federation (ICF) had almost 19,000 members at the end of 2011 according to the 2012 ICF Global Coaching Study, which are less than half the estimated coaches in the world. The absence of regulation and consistency associated with a recognized profession has prompted professionals in related fields to voice concerns about quality and effectiveness in the conduct of executive coaching (Kampa-Kokesch & Anderson, 2001). In the 2012 ICF Global Coaching Study 43% of the coaches are identifying ‘untrained coaches’ as one of the key issues facing the industry.

Four Levels of Evaluation plus ROI

Donald Kirkpatrick (1994) recommended evaluating the effectiveness of development programs in four levels. It was originally developed in the 1950’s, but more recently ROI has been added to the evaluation as the fifth level in 1997 (Phillips, 2005). The 5 levels of evaluation are:

Reaction
Learning
Behavioral change
Business results
Return on investments (ROI)

The fundamental idea of Kirkpatrick (1994) is to have the evaluation including all levels to ensure a high level of documentation.

Kirkpatrick (1994) developed the model for all kind of training programs. In the following Kirkpatrick’s approach will the used specifically in the context of coaching.

Level 1 – Reaction:

The participants own reaction to and opinion about the coaching is the basic level of evaluation. This will normally be done in a questionnaire with closed as well as open ended questions. It is important because the likelihood of creating positive results from the coaching will increase if the coachee actually is positive about the coaching. Another reason is that if the coachee does not find the coaching useful; it is likely that the company will not continue the executive coaching. Finally the feedback from coachee will help to further improve on the effectiveness of the coaching.

A lot of research has been done on this level of evaluation. One of the more resent and most cited is the empirical research done by Wasylyshyn (2003). The coaching of the 87 participants was combined with 360 degree survey. But the measurement of the effect of coaching was based on interviews with the coaches. 63% of the participants indicated that they experienced sustained behavior change, 48% reported increased self-awareness and understanding, and 45% cited that they were more effective leaders. Even though the research by Wasylshyn (2003) is trying to measure all the evaluation levels from 1-3 it is purely based on self-ratings by the coachees and therefore the main conclusion of this research paper is that the coachees themselves are very positive towards the coaching (level 1).

Another supporting evidence to positive reaction to coaching (level 1) is the general perception of coaching seems to be increasingly positive. The Sherpa Executive Coaching Survey (2012) shows that among non-coaches, the number of people who see the value of coaching as ‘very high’ is now above 60%, which is an all-time high compared to around 40% in 2009.

Level 2+3 – Learning and Behavior Change:

The second level of evaluation is to determine what the coachee has learned. When the program is skill training it can be tested in some kind of performance test according to the skill in question. In the case of coaching the objective will normally be attitude and behavior related. Coaching will support these objectives by increasing the self-awareness of coachee.

Behavior change (level 3) will measure if the coachee is succeeding in transferring the learning from level 2 into behavior changes in his/her job. The coaching should ensure coachees understand the general principles behind the behaviors and thereby helping the wanted changes in the job. It is recommended by Kirkpatrick (1994) to have both pre- and post-test as well as a control group.

Kirkpatrick (1994) has two separate levels (level 2 for learning and level 3 for behavior change), which is very relevant for training programs, which was the original purpose of Kirkpatrick’s model. In this research paper it has been decided to combine the level of learning and behavior for evaluating the effect of coaching. The self-awareness of coaching is the fundament for the behavior change coming out of coaching, and therefore the measured change of behavior will demonstrate the effect of coaching in both learning and behavior.

The Level 2-3 is important because the company wants actual changes in behavior, which can lead to positive effect on the business results.
Several published research papers have been trying to evaluate coaching in level 2-3 by measuring coachee via multi-rater feedback instruments. The most cited of these research are Smither, London, Flautt, Vargas and Kucine (2003), Thach E.C. (2002) and Kampa-Kolasch (2001). The best of the researches for level 2-3 is Smither et al. (2003), which has implemented a multi-rater feedback instrument as both pre and posttest and at the same time including a control group (who is not receiving coaching) as recommended by Kirkpatrick (1994) to document the effect of executive coaching.

Smither et al. (2003) conducted the research with 1361 managers who all did a 360 degree test before and after the coaching. 404 received coaching and the rest did not. The result showed improvement of ratings of direct reports and supervisor (but no significant improvement in ratings from peers).