Evaluating Training Results
A Case History from the Mid-West
How can managers know that training has paid off? Have training objectives been achieved? Are managers satisfied with what training has accomplished? These questions have dogged trainers, performance improvement professionals and managers since the earliest days of industrial adult education at about the turn of this century.
Though many adult-learning and management theorists, pioneers such as Bramley, Brinkerhoff, Dressler and Clegg, have made efforts to provide tools to help answer these questions, the most practical and popular of these tools is the Kirkpatrick model by Donald Kirkpatrick.
The Kirkpatrick model presents four levels of training evaluation:
Level 1: Participant reactions to training and the training environment.
Level 2: Learning acquired by participants through training.
Level 3: Application of learning to on-the-job situations.
Level 4: Impact of training on business performance.
In this model, each successive level provides more credible data that permit an objective evaluation of the impact of training. Also, each successive level becomes more complex and more time-consuming. In this Case History, analysis is restricted to looking at levels two and three.
Background
‘T’, a Dublin-based trainer, had the opportunity to develop and present leadership skills training at M Ltd, a small (20 employees) manufacturing facility in the Mid-West. ‘T’ trained three key employees: a production manager, a supervisor and a senior employee.
The goals were to train participants to identify and resolve interpersonal and technical problems; manage conflicts and disagreements more effectively; and to communicate more clearly with subordinates, with each other and with management. The average formal educational level of the trainees was Pass Leaving Cert. With the exception of one participant, none had participated in any formal skills training.
Of the 32 hours of this programme, 20 were classroom training and 12 were one-on-one coaching. Coaching was used to help employees transfer the skills acquired in the classroom to the shop floor. Coaching sessions took place once a week, and were structured around the training objectives for that week and personalised to the learning needs of each participant.
‘G’, the general manager of the facility, reinforced the classroom training by conducting a series of evaluation meetings.
Level 2: Learning and the Evaluation Meeting
The evaluation meeting is fairly straightforward in both design and execution. Its purpose is to help managers assess the extent to which new skills were acquired and if these skills are used on the job. It's also a way for line managers to be directly involved in evaluating training effectiveness. Such a format clearly demonstrates top management’s interest in training’s success and support for training efforts. It also strongly communicates to trainees that management is interested in their performance during and after classroom training.
Toward the end of the training, ‘G’ told the participants that he wanted to meet with them two weeks after training was completed to determine how much they had learned. He asked the three trainees to present individual 30-minute summaries of key ideas or skills acquired during training. He told them to decide among themselves which topic and skills areas each would present. ‘G’ offered to assist them in preparing their presentations by helping them create outlines and providing them with supplies and administrative support.
About a week later, ‘G’ discovered that the trainees had not agreed on how to make their presentations. He asked trainees to refer to their training manuals and divide the topics among themselves by subject matter so that each participant presented approximately one-third of the training content.
At the evaluation meeting a week later, there were three presentations. The first trainee presentation covered modules one through four addressing interpersonal communication, leadership, conflict resolution and delegation. The second presentation of modules five through eight addressed motivation, discipline, problem solving, train-the-trainer skills and legal issues. The third presentation dealt with the how to use the Ishikawa diagram to solve work problems.
During each presentation ‘G’ asked questions: What was the most important skill you learned? What are the most common conflict-resolution styles that you see us using here at M Ltd.? How do you use the four basic communication styles with the people you supervise?
Level 3: Changes on the Job as a Result of Training
Overall, results were mostly positive. The employees reported that the training had enhanced their ability to communicate and listen. Evidence of improved communication at work was noticed in less time spent by the manager and the employees in locating materials, messages and equipment. Better communication also improved message flow and reduced lost time. For example, after training ‘G’ instituted 30-minute overlap meetings between day- and night-shift supervisors. Previously, shift overlap lasted no more than five minutes, which resulted in confusing or missed communication. With increased overlap, day- and night-shift problems related to scheduling, machine operation and personnel issues were minimised resulting in shortened lines of communication and a smoother turnover of work shifts.
Other evidence that some classroom learning was being applied to workplace situations was the institution of Monday morning meetings among the general manager, the production manager, the supervisor and the senior employee to review issues and anticipate problems for the upcoming week. Prior to training, these meetings were sporadic and did not always include all key personnel.
How the Process Can Be Improved
‘T’ submitted report at the conclusion of training recommending that managers and the three trainees needed to develop formal job descriptions defining the trainees’ duties and responsibilities, and that management and trainees needed to define performance and career-development goals. Additional recommendations included formally training them in basic quality tools, developing work teams, and training temporary employees and other shop-floor personnel.
The evaluation meeting was a useful way to assess the overall value of training to M Ltd. Though management and those involved in the training agreed that the training was worth the investment, future evaluation meetings could be improved in the following ways:
- advise participants on how to give a presentation. This should include coaching on how to select appropriate presentation media and how to prepare an outline.
- allow participants to rehearse their presentation. This preparation is necessary if the evaluation meeting is to yield desired results.
- discuss what each person expects from the evaluation meeting prior to the meeting. Trainees and managers should agree on expectations and prepare an evaluation meeting agenda reflecting these expectations. Distribute the agenda before the evaluation meeting.
- document results from the evaluation meeting. This can then be used to plan further meetings, and as the basis of discussion between management and key personnel about career development needs.
- define the purpose of the coaching. The contribution of coaching to the overall effectiveness of this training effort was unclear. This is partly because the evaluation meeting did not directly address what was learned from the coaching sessions. Consequently, there was no format to evaluate the extent to which coaching did or did not supplement the classroom activities. Future coaching efforts could be improved by more formally and explicitly linking coaching to classroom work, and by using the evaluation meeting to assess the coaching's relationship to classroom accomplishments.
Managers at M Ltd. concluded that the evaluation meeting was a cost-effective way to assess training impacts and permitted them to draw conclusions about the value of training to the organisation. In addition, the training effort clearly demonstrated management’s interest and support in training and developing employees
-------------------------------------------------------------------------------------
Corporatetraining.ie is interested in receiving further Case Histories on training interventions: with suggestions on what went well and on what could be improved. Send your articles to info@corporatetraining.ie
