Why training can make or break

a good service model

by Jon Sweigart, Manager of Training & Development and Isidoro Gonzalez, Senior Vice President of Programs

Many great models have been developed that have demonstrated to be effective in creating social change, but often do not produce the same outcomes when replicated or scaled.  Core to the effectiveness of any program model is the training and support to the individuals delivering the service. 

In 2008 Congreso created a new client management model to achieve better client outcomes.  In addition to investments in the Primary Client Manager Model Congreso invested in the addition of a training manager to devise a strategy to support its implementation agency-wide.  In short, the business plan directly connected employee training and development to service quality and client outcomes.

In developing the PCM model, Congreso articulated a general approach to client change based on a brief health behavior change counseling model and a data system connecting employee efforts more directly to client outcomes. PCM Basic Training is Congreso's first-ever sustained training initiative directly connected to business plan imperatives and as such is the agency's theory of change in action. To date, Congreso has trained over 100 client managers and supervisors in PCM service delivery basics.

The training also provided one piece in the comprehensive strategy to attract, select, develop and reward high-performing client managers. PCM Basic Training sets a standard for skills and knowledge competencies among client managers who deliver PCM services. In so doing, the training brings together multiple professional development and client outcome goals:

• Improvements in service quality and client outcomes. Congreso measures service quality by level of fidelity to the PCM model and the number of clients who achieve or are on track to achieve high school graduation, secondary education (for adults), sustained employment, health and family stability.
• Improvements in employee retention (especially among those engaged in Primary Client Management, Congreso's service delivery model). Congreso's attrition rate is approximately 35% annually. Direct service employees have been traditionally difficult to retain. The Primary Client Management model depends on sustained, supportive helping relationships. The connection between retention, service quality, and client outcomes is obvious.
• Clear, competency-based career pathways. A clear path for career development motivates employees and provides opportunities to develop professionally through training, on-the-job learning, and supervision.

In its early stages, training has established a standard for compliance (Are they doing it?).   However, as the agency matures in its ability to carefully select, develop, evaluate and reward employees based on competency-based performance, service quality becomes the metric (How well are they doing it?), connecting the work of employees to client outcomes in a way that captures the dynamics of a helping relationship aimed at influencing lasting social change both in individuals and the community.

What we learned is that effective model delivery does not just happen because an individual is trained.  To be effective the training needs to be followed up with supervisory coaching in which the supervisor reinforces the principles of the model, discusses the challenges within its implementation, and helps  develop solutions to unanticipated barriers.  It is our understanding that the real training happens not in the classroom, but through the provision of a constant feedback and improvement loop.  The ideal situation, although not always possible, is training through peer to peer feedback, but given the limitations of time a supervisory coach can reach the same outcomes.

We measure the effectiveness of PCM training at several levels roughly parallel to Donald Kirkpatrick's widely-referenced model of four levels of evaluation: reaction, learning, behavior and results. What we are learning from this evaluation helps guide our performance improvement efforts. Four basic questions correspond to each of Kirkpatrick's levels:

• Did they like it? Congreso surveys participants about the training itself immediately after the completion of Basic Training. This survey measures initial emotional reactions to the training and provides feedback for facilitators, but says little about the effectiveness of training beyond the event itself.
• Did they learn anything? Congreso conducts a brief knowledge survey after PCM Basic Training to make sure participants share an understanding of basic concepts that contribute to a standardization of service delivery across the agency.  Supervision and coaching about depend on a consistent, clear, and accurate articulation of the model.
• Are they doing it? Congreso's Efforts-to-Outcomes data system (UNIDAD) allows for an increasingly in-depth evaluation of the transfer of learning from classroom to client interactions. Initially, evaluation at this level focused  primarily on compliance, but has since moved more toward qualitative analysis ('How well are they doing it?'). Supervision is the most important link that promotes the transfer of learning from classroom to actual work. Peer-to-peer information exchange also contributes. After an initial focus on basic training, Congreso training efforts have now shifted toward supporting the model through strengthening supervision and creating informal learning opportunities for client managers.
• Was it worth it? This level of evaluation seeks to determine return on investment. The model and its evaluation are too early in development to say much at this level currently. It would seem that client high school graduation and workforce participation would be the best ultimate measures of return on investment. However, recent discussions have acknowledged that a client manager's fidelity to the model may be a better measure of training effectiveness. If client managers demonstrate fidelity to the model as they were trained to do, but outcomes do not respond as expected, then a re-examination of the model's effectiveness seems in order.

Return to Annual Report

Return to Congreso hompage