Evaluation of the Career on the Move Development Program

Final Report
November 12, 2009

[ PDF 226 KB ]


Table of Contents

  1. Executive Summary
  2. 1.0 Introduction
    1. 1.1 Overview of Career on the Move
    2. 1.2 Program Context
    3. 1.3 Program Objective
    4. 1.4 Intended Beneficiaries
  3. 2.0 Evaluation Methodology
    1. 2.1 Scope and Objective
    2. 2.2 Logic Model for Career on the Move
    3. 2.3 Evaluation Questions
    4. 2.4 Methodology
    5. 2.5 Limitations of Methodology
  4. 3.0 Findings
    1. 3.1 Relevance
    2. 3.2 Performance
    3. 3.3 Cost Effectiveness/Alternatives
  5. 4.0 Conclusions and Recommendation
  6. 5.0 Management Action Plan
  7. Appendix A - Evaluation Matrix
  8. Appendix B - Budget Variance Analysis
  9. Appendix C - Comparison with Similar Programs

Executive Summary

Introduction

Career on the Move (COTM) is a developmental program targeted at talented, high-potential members of three employment equity groups who are already at the Executive (EX) minus one, EX minus two or EX equivalent levels. Successful applicants to the program receive a one- to two- year secondment assignment at their substantive group and level at the Privy Council Office (PCO).

The COTM program is not a management training program, although the experience gained at PCO could significantly help an individual progress toward the executive category. Introduced in 2005, there have been 30 COTM participants placed into secondment positions at PCO to date. The Human Resources Division of the Corporate Services Branch manages the program.

Scope and Objective

This evaluation covers a three and a half year period from the first COTM intake in August 2005 to March 2009, and examines the extent to which the program has achieved its outcomes over this time. The objective was to provide evidence‑based information on the program’s relevance, performance, and cost effectiveness, which is in line with the Treasury Board Evaluation Policy.

The commitment to evaluate this program was identified in the 2005 Employment Equity Fund: Application for Funding submitted to the then Public Service Human Resources Management Agency of Canada (PSHRMAC), as well as in the 2008‑2009 Report on Plans and Priorities for PCO.

Key Findings and Conclusions

After four years of operation and three separate intakes, COTM is at a crossroads. While the evaluation has identified some successes, it has also revealed weaknesses that have implications concerning the program’s relevance, performance, and cost-effectiveness. Key findings and conclusions for each of these evaluation issues are discussed below.

Relevance

The evaluation has shown that the COTM program is consistent with the government priority to improve representation of members of designated employment equity groups, reflective of their workforce availability, within the executive ranks of the public service. The importance of achieving these levels of representation has been featured prominently in the Clerk’s Public Service Renewal initiative and has been a consistent message in his annual reports to the Prime Minister on the public service.

The placement of the COTM program at PCO, however, is inconsistent with recent government changes to streamline the public service human resources (HR) governance structure. The Prime Minister’s Advisory Committee on the Public Service found the previous HR governance structure to be “overly complex, with multiple players and a resulting burden of duplicative and often unnecessary rules. 1COTM is symptomatic of the problems observed by the Prime Minister’s Advisory Committee. PCO and the former Canada Public Service Agency (CPSA), now the Office of the Chief Human Resources Officer (OCHRO), operate competing development programs that have similar objectives and methodologies, and target the same level of public servants. Under the renewed HR governance structure, as a central agency, PCO’s roles are limited to providing support to the Head of the Public Service, supporting Public Service Renewal, and providing talent management for deputies and Governor in Council appointments.

Performance

During its first intake of participants that occurred in 2005-2006 the COTM program was very successful in attracting and placing participants into the program with 18 placements that first year. The second and third intakes, however, experienced significant declines in the number of new participants, dropping to eight in 2007 and most recently four in 2008. There are still sufficient numbers of public servants wanting to come to PCO as program participants; however, this interest is not translating into successful placements. PCO managers have not seen the right mix of knowledge, skills, and experience in candidates put forward for their consideration. The program is focused on producing an adequate supply of candidates without sufficient consideration of the operational demands of the positions intended to be filled by COTM participants.

There is some indication that participants have been successful in developing key leadership competencies of strategic thinking and engagement, and to a lesser extent values and ethics; in-line with the key leadership competency profile used as a main component of the EX Qualification Standard. However, participation in the program has not as of yet translated into success in competitions at the executive level. Among the 86 percent of participants, past and current, who completed the evaluation survey, none have obtained an executive level position through the competitive process; though, three past participants did indicate that they are currently occupying executive level positions obtained either through appointment or in an acting capacity.

The transition process at the end of the program back to either the participant’s home organization or their next assignment has not resulted in effective use of knowledge and experiences participants gained while in the program. Only 18 percent of participants indicated that they were provided an opportunity to use new skills and abilities after leaving the program. A key contributor to this finding is a lack of communication between PCO and home organization managers before, during, or after the COTM assignment.

Cost Effectiveness

Since program inception in 2005, COTM program support and operating expenses, not inclusive of participant salaries, have totaled about $1.3 million, which equates to almost $39,000 per participant on an annualized basis. Cost-effectiveness could be improved if the number of participants increased to meet expected levels as some program costs do not vary directly with the number of participants and therefore would not grow substantially as participation grew.

The travel and accommodation category of operating expenses stood out from the others due to its size relative to all other program expenses. Travel and accommodation represents 35.5 percent of all program support and operating expenses, most of which is attributable to a limited number of regional participants. Regional participants are considered to be on travel status for the one to two years they are in the program and are compensated based on the Treasury Board Secretariat Travel Directive. As a result of this practice, when segmented between regionally and locally based participants, the annualized average cost to host a regional participant is about $76,500 or almost 300 percent of the $26,000 cost for a locally based participant. Similar development programs administered by the OCHRO establish much more modest limits for relocation costs.

Recommendation

The results of the evaluation do not support the status-quo as an option moving forward. The evaluation recommends that management consider alternative approaches to achieving the objectives of the COTM program. That said, regardless of the course of action taken by management, it should be made clear that PCO continues to support and encourage advancement of visible minorities, Aboriginal people, and persons with disabilities.

Management should consider alternative approaches to achieving the objectives of the COTM program. The evaluation presents three possible approaches that could be taken by management including implementing targeted improvements to the COTM program; enhancing employment equity aspects of the PCO Operations Branch Analyst Initiative; or increasing support for similar development programs operated by the OCHRO. Selection of the latter two approaches would imply an end to the COTM program.

Management Response

It is agreed that the status quo is not a viable option for COTM moving forward. The integration of the Employment Equity component into the PCO Analyst Recruitment Initiative has been identified as the preferred option.

The Assistant Deputy Minister of the Corporate Services Branch and the Deputy Secretary of the Operations Branch are jointly responsible for the development of the Management Action Plan, presented in section 5, going forward.

Original signed by the Chief Audit and Evaluation Executive

Signature of the Chief Audit and Evaluation Executive
Jim Hamer

1. Introduction

1.1 Overview of Career on the Move

Career on the Move (COTM) is a development program introduced at the Privy Council Office (PCO) in 2005 to provide opportunities to promising members of the federal public service from designated employment equity groups to broaden their skills and experience and help them prepare for higher-level positions. The COTM program is not a management training program, although the experience gained at PCO could significantly help an individual progress toward the executive category.

During its first year of operation COTM received external funding from the Employment Equity Fund administered by the then Public Service Human Resources Management Agency of Canada (PSHRMAC). That fund has since been closed and now PCO funds the program directly. Program operating and administrative costs fall within the budget of Corporate Services Branch, while participant salaries are paid by the host secretariats. To date, 30 participants have been placed into secondment positions under the program.

1.2 Program Context

The Government of Canada recognizes that, as an institution that is critical to the nation's continued economic, social and cultural prosperity, the public service must be dynamic and innovative, drawing on the talents of Canadians of all origins, cultures and views.

The implementation of employment equity in the core public administration is based on a strong legislative framework. The Employment Equity Act of 1995 requires employers to institute positive policies and practices to ensure representation reflective of that in the Canadian workforce or in those areas that are "identifiable by qualification, eligibility or geography and from which the employer may reasonably be expected to draw employees." The revised Public Service Employment Act of 2003, which came into force in 2005, reinforces the importance of HR planning in staffing and recognizes the importance of a core public administration that reflects the diversity of Canadian society.

More than a decade after the Employment Equity Act came into force, the renewal of the public service is still a priority, but it has taken on greater importance and momentum. The average age in the public service including members of designated employment equity groups has risen, especially at senior levels. Like other employers, the public service faces stiff competition in its efforts to recruit knowledge workers to replace retiring employees.

1.3 Program Objective

COTM is designed to enable federal public servants from across Canada who meet the program’s requirements to develop their career potential in a stimulating environment.

The program encourages career self-management. It allows participants to build on their existing skills, learning and knowledge, while working in a central agency involved in the highest levels of government operations and policy development. Through exposure to the government agenda, challenges and decision making mechanisms, participants have a unique opportunity for career advancement.

After working at PCO, participants are in a better position to compete for positions involving increased responsibilities and a higher level of expertise, either within their home organization or elsewhere in the public service.

1.4 Intended Beneficiaries

The key beneficiaries of this program are the participants and by extension participating departments and agencies and PCO host managers. For participants the program offers valuable, career-enhancing exposure to a broad view of government, decision making mechanisms, and central agency activities.

COTM is targeted at talented, high-potential members of three employment equity designated groups being Aboriginal peoples, persons with disabilities and visible minorities who are already at the EX minus one, EX minus two, or EX equivalent levels, and who are indeterminate employees of departments and agencies for which Treasury Board is the employer.

For participating departments and agencies, the program is intended to contribute to achievement of the public service’s objective of providing career development opportunities to members of designated groups. It contributes to building senior management capacity for the future and to knowledge transfer as participants take back to their home organizations what they have learned at PCO.

For PCO host managers, COTM is intended to foster a diverse and inclusive work environment and organizational culture, contribute to good public policy by promoting a diversity of views, and contribute to employment equity objectives within PCO.

2. Evaluation Methodology

2.1 Scope and Objective

This evaluation covers a three and a half year period from the first intake of the program in August 2005 to March 2009, and examines the extent to which the program has achieved its outcomes over this time. The objective was to provide evidence‑based information on the program’s relevance, performance, and cost effectiveness, in line with the Treasury Board Evaluation Policy.

The commitment to evaluate this program was identified in the 2005 Employment Equity Fund: Application for Funding submitted to PSHRMAC, as well as in the 2008-2009 Report on Plans and Priorities (RPP) for PCO.2

The evaluation was conducted by the Audit and Evaluation Division of PCO with the assistance of professional evaluators from the Government Consulting Services Division of Public Works and Government Services Canada who contributed to the data collection, analysis, and writing of this report.

2.2 Logic Model for Career on the Move

During the planning phase of this project, a logic model was developed to guide the evaluation. The logic model provides a visual road map describing the sequence of related events connecting the program activities and outputs with desired results. This section provides a description of the logic model including supplementary information and key assumptions that are critical to effective, efficient, and economical achievement of program outcomes. It is followed by a graphical representation of the program logic model.

As an ongoing program it is somewhat arbitrary to choose a starting point for the logic model. However, as COTM generally follows an annual cyclical pattern, for simplicity we choose the point in time when PCO managers identify suitable secondment opportunities within their organizations that could be filled by program participants. Having the right secondment opportunities that offer participants exposure to the highest levels of government operations and policy development are crucial to the program’s success.

Key Assumption 1
A suitable number of secondment opportunities exist within PCO that could be filled by program participants that will enable them to gain exposure to high level government operations and policy development.

With secondment opportunities having been identified, PCO, through a call letter from the Clerk, sends a request to departments and agencies to identify and put forward potential program candidates. The COTM manager reviews candidate applications to ensure they meet the program qualification requirements and attempts to match candidates to potential secondment opportunities. Candidates then proceed through the selection process, which can involve written tests and/or interviews, ultimately resulting in placement of program participants and signing of secondment agreements between PCO and the participants’ home departments or agencies.

Key Assumption 2
Suitable candidates meeting program criteria and possessing required competencies and experience to perform the job requirements exist and these individuals want to work at PCO under the terms and conditions of the program.

To this point in the process, all efforts have been focused on bringing participants into the program. However, an added short-term outcome of positive attention being brought to PCO for demonstrating leadership for employment equity may also be realized.

Once the most suitable candidate has been selected attention shifts from participant selection to participant development. The PCO host manager works with the program participant to develop a plan that articulates the participant’s goals and objectives and how they are going to achieve them. Planning covers on-the-job as well as formal training objectives, including limited language training.

The exposure and experience participants have while on secondment at PCO along with mentorship opportunities and formal training leads to development of skills and abilities that will ultimately enable participants to be better prepared to present themselves for opportunities for career advancement.

Key Assumption 3
Working at PCO provides participants with exposure and experiences that are not readily accessible in their home departments and agencies thereby enabling participants to develop their skills and abilities more effectively and efficiently than could be done in their home organizations.

From the PCO host manager’s perspective, hosting a participant may involve additional effort to guide and support the individual while they are in the program; however, during their time at PCO the participant is expected to be an active member of the host organization thereby contributing to achievement of the unit’s operational objectives. Policy advice developed by the organizational unit may also be enhanced by having been prepared by analysts with a diversity of points of view.

The formal portion of the program concludes with the participant’s transition back to their home department or agency. Prior to the end of the secondment PCO is to communicate with the home organization to plan for the participant’s return. This step is important so that the participant is given the opportunity to use their newly developed skills and abilities to benefit their home organization. The department or agency is receiving back a more highly skilled and experienced individual, which should be reflected in their work assignment.

Key Assumption 4
After leaving the program, participants return to challenging assignments that enable them to use and further enhance skills and abilities developed while they were directly involved in the program.

An added benefit to the home organization is an employee who possesses a better understanding of how PCO operates; information that the participant can share with his or her colleagues upon return.

Ultimately, the program is designed to contribute to employment equity objectives and executive succession. Providing members from certain designated groups who have shown potential to move to the executive level with experience and opportunity will help prepare them to become the leaders of tomorrow.

Logic Model for the Career on the Move Development Program


Text version

2.3 Evaluation Questions

The following research questions were addressed in the evaluation:

Relevance:

  1. Does the COTM program continue to be consistent with PCO's strategic objectives and federal government priorities?
  2. Is there a continued need for the COTM program?

Performance:

  1. To what extent have secondment positions been filled with suitable candidates?
  2. How has the COTM program contributed to the achievement of operational objectives at PCO?
  3. To what degree have the graduates developed skills and abilities associated with leadership?
  4. To what extent are graduates better prepared to present themselves for senior level and/or promotional opportunities upon completion of the COTM program?
  5. To what extent have graduates been provided with opportunities to use their newly developed skills and abilities in their home organization or next assignment?
  6. To what extent has there been knowledge transfer from PCO to departments and agencies?

Cost-Effectiveness:

  1. Is the COTM program being delivered efficiently compared to alternative design and delivery approaches?

The indicators used for each of these evaluation questions are shown in the evaluation matrix in Appendix A.

2.4 Methodology

Prior to conducting the evaluation study, an evaluation framework was developed that included a program profile and logic model as well as the evaluation questions and associated indicators and data sources. The Director of Audit and Evaluation Division for PCO provided management with an opportunity to comment on the methodology prior to commencement of the evaluation study. Agreement on the methodology was reached between Audit and Evaluation Division and the Assistant Deputy Minister, Corporate Services Branch in January 2009.

The following three lines of inquiry were used for this evaluation:

Document Review

The evaluation team reviewed documents and administrative data. This included documentation about the program prepared by program management, e.g. progress reports, as well as administrative data such as participant records and budget forecasts. The document review also included broad based documents that were not specific to the COTM program, including annual reports from the Clerk to the Prime Minister on the public service and Canada Public Service Agency (CPSA) reports to Parliament on employment equity in the public service.

Interviews

A total of 24 stakeholders were interviewed, either through individual or group interviews, including PCO host organization managers; participants’ home organization managers; COTM program personnel; potential PCO host organization managers, i.e. PCO managers who could host but who have not hosted COTM participants (herein referred to as “non-host” managers); and senior officials at PCO. The number of interviewees per group is outlined in the table below:

Interview Category Number of Interviewees
PCO Host Organization Managers 10
Participants’ Home Organization Managers 5
COTM Program Personnel 3
PCO Non-host Managers 4
PCO Senior Officials 2
Total Number of Interviewees 24

Where the number of interviewees responding to a question was less than five, the results are reported as the actual number of interviewees out of the total (e.g. 3 of 5). Otherwise, the following terminology is used in this report for reporting interview results:

Terminology Percent of Interviewees
None 0%
A few 1-24%
Some 25-49%
Many 50-74%
Most 75-99%
All 100%

Web-based Survey of Participants

A web-based survey of participants was developed by the evaluation team and administered in both official languages. In advance of the survey’s release, COTM program management prepared an email notification that was sent to all current participants as well as those who have completed the program requesting their participation in the survey. The survey was posted for two weeks starting February 17, 2009 and ending March 3, 2009. One reminder email was sent approximately one week after the start of the survey period and finally, one reminder phone call was made to outstanding respondents to encourage participation. Survey responses were tabulated and the data were analyzed according to the performance indicators and evaluation questions identified in the evaluation framework.

Twenty-four (24) surveys were received out of 28 potential responses3 (86 percent response rate), with 19 different federal organizations being represented. The breakdown of responses received by year the participant completed their secondment is as follows:

Year Completed Number of Respondents
2006 3
2007 8
2008 7
2009 0
Not yet completed 6
Total 24

2.5 Limitations of Methodology

As with any evaluation, there are limitations to its methodologies, which are summarized below. Note that generally, the use of multiple lines of evidence helps to minimize the limitations of an evaluation.

Representativeness of data collected

Since it was not possible to carry out an exhaustive set of interviews within the project timelines, the representativeness of the data collected is necessarily dependent on the characteristics (e.g. location, availability, etc.) of those who agreed to participate. Understanding this limitation, every effort was made to select interviewees that represented all stakeholder groups involved in the COTM program. As well, gaining input from Deputy Ministers who have participated in the COTM program nomination process would have enriched the interview data.

Following this, the population of those who have participated in the program is quite small (N=28); therefore, though 86 percent of participants completed the survey, in absolute terms, the survey findings are based on a limited number of responses (n=24).

Administrative Data

Limitations of administrative data primarily revolved around the availability of data that could be used to demonstrate program performance. In order to confidently demonstrate performance, the following administrative data would have been required:

  • Participant exit/feedback interviews;
  • Participant assessment forms to compare against objectives;
  • Yearly survey/follow-up data on promotions and successes of participants.

However, this administrative data was not available for the evaluation, and as such the evaluation has had to rely to a large extent on stakeholder perceptions, which are qualitative in nature, and may be subjective.

3. Findings

3.1 Relevance

This section describes the findings for the evaluation issue of “relevance” which focuses on the program’s consistency with departmental and government-wide priorities. Further under the issue of relevance the evaluation sought evidence to determine if the program addresses a continuing need.

3.1.1 Consistency of Program Objectives with Federal Government and PCO Priorities

Finding: The COTM objective to provide development opportunities for members of designated employment equity groups continues to be consistent with federal government priorities.

Launched in 2005 by the then Clerk of the Privy Council, Mr. Alex Himelfarb, COTM was developed to help achieve the government priority of improving the representation of the three designated group within the senior levels of the federal public service – Aboriginals, members of visible minorities and persons with disabilities. This national program was initiated to contribute to these employment equity objectives by providing career-enriching one- to two-year secondment opportunities at PCO to members of the designated groups who have demonstrated the potential to become executives in the public service.

The importance of having a representative public service workforce continues to be a priority of the Clerk of the Privy Council and Head of the Public Service and has been a consistent message in the Clerk’s last three annual reports to the Prime Minister on the public service. In the Fourteenth Annual Report to the Prime Minister, for the year ending March 31, 2007, the Clerk expressed the importance of working toward the equitable representation across the federal public service in accordance with the requirements of the Employment Equity Act. The following year, in the Fifteenth Annual Report to the Prime Minister, the Clerk reiterated the need for a public service workforce that is more broadly representative of the Canadian population.

The Clerk’s most recent report, the Sixteenth Annual Report to the Prime Minister on the Public Service of Canada, continues to identify the need to ensure that Canada is served by a public service representative of the population. The Clerk notes that while significant progress has been made in increasing the representation of members of all four employment equity target groups: women, Aboriginal peoples, persons with disabilities and members of visible minority groups there is more to be done, especially in terms of ensuring adequate representation of Canada’s diversity at executive levels of the public service.

Finding: Though the objectives of COTM are consistent with government priorities, the program’s placement within PCO is inconsistent with the renewed governance structure for human resources (HR) management in the government of Canada.

In its February 2008 report, the Prime Minister’s Advisory Committee on the Public Service found the existing HR governance structure to be “overly complex, with multiple players and a resulting burden of duplicative and often unnecessary rules. This complexity slows down every internal process and prevents efficient Human Resources operations across the Public Service4.” The Clerk, in his Fifteenth Annual Report to the Prime Minister agreed with the committee’s diagnosis and recommended steps to guide a restructuring of HR governance. In addition to reaffirming that Deputy Ministers have primary responsibility and accountability for managing their employees, the Clerk stated “central human resource agencies should only undertake those roles that must be carried out corporately, and the overlaps and unclear accountabilities among these agencies need to be sorted out5.”

In February 2009, the Prime Minister announced changes to streamline and improve the management of human resources in the Public Service of Canada. Effective March 2, 2009, the Office of the Chief Human Resources Officer (OCHRO) was created to consolidate CPSA and those parts of the Treasury Board Secretariat that deal with compensation and HR issues. This change was implemented to simplify the organizational structure for human resources management, reduce overlap and duplication, and provide Deputy Ministers with the primary responsibility for managing the people in their own departments and agencies. Within this renewed HR governance structure, PCO continues to have a role to support the Head of the Public Service, ensure the development of Deputy Ministers and Governor in Council appointments, and support Public Service Renewal.

COTM, and specifically PCO’s role as program owner, extends the department’s involvement into operational areas that overlap with the newly created OCHRO resulting in duplication and ambiguous lines of accountability among PCO, the OCHRO, and Deputy Ministers.

Finding: Opinions are mixed on whether support for the program has remained unchanged or has decreased since the inception of the program.

With the success of any program highly dependent on the level of support it receives, interviewees were asked to comment as to whether they perceived senior level support for the COTM program as changing over time. About a third of interviewees commented that support for the program has not changed over time. These interviewees noted that the COTM program is being mentioned at PCO Executive meetings and managers are being encouraged to replace candidates expected to be completing the program soon. As well, support from the Clerk and Deputy Secretaries is still being observed, Deputy Ministers from other government departments are showing continued interest in the program, and the PCO COTM Program Manager is providing assistance to host managers in conducting hiring activities and sending out communications regarding the importance of the program. However, another third of the interviewees noted that the visibility of the program has decreased since its launch in 2005 with the level of enthusiasm for the program not as noticeable as when the program was launched. The final third of interviewees did not comment on level of support for the program.

The Public Service of Canada, like other employers, is facing significant pressures from the impending retirement of their staff6. In response to the aging population, new technologies, globalization and other factors, the Clerk launched the Public Service Renewalinitiativein March 2007, which encourages the development of a workforce that draws on a diversity of origins, cultures, ideas, experiences and perspectives from all regions of Canada.

In support of Public Service Renewal, annual action plans have been developed by the Deputy Minister Committee on Public Service Renewal to advance priorities of planning, recruitment, employee development, and enabling infrastructure identified by the Clerk. In the 2008-2009 action plan, the first commitment made is for each Deputy Head to update their department’s integrated business and human resources plan, including identifying progress against their 2007-2008 plan. Each updated plan is to include a strategy for the recruitment, development and advancement of visible minorities, as well as Aboriginal people and persons with disabilities, setting out how to achieve representation at all levels that reflects their workforce availability. COTM with its focus on providing an opportunity for members of certain designated groups to develop their potential as senior level public servants is consistent with this commitment.

3.1.2 Continued Need for the COTM Program

Finding: Statistics for workforce representation among employment equity groups targeted by COTM have shown improvement, but there is a need for more to be done.

The Employment Equity Act requires the public service to monitor and, where necessary, enhance the representativeness of its workforce. Workforce availability estimates assist in this process by allowing a comparison between representation in the public service workforce of the designated employment equity groups and workforce availability of these groups in the labour pool from which the public service can recruit.7

Over the fiscal years 2006–2007 and 2007–2008, two of the three designated groups targeted by COTM were well represented in the public service when compared with workforce availability estimates being 2.5 percent for Aboriginals and 3.6 percent for persons with disabilities. Aboriginal representation increased from 4.2 percent in March 2005, to 4.4 percent by the end of 2007–2008. Representation of Aboriginal executives remained constant at 3.4 percent for both 2006–2007 and 2007–2008, slightly below the workforce availability estimate of 3.5 percent for that category. Statistics for persons with disabilities show that this group continues to be represented at levels above workforce availability estimates. The representation of persons with disabilities in the public service was 5.9 percent at the end of 2007-2008. Specifically in the executive category, this group increased from 5.5 percent in 2005–2006 to 5.8 percent in 2006–2007; it then decreased to 5.7 percent in 2007–2008.

The representation of employees from visible minority groups in the public service increased to 9.2 percent by the end of 2007–2008. Although representation continues to climb, its level remains well below a workforce availability estimate of 10.4 percent. Representation of visible minorities in the executive category was at 6.7 percent in 2007–2008, up from 6.2 percent in 2006–2007.

These statistics show some success has been achieved, but as noted by the Clerk in his latest annual report to the Prime Minister, there is more to be done, especially in terms of ensuring adequate representation of Canada’s diversity at executive levels of the public service.

Finding: All COTM program stakeholders identified an ongoing need for the program. Managers saw the program as addressing employment equity objectives, and participants viewed COTM as a way to gain the necessary central agency experience in order to build on their leadership skills and abilities.

Feedback from interviews and the participant survey indicates that the COTM program is serving a particular need. Results from interviews indicate:

  • Most interviewees noted that the COTM program contributes to an ongoing need, with half of these interviewees commenting specifically on the need to address government-wide employment equity objectives and increasing opportunities for members of designated groups;
  • Some interviewees noted that the COTM program provides members of the public service with central agency experience including a better understanding of portfolio management and an understanding of a key central agency in government;
  • Some interviewees saw the COTM program as being needed in order to broaden the experience of the designated groups to be considered for senior level positions;
  • A few of the interviewees commented that the COTM program contributes to the renewal of the public service, and enhances the government’s contribution to succession planning and career development.

These findings were consistent with the participant survey responses, with 84 percent of respondents indicating there is a benefit to having the COTM program at PCO. Analysis of the open-ended responses of this group indicated that:

  • Most survey respondents commented that COTM offers valuable central agency experience, particularly in terms of helping the participants better understand government programming and delivery and how Cabinet committees work;
  • For about a third of respondents, developing networks with PCO colleagues and high profile public servants was seen to be a benefit;
  • COTM was viewed to be helpful for participants to better understand the roles and responsibilities of departments and central agencies when working together as well as the importance of learning about policy development;
  • Lastly, other comments mentioned that COTM provided participants with a sense of personal achievement including confidence built through working on high profile and time sensitive files; that COTM broadens opportunities; and that COTM provides exposure to training opportunities that were not available in their home organizations.

As evidenced in the survey, there was general agreement among participants that COTM is a valuable program that should be continued at PCO, with 76 percent of respondents agreeing.

3.2 Performance

This section describes the findings for the evaluation issue of performance which focuses on the extent to which the program has achieved expected results. There were six evaluation questions, as identified in Section 2.3 of this report, answered for the issue of performance.

3.2.1 Suitability of Candidates

Finding: The number of COTM placements has fallen off severely over the life of the program. The primary reason for the decline is that potential PCO host managers do not perceive COTM candidates to have the necessary skills and abilities to meet business requirements.

Success of the COTM program is highly dependent on PCO managers making available a sufficient number of secondment positions for each intake. COTM program management, as well as senior management within PCO, encourages the identification of positions. While program management expected to have at least 12 COTM secondment placements per year, this level was only achieved in the first intake of participants that occurred in 2005-2006. Over the three program intakes that have occurred to date, the number of new participants has decreased from 18 in the first intake, to 8 in the second intake (2007), to only 4 in the latest intake (2008). The evaluation explored the potential reasons for this decline.

Based on their own experiences and perceptions, interviewees were asked to identify reasons for the decline in the number of COTM program positions being offered by PCO managers. Each of the following reasons was identified by at least a few interviewees:

  • Candidates have been put forth that were not good “fits” to the organization, and managers are not willing to risk having this situation happen again;
  • Based on the fast-pace of the PCO environment, hiring managers do not see themselves as having time to invest in COTM program participants;
  • COTM is in competition with other PCO and other government development programs and there is confusion about where COTM fits vis-à-vis these others;
  • Some PCO managers are making the choice to use other programs/hiring methods they are more familiar with and that have proven successful in the past;
  • There has been a decrease in commitment from senior levels within PCO; and
  • There are negative views about the program within the PCO management cadre.

Finding: There have been a sufficient number of candidates for PCO managers to consider for the COTM program.

In addition to identifying secondment opportunities within PCO, the success of the program depends on sufficient interest among potential candidates to come to PCO as COTM participants. In order to encourage participation, the Clerk sends out invitation letters to Deputy Ministers notifying them of the COTM program. The COTM program manager then sends information kits to the Deputy Heads as well as to the HR offices in these departments and agencies in order to provide information on the nomination process.

An analysis of the data provided for the evaluation revealed that there has been a consistent “supply” of potential candidates available for PCO managers to consider. A total of 157 applications were received in the first intake, with approximately 60 applications being received in each of the second and third intakes. The COTM program manager has noted that the dramatic decrease in applications can be partially explained by the departments and agencies being provided more time during the second and third intakes to conduct a more thorough screening process within their organizations. Candidates remain on the “pool” list for the entire year in which they apply, with many applicants choosing to reapply in subsequent years.

Finding: COTM program managers have had moderate success with the performance of COTM participants. While there is some concern about the thoroughness of the screening process, COTM will continue to be considered as a hiring option.

Guidance is provided to departments and agencies to help them identify and assess potential candidates to ensure that appropriate individuals are nominated for the COTM program. This guidance is included in a “General Information Kit” prepared by COTM program management for HR professionals. Eligibility criteria, as described in the information kit, state that a potential participant must:

  • Have self-identified as an Aboriginal person, a person with a disability and/or a member of a visible minority community;
  • Be an indeterminate employee occupying an executive feeder group position at the EX minus one or EX minus two level, or be at the EX equivalent level;
  • Have been identified by their department/agency as talented, promising employees with potential for advancement to the senior executive level;
  • Generally demonstrate competencies based on the Key Leadership Competencies for the Public Service8;
  • Have or be able to obtain and maintain a valid security clearance at Level II (Secret) or higher; and
  • Be able to cope effectively with a highly demanding working environment that may often involve long hours and pressure to perform at a high level with very short and sometimes concurrent deadlines9.

The information kit also provides several possible approaches to decide whether to nominate someone, including:

  • The establishment of an ad hoc committee consisting of managers at the EX level to review the applications;
  • Use of the services of an existing performance review committee/board;
  • Use of the services of the Deputy Head’s management committee/board;
  • Establishment of an ad-hoc committee consisting of senior HR personnel at the EX level; and
  • Use of any other mechanism or approach that suits the organization, as long as it identifies and assesses applicants with the potential to become senior executives10.

One line of inquiry in the evaluation was whether those who were placed in secondment positions were considered good “fits” to the position. Six participants were commented on by interviewees, with four being described as a “good fit” to their position. Interviewees noted that participants who did not fit well with the position required more guidance than other employees, and that the PCO host manager did not have the time to commit to these participants.

PCO host managers noted two main challenges. First, the managers noted that there is a concern with the quality of screening of COTM candidates. The interviewees commented that while the candidates looked like good matches on paper, once they came into the position it was evident that they were not as competent as the PCO host manager had anticipated. Since the host managers had seen applicants get nominated and accepted into the program that did not have the appropriate skill set to succeed at PCO, there is a question as to whether nominating departments and agencies understand the expectations of the COTM program. Eligibility criteria, as they are stated, are not sufficiently linked to skills and abilities that would predict a participant’s success as COTM participants at PCO. The second challenge noted by managers was that during the time they took to review CVs and make a choice on whom to interview, some applicants had already accepted other positions and were no longer available.

Regardless of the challenges with identifying the best candidates for positions within PCO, it appears that the COTM program continues to be considered as a hiring option. Many of the PCO host managers noted that they would hire through the COTM program again. However, those who stated that they would not consider hiring through the COTM program again indicated that they had had a bad experience with a participant, e.g. they required more guidance than other employees hired through other processes, or they left the secondment early.

Three of four PCO non-host managers interviewed would also consider the COTM program as a hiring option in the future. However, two of these identified a need to ensure that the CVs received for review by the hiring managers demonstrate that the candidate has the appropriate skill sets for the position being filled. They noted that while they had reviewed several candidates’ CVs provided to them, very few of those candidates actually possessed the skill set that is required for someone to be successful in their secretariats. As well, one manager noted that participation was contingent on whether notification of the program intakes could be delivered in time to incorporate the hiring of a COTM participant within the regular yearly planning cycle.

3.2.2 Achievement of PCO's Operational Objectives

Finding: Perceptions of the extent of participants’ contribution to operational objectives were generally agreed upon by both managers and participants. However, participants viewed themselves as having exceeded expectations more often than was noted by managers.

The evaluation explored the extent to which the performance of participants matched or exceeded performance objectives. Three out of the four COTM participants commented on by PCO host managers were perceived as matching the performance objectives expected of them and one was perceived as not meeting the objectives. Interviewees were also asked to comment on the extent to which they believe that participants contributed to the achievement of PCO’s operational objectives. Interviewees commented on a total of 12 participants, with 8 of these participants being seen as contributing to operational objectives at a rate equal to or better than other new employees at the same level. Four COTM participants were seen as contributing less than other employees at the same level. This was in line with data from the survey; with 75 percent of participants indicating that they believed that their performance on the program matched or exceeded the objectives set out at the beginning of their secondment; 35 percent of participants viewed themselves as having exceeded expectations. Comments from participants indicated they:

  • Contributed to complex and high-profile files;
  • Unexpectedly led government wide activities;
  • Organized cultural workshop sessions;
  • Initiated and managed task forces;
  • Chaired various committees; and
  • Were allowed to manage additional work at PCO that had not been planned.

Twenty-five percent of participants believed that their performance fell short of objectives. Reasons identified included that they did not receive proper supervision at PCO and their knowledge and skills were under-utilized while in the program.

3.2.3 Development of Leadership Skills and Abilities

As outlined in the COTM General Information Kit, the program is designed to provide opportunities for participants to improve their skills and abilities associated with leadership. Logically, a secondment at PCO will provide participants with experiences that are not readily accessible in their home departments and agencies, thereby enabling participants to develop their skills and abilities more effectively and efficiently than could be done in their home organizations. The evaluation explored the extent to which the opportunities at PCO, including the learning and training opportunities such as the Simulations for the Identification of Leadership at the Public Service Commission have contributed to the improvement of leadership competencies as identified by the Key Leadership Competency Profile as well as other leadership skills and abilities.

Finding: The most significant contribution of COTM to the development of participants’ skills and abilities was in providing opportunities to improve on the leadership competencies of Strategic Thinking and Engagement.

The following chart presents the survey results for the extent to which participants agreed that they were provided sufficient opportunities to develop leadership competencies during their time at PCO.

Chart 1 - Opportunity to develop competencies during the Career on the Move Program

Chart 1 - Opportunity to develop competencies during the Career on the Move Program
Text version

As indicated in Chart 1, participants most often agreed that they were provided with the opportunity to develop the competencies of strategic thinking, values and ethics, and engagement. A minority of participants saw themselves as having the opportunity to improve on the competency of management excellence.

Interviewees, when offered these four key competencies to select from, commented that strategic thinking, engagement, and to a lesser extent values and ethics, were most developed by the COTM participants. Management excellence was considered by the interviewees to be one competency that is difficult for COTM participants to improve, based on the nature of the policy work at PCO and with participants not having the opportunity to manage people or budgets. Only one sub-competency of management excellence, namely action management, was identified by interviewees as being improved upon by COTM participants.

The extent to which participants agreed that they had developed their competencies is highlighted in the following chart.

Chart 2 - Development of competencies during the Career on the Move Program

Chart 2 - Development of competencies during the Career on the Move Program
Text version

As indicated in Chart 2, strategic thinking and engagement were perceived to be the two most improved competencies, followed by values and ethics. Consistent with the previous chart, management excellence competencies were perceived to have been less developed during the COTM assignment.

Finding: Although data shows that participants did acquire other skills, abilities and knowledge, they were only moderately satisfied with the level of skills and abilities acquired during their participation in the COTM program.

In addition to the key leadership competencies, managers and participants also identified skills and abilities developed by participants during the COTM program. Interviewees identified writing and research skills as having improved for participants. Some interviewees also noted that participants’ self-confidence had improved as a result of their participation in the COTM program. These findings were supported by participants, who indicated writing, negotiation, leadership, conflict management, and time management skills as well as decision making abilities had improved as a result of their secondment at PCO. However, only 67 percent of participants agreed that they were satisfied (19 percent disagreed) with the skills acquired as a result of their participation in the program.

A few knowledge areas were also identified as having improved because of participation in the COTM program. Interviewees mentioned increased knowledge of the machinery of government, such as how Cabinet works, policy development processes, and secretariat specific knowledge. Participants’ comments in the survey indicated that their knowledge had increased in the areas of PCO roles and responsibilities, techniques on dealing with organizations on horizontal issues, and how international affairs impact the Canadian economy.

Finding: Participants were generally aware of the training and learning requirements of the COTM program. However, just over half of participants agreed that their PCO host manager worked with them toward the achievement of learning and training objectives during their time at PCO.

The COTM “Participant Guide” indicates that participants are responsible for drawing up their learning and training plans within two months of their arrival, and notes that participants can expect PCO host managers to help in setting the objectives. The guide strongly encourages COTM participants to take part in the many learning and training opportunities available at PCO. For example, as stated in the guide, PCO offers the Mentoring Program, Learning Passport Program, Official Languages Training, and the PCO In-House Training Program11. Survey results indicated that a majority (71 percent) of participants agreed that they were aware of COTM program training requirements while in their placement at PCO. As well, 75 percent of respondents agreed that they had completed a learning plan during their placement at PCO. When participants were asked whether they agreed that they and their PCO manager worked together toward the achievement of learning objectives during their time at PCO, only 58 percent agreed that this had actually occurred.

With regard to objective setting, all PCO host managers interviewed stated that they had set objectives for their participants, either informally or through the federal public service assessment process. This differed from survey responses, with only 63 percent of participants agreeing that they were provided with a clear set of objectives at the beginning of their placement. The expectation that PCO host managers assist participants with setting objectives in their learning plans was identified in the “Guide for PCO Host Managers” as well as the COTM Participant Guide.

3.2.4 Promotional Opportunities

As outlined in the 2005 Employment Equity Fund: Application for Funding, the COTM program “will provide developmental opportunities to promising designated group members that will enable them to broaden their skills and experience and help them prepare for higher level positions.” While this does not state that the success of the program is dependent on the participant acquiring an executive level position after completion of the program, in the application for funding, a long-term indicator of success for the program was identified as contributions to participants’ career advancements, identified through the percentage of participants promoted within a three-year period following the secondment. Therefore, promotion rates were explored as an indicator of success in this evaluation.

Finding: Participants were not successful in obtaining executive level positions after completing the COTM program through regular competitive processes. However, there was evidence that executive level positions had been obtained by three participants through referral or acting assignments.

According to the survey results, 59 percent of participants applied for a senior level (such as EX or equivalent) position during the two years prior to entering the COTM program. These numbers varied only slightly when compared to participants’ post COTM behaviors. A total of 53 percent responded that they had applied for a senior level position after completing the program. Of those who had applied, only 11 percent (one participant) qualified, i.e. entered into a pre-qualified pool but not yet selected, for the position.

While the survey results noted above could seem unfavourable in demonstrating long-term success of the program, a few observations need to be made. First, with the COTM program starting participant intake only in August of 2005, the earliest graduates in November 2006 are just over two years out of their secondment and have not yet reached the three-year indicator target for success. As well, the survey question asked respondents to comment specifically on the number of “applications” to executive level positions they had made before joining and after completing the program, without giving the respondent the freedom to indicate when an “appointment” to an executive level position had been attained. This limited the respondents from identifying promotions that may have been achieved through appointment or acting positions. To elaborate on that point, participants did have the opportunity to comment in an open-ended fashion about their promotion experiences, and it was discovered that at least for the three participants who offered this information, they had acquired executive level positions since completing the program. Two survey respondents noted that they had obtained executive level positions after completion of the program, and one noted having been asked to act in an executive level position.

Overall 57 percent of survey respondents agreed that the program met their expectations (29 percent disagreed) and a total of 71 percent agreed that they would recommend this program to others (14 percent disagreed). Comments from the survey indicated that some participants were expecting to be given promotions immediately upon returning to their home organizations. This clearly did not happen.

The program information kit does state that COTM is not a management trainee program and it makes no promise of promotion at the end of the program. However, it also states that “the experience gained at PCO could significantly help an individual progress within the executive category” and that “…participants have a unique opportunity for career advancement.” The lack of advancement among COTM participants into the executive ranks could explain why almost half believe the program did not meet their expectations.

3.2.5 Participants’ Use of Skills and Abilities in their Home Organizations

The COTM General Information Kit states that it is the responsibility of participating departments and agencies to develop the participant’s career plans in the context of the organization’s human resources plan three months before the end of the secondment. The information kit also states that home department managers are to hold discussions with the returning participant regarding career development plans and re-entry, in consultation with the PCO host manager and the home organization’s HR advisors. For example, these discussions could highlight strategic projects for the participant to take on upon their return to the home organization. As well, Deputy Heads at the home organizations receive letters from the Clerk notifying them of their responsibility to reintegrate participants approximately two months prior to participants completing their assignments. The evaluation explored to what extent these reintegration activities were occurring and their results.

Finding: The process to transition a participant from their PCO secondment back to their home organization or next assignment is a crucial step in ensuring that the participants’ new skills and abilities are utilized. There is some indication that actions to facilitate the participants’ transition out of the program are taking place; however, participants’ experiences indicate that these actions, when taken, are largely ineffective.

While program documentation outlines the roles and responsibilities of the home organization in the reintegration of the participant upon completion of the program, only 18 percent of survey respondents agreed that they were provided with opportunities to use their newly acquired skills and abilities when they returned to their home organization or next assignment (50 percent disagreed). As well, 73 percent of participants disagreed that their home organization developed a plan for their return. These participants commented that their home organization made no effort to plan for their return and seemed disinterested in the skills and abilities they brought back to the organization. Only five percent (or one person) agreed that a plan had been developed for their return. This person noted that the letter from the Clerk to their Deputy Minister resulted in a reintegration plan that implicated senior executives from their department.

Interview findings somewhat disagree with the survey results. Two of the five home organization managers interviewed received the participant back into their organization after the secondment. One of these managers stated that planning for reintegration started approximately six weeks prior to the end of the assignment, specifically noting a 1.5 hour meeting with the candidate as well as the exchange of emails to plan for the participant’s return. Discussions focused on the types of projects and files that would be best suited for the returning participant. The second manager remembered trying to contact both the PCO host manager as well as COTM program personnel to find out how to reintegrate the participant but that there was no response from PCO. Regardless, this home organization manager worked with the returning participant to ensure there were opportunities given to work on more strategic files.

The other three home organization managers did not manage the participant after the secondment. Two of these managers did comment that they met informally with the participant throughout the secondment. As an example, one of these managers mentioned meeting with the participant two or three times during the assignment to discuss progress. This manager also recalled that efforts to highlight the participant’s experience through their management committee meetings led to two or three offers of interest to the participant from other directors in the home organization.

Similar to regular secondments in the federal government, COTM participants have the option of returning to the home organization after the completion of their assignments or moving to opportunities elsewhere in the public service. Considering that 72 percent of the COTM participants returned to their home organization upon completion of the program, the focus on the role that home organizations play in the reintegration of COTM participants is warranted. Participants noted several reasons why they chose to return to their home organization, including: appointment to a senior level position; challenge balancing work and home-life at PCO; response to the perceived COTM program expectation to return to the home organization; being offered a complex project which was exciting; and desire to apply new skills and to make a difference in their home organization. Reasons for not returning to the home organization included: did not get an offer or incentive to return to the home organization; was encouraged to find a different opportunity; and received another offer for promotion within another department while in the COTM program.

Finding: Communication between the PCO host and home organization managers does not occur in the majority of cases, but was identified as a way of helping home organizations to better plan for the participant’s return.

According to COTM program documentation; there are very few program expectations concerning ongoing communication between the PCO host manager and the home organization manager during the participant’s assignment at PCO. However, as stated in the Guide for PCO Host Managers, “In the last three months of the assignment, the PCO host manager should contact the home manager to discuss the career development plans for the participant once he/she returns to the home department.” Despite this expectation, according to most of the PCO host managers who commented as well as all of the home organization managers, no communication occurred between PCO host managers and home organization managers before, during, or after the participant’s COTM placement. One PCO manager, who is currently hosting a participant, does intend to make contact with the home organization manager prior to the participant completing the program. As well, one home organization manager recalled trying to contact the PCO host manager but was unsuccessful.

Consistent findings were observed in the survey results, with only 27 percent of respondents agreeing that there was communication between PCO and their home organization during their placement at PCO, and no respondents agreeing that the communication that did occur was sufficient. Just over half of those who commented in the interviews noted that they did not have any expectations with regards to communication between the home organization and the host organization (the majority of these being home organization managers with only one of the ten PCO host managers making the same comment). This finding demonstrates that there is a lack of alignment between the reality and the expectation for PCO host managers and their communications with the home organizations.

All five home organization managers noted that establishing communication with PCO host managers would have its benefits. From the perspective of three of five home organization managers, maintaining contact with the PCO host manager would enable the home organization to gain insight into what is required of them to better reintegrate the participant upon completion of the program. Specifically, communication would provide the home organization with a better understanding of the participant’s strengths and weaknesses and of what they have learned at PCO in order to better place them upon their return to the organization. In fact, two home organization managers commented that while they knew their staff member was not happy in their position at PCO, communication was seen as a potential way that home organizations could have worked jointly with PCO to ensure maximum skill development on behalf of the candidate.

3.2.6 Knowledge Transfer from PCO to Departments and Agencies

Finding: Home organizations see value in continuing to have staff from their organizations participate in the program and have the opportunity to learn from their work experience at PCO. Benefits identified include exposure to the workings of PCO as a central agency and the building of management capacity within the federal government. Most home organization managers interviewed would encourage their staff to participate in the COTM program in the future.

An added benefit, and long-term outcome, for the COTM program is to contribute to the transfer of knowledge from PCO to departments and agencies. The evaluation explored the achievement of this outcome by inviting comments from home organization managers on what they saw as the value of having staff from their organization participate in the program. In addition to four of five of these interviewees noting that they would continue to encourage staff to participate in the program, all five home organization managers interviewed noted that it was of benefit to have their staff take part in the program. The main benefits were identified as gaining central agency experience thereby developing their policy horizons, and helping staff to better understand the complexities of inter-departmental consultations. Another benefit was identified as building management capacity in the federal government through providing a developmental opportunity to members from designated groups during the formative years in their careers. As well, one interviewee suggested that the government as a whole is losing opportunities to train employees to serve in a policy capacity in the government, and with the COTM program providing this, there is value in having the program.

3.3 Cost Effectiveness/Alternatives

This section describes the findings for the evaluation issue of cost-effectiveness including analysis of alternatives that could be employed by PCO to achieve the program’s intended outcomes. Specifically the evaluation collected and analyzed data to answer the question: “Is the COTM program being delivered efficiently compared to alternative design and delivery approaches?”

External Funding

When the COTM program was initiated in 2005, PCO received external funding from the Employment Equity Fund administered by CPSA12. During the program’s first year of operation the Employment Equity Fund paid costs including participant travel and accommodation, professional development, language training and a portion of administrative support costs; while PCO was responsible for participant salaries and remaining program support salaries not covered by the Fund. In 2006, however, funding through the Employment Equity Fund ended and since that time PCO has paid the full cost of the program without external funding sources. COTM has no provisions for cost sharing arrangements with participants’ home departments or agencies.

Participant Salary Costs

Participant salaries are the greatest cost attributable to the program, representing 70.7 percent of total program expenditures. Since program inception, the cost of salaries13, including Employee Benefits Plan (EBP), for COTM participants has been just over $3 million, which equates to about $94,000 per participant on an annual basis. For purposes of cost-effectiveness analysis however, participant salary costs are not particularly meaningful. Participant salaries do not represent an incremental cost to PCO relative to bringing in someone at the same level to do the same job by alternative means, i.e. regular secondment or deployment process. Therefore, participant salary costs will only be considered in comparison with other development programs or alternative approaches.

Finding: Overall COTM administrative support and operating costs are reasonable with exception to travel and accommodation costs for a limited number of regional participants. At an average annual cost of over $76,000, the cost to PCO to host a regional participant in the program is almost three times that for a locally based participant; the difference is fully attributable to travel and accommodation costs.

Excluding participant salary expenses, it has cost about $39,000 to host one COTM participant for one year in the program, 25 percent of which represents program support salaries and expenses. The balance of the yearly cost per participant is made up of travel and accommodation (35.5 percent); informatic technology services, e.g. database development (13.9 percent); professional services (11.0 percent); professional development and language training (9.3 percent); information services, e.g. writing and printing services (5.0 percent); and other costs (0.4 percent).

Table 1 - Breakdown of Program Support Salary and Operating Costs
Category Total Cost Annualized Cost Per Participant
Program Support Salary Costs
(including EBP)
$ 314,327 $ 9,672
Operating Costs
Travel and Accommodation $ 447,842 $ 13,780
Training $ 117,538 $ 3,617
Informatic Technology Services $ 174,788 $ 5,378
Professional Services $ 138,480 $ 4,261
Information Services $ 63,532 $ 1,955
Other $ 4,835 $ 149
Total Operating Costs $ 947,015 $ 29,139
Combined and Program Support Salary and Operating Costs $ 1,261,343 $ 38,811

As shown in Table 1, travel and accommodation, at a total cost of $447,842 or $13,780 per participant on an annualized basis, is by far the single greatest program operating expense. Most of the travel and accommodation expense ($413,192) is directly attributable to a limited number of regional participants who come to Ottawa to work at PCO. While enrolled in the program, these participants are classified as being on travel status and are paid travel and accommodation expenses, including meals and incidentals, weekend travel home (on average every three weeks), and local accommodation, in accordance with the Treasury Board Secretariat Travel Directive. By allocating this $413,192 to only those participants who actually incurred these expenses, the annualized cost per participant can be more precisely stated as $26,097 for a locally based participant and $76,486 for regional participants, i.e. those on travel status. Table 2 presents the breakdown of costs per participant per year for local versus regional participants.

Table 2 - Segmented Analysis - Local versus Regional Participants
Category Total Cost Annualized Cost per Participant
Local (National Capital Area) participants
Program support salary costs including EBP $ 314,327 $ 9,672
Operating costs – excluding travel and accommodation directly attributable to nine regional participants $ 533,824 $ 16,425
Combined program support salary and operating costs for non-regional participants $ 848,151 $ 26,097
Regional participants
Add: Travel expenses directly attributable to regional participants $413,192 $50,389
Combined program support salary and operating costs for regional participants $1,261,343 $76,486

As an alternative approach, the annualized cost per participant could also be calculated with only ongoing operating costs. When the program began there were costs incurred during the first and second years of operation related to initial program set-up and should not be re-incurred at similar levels in the future. These could be considered sunk costs because they do not represent ongoing operating costs and therefore are not relevant to future decision making. Excluding the informatic services and information services costs of $238,320 combined, both of which were not incurred beyond fiscal year 2006-2007, the average annualized cost per participant is $31,478; or about $7,500 less than the $38,811 that was calculated when all operating costs were considered. Segmented between local and regional participants as was done in Table 2, the annualized cost per participant, excluding informatic and information services, becomes $18,764 and $69,153 respectively.

Finding: Actual participation levels being consistently lower than expected have resulted in actual program expenses being consistently lower than planned.

Program management develops a budget in which they estimate annual COTM expenditures using key assumptions including the number of new participants that will enter the program that year, when new participants will enter the program, i.e. early in the fiscal year versus later in the year, the number of continuing participants, and the number of regional participants. During the program’s four years of existence, actual costs have consistently been less than planned costs. For the first year (2005-2006) this was due in large part to participants entering the program later in the year than had been anticipated. This had a particularly significant impact on the travel and accommodation estimate, with actual costs being one-third of what had been forecast. In subsequent years’ intakes, the number of participants entering the program has been less than anticipated. Because a key cost driver for the program’s budgets was the number of participants, the drop off in participation has led to the budget variance.

A detailed comparison of annual budgeted versus actual program costs is provided in Appendix B.

Finding: COTM is viewed as being on par with other programs in terms of its efficiency, though managers did identify up-front screening as an area that could be improved.

In terms of efficiency of the process, COTM is viewed as being on par with other development programs such as the Career Assignment Program (CAP), the Management Trainee Program (MTP), and the Accelerated Economist Development Program (AETP). These development programs, in general, make it easier for a manager to bring in a new person relative to running a full competition. However, as was noted in interviews conducted, managers at PCO often rely on their networks within government to identify and recruit the highest performing talented employees for PCO, e.g. through deployments or secondments and not full competitions, and therefore the comparison to full competitions is not entirely relevant. That said COTM is seen as being “user friendly” from the host managers’ perspective. The host managers are presented with a list of candidates from which they can select individuals to interview. One area that managers consistently identified where the efficiency of the process could be improved was with the up-front screening of candidates. With a more rigorous screening process by program management, host managers would be presented with a list of candidates that would be more closely aligned with their requirements. This issue was discussed in more depth in this report in Section 3.2.1 – Suitability of Candidates. From the participants’ perspective, two-thirds agreed that COTM was delivered efficiently, e.g. well-managed, timely responses to questions from COTM management.

Finding: Alternative programs do exist that are targeted at the same level of public servant, as is COTM; however, they are not specifically targeted at members of employment equity groups. For the programs most closely related to COTM, there would be no difference to PCO for salary costs, though operating and administrative support costs incurred by PCO corporate could be reduced.

A quick search on the PCO InfoNet reveals a listing of 15 distinct leadership development programs; interestingly the list does not include COTM. Most of these programs are administered by OCHRO or the Canada School of Public Service (CSPS) and follow a variety of curricula ranging from those focused on formal classroom training to others, like COTM, that focus more on on-the-job development. Of these other programs, the OCHRO CAP program is most closely related to COTM in terms of the program objective, target population, and methodology. Refer to Appendix C for a comparison between COTM and other selected development programs.

PCO has a history of being actively involved in the CAP program by providing central agency experience to employees from other departments at the EX minus one and EX minus two levels. In the CAP program, participants normally complete three one-year developmental assignments; the first and third in the participant’s home organization and the second in a central agency. Between 1999 and 2006, PCO hosted 36 CAP participants, 24 of whom represented one of the four employment equity groups, including 17 women, 5 members of visible minority groups, and 2 Aboriginal persons. However, since 2006 no new CAP participants have been placed at PCO; 2006 was also the first full year of the COTM program. It is plausible that some of the COTM placements were made at the expense of the CAP program as managers shifted their support for one program to another.

In line with the PCO HR priority to ensure the department is able to attract, identify and hire highly qualified individuals by developing broader pools of talent, the Operations Branch has established an initiative targeted at the analyst community. Similar to COTM, the “Analyst Initiative” targets high-performing public servants at the EX minus one and EX minus two levels, as well as those at the EX-01 level. Under the Analyst Initiative employees can come to PCO on secondment, similar to COTM, or through deployment, depending on the circumstances. The program is still evolving, but the Operations Branch is establishing a training and development regime for program participants.

Earlier in the discussion of cost effectiveness we presented participant salary figures indicating that they represent the greatest portion of total COTM program costs, but that these costs were only relevant in comparison with other similar programs. Cost models followed by other development programs are similar, but not equivalent, to COTM. In terms of treatment of salary costs, the CAP program is closest to COTM. Under both programs, the host organization (PCO) is responsible to pay participant salary costs while they are on assignment in the department.

The MTP is similar to the CAP program, however it is targeted at lower level public servants, not yet at the EX minus one or EX minus two levels. Though there have been some recent changes, PCO typically shares the salary costs for MTP participants with the home organizations while the participant is on assignment at PCO. Current MTP participant salaries are paid 50 percent by PCO and 50 percent by the home organization. Under the AETP, PCO currently does not pay participant salaries while they are on assignment, which is typically six months in duration. Like COTM, the Operations Branch Analyst Initiative does not have any provision for cost sharing of salaries with the home departments.

Next to participant salaries, travel and accommodation represents the greatest COTM cost. Other development programs include provisions for relocation allowances in certain circumstances. For example, for the CAP and MTP programs, discussed in this section, the Directive on the Administration of Leadership Development Programs - Management Trainee Program and Career Assignment Program describes the treatment of travel costs. Certain travel costs related to educational components of the MTP program are paid centrally (by OCHRO), while travel costs for all other MTP and CAP participants are the responsibility of the home organization. Costs associated with the temporary relocation or accommodation of candidates and participants recruited by OCHRO are provided by OCHRO, as are the funding levels.

Other costs such as training and development are paid fully by PCO for participants in the COTM program. In the CAP, MTP, and AETP programs these costs are not the responsibility of PCO as the host department, unless it is training specific to the participant’s assignment. Administrative costs for PCO are greater for COTM than CAP, MTP, and AETP because these other programs are administered through OCHRO.

4. Conclusions and Recommendation

After four years of operation and three separate intakes, COTM is at a crossroads. While the evaluation has identified some successes, it has also revealed weaknesses that have implications concerning the program’s relevance, performance, and cost-effectiveness. This section includes conclusions for each of these evaluation issues followed by one recommendation to management.

Relevance

The evaluation has shown that the COTM program is consistent with the government priority to improve representation of members of designated employment equity groups within the executive ranks of the public service. Achieving representation of designated employment equity groups reflective of their workforce availability has been featured prominently in the Clerk’s Public Service Renewal initiative and has been a consistent message in his annual reports to the Prime Minister on the public service. However, the placement of the COTM program at PCO is inconsistent with recent government changes to streamline the public service HR governance structure.

The Prime Minister’s Advisory Committee on the Public Service found the previous HR governance structure to be overly complex, with multiple players and a resulting burden of duplicative and often unnecessary rules. COTM is symptomatic of the problems observed by the Prime Minister’s Advisory Committee. PCO and OCHRO operate competing development programs that have similar objectives, methodologies, and target the same level of public servants. Under the renewed HR structure, as a central agency, PCO’s roles are limited to providing support to the Head of the Public Service, support for the Prime Minister’s Advisory Committee on the Public Service and the Deputy Minister Committee on Public Service Renewal, and providing talent management for deputies and Governor in Council appointments.

Performance

During its first intake of participants that occurred in 2005-2006 the COTM program was very successful in attracting and placing participants into the program with 18 placements that first year. The second and third intakes, however, experienced significant declines in the number of new participants, dropping to eight in 2007 and most recently four in 2008. There are still sufficient numbers of public servants wanting to come to PCO as program participants; however, this interest is not translating into successful placements. PCO managers have not seen the right mix of knowledge, skills, and experience in candidates put forward for their consideration. The program is focused on producing an adequate supply of candidates without sufficient consideration of the operational demands of the positions intended to be filled by COTM participants.

There is some indication that participants have been successful in developing key leadership competencies of strategic thinking and engagement, and to a lesser extent values and ethics; in-line with the key leadership competency profile used as a main component of the EX Qualification Standard. However, participation in the program has not as of yet translated into success in competitions at the executive level. Among the 86 percent of participants, past and current, who completed the evaluation survey, none have obtained an executive level position through the competitive process, though three past participants did indicate that they are currently occupying an executive level position obtained either through appointment or in an acting capacity.

The transition process at the end of the program back to the participant’s home organization or to their next assignment has not resulted in effective use of knowledge and experiences participants gained while in the program. Only 18 percent of participants indicated that they were provided an opportunity to use new skills and abilities after leaving the program. A key contributor to this finding is the lack of communication between PCO and home organization managers before, during, or after the COTM assignment.

Cost Effectiveness

Since program inception in 2005, COTM program support and operating expenses, not inclusive of participant salaries, have totaled about $1.3 million, which equates to almost $39,000 per participant on an annualized basis. Cost-effectiveness could be improved if the number of participants increased to meet expected levels as some program costs do not vary directly with the number of participants and therefore would not increase substantially if participation grew.

The travel and accommodation category of operating expenses stood out from the others due to its size relative to all other program expenses. Travel and accommodation represents 35.5 percent of all program support and operating expenses, most of which are attributable to a limited number of regional participants. Regional participants are considered to be on travel status for the one to two years they are in the program and are compensated based on the Treasury Board Secretariat Travel Directive. As a result of this practice, when segmented between regionally and locally based participants, the annualized average cost to host a regional participant is about $76,500 or almost three times the $26,000 cost for a locally based participant. Similar development programs administered by the OCHRO establish much more modest limits for relocation costs.

Recommendation

The results of the evaluation do not support the status-quo as an option moving forward. The evaluation recommends that management consider alternative approaches to achieving the objectives of the COTM program. That said, regardless of the course of action taken by management, it should be made clear that PCO continues to support and encourage advancement of visible minorities, Aboriginal people, and persons with disabilities.

Below are three options for management’s consideration, including pros and cons for each. Selection of the first option would mean continuing the program with some changes, while selection of the latter two options would imply ending the COTM program and opting for a different approach. The three options presented are not intended to be an exhaustive list, management may choose to pursue one or more of the options presented or pursue an alternative approach.

Option 1

Continue to operate the COTM program with targeted changes to improve performance and cost-effectiveness. If this option is chosen, the evaluation recommends the following five steps:

  1. Revise program eligibility criteria to be better aligned with operational requirements of positions to be filled by COTM participants.
  2. During the candidate nomination process, include a requirement for the nominating organization to submit a plan demonstrating how the organization intends to use the nominee’s new skills and abilities upon completion of the assignment at PCO.
  3. Clarify the objective of the program and communicate expectations to stakeholders.
  4. Improve rigor around collection of data to track performance of the program. Examples would include participant exit/feedback interviews, tracking how well managers’ and participants’ expectations have been met, and yearly survey/follow-up data on promotions and successes of participants.
  5. End the practice of paying travel and accommodation costs for regional participants based on the TBS Travel Directive and instead compensate regional participants for relocation costs up to a defined ceiling limit. Relocation costs above the ceiling amount would be negotiated between the home organization and the participant.

If management concentrates its efforts on fixing what is not working, the COTM program could perform to expectations. Maintaining and improving the program could also signal to the rest of the public service that the Clerk and PCO remain committed to improving opportunities for visible minorities, Aboriginal people, and persons with disabilities to develop their career potential. However, this course of action does not overcome the issues raised in the evaluation regarding duplication among COTM and other existing development programs. The COTM program is not the only means the Clerk has at his disposal to demonstrate his commitment to achieving employment equity objectives. These ideas are pursued in the second and third options presented for management’s consideration.

Option 2

End the COTM program and in its place enhance employment equity focus of the PCO Operations Branch Analyst Initiative.

One of the five priorities identified in the PCO Strategic Human Resources Plan for 2007-2010 was to ensure PCO’s ability to attract, identify and hire highly qualified individuals by developing broader pools of talent. The Analyst Initiative was created to attract high-calibre policy analysts to PCO at the EX-01, EX minus one and EX minus two levels for deployments or secondments, but it does not currently have an employment equity focus/component.

The Analyst Initiative specifically targets policy analysts with skills and experience that are complementary to PCO’s core functions thereby overcoming one of the primary reasons why there has been a decline in COTM participation. Further, consolidating the COTM program with the analyst initiative would eliminate duplication that currently exists between the two PCO programs. This option would also somewhat overcome the issue of inconsistency with the renewed HR governance structure. Because the Analyst Initiative focuses more on PCO’s HR requirement to ensure that the department is able to attract and hire highly qualified individuals, it does not overlap OCHRO programs to the same extent, as does COTM.

Moving to a single program approach that would include members of employment equity groups in a larger pool of candidates could dilute the positive impacts for these individuals. There is also the potential for misperception that the Clerk and PCO are not supporting employment equity objectives or that support has been reduced. Finally, though this option would somewhat overcome the issue of inconsistency with the renewed HR governance structure, the analyst initiative is not well aligned with the three roles identified for PCO as a central agency in the renewed HR governance structure.

Option 3

End the COTM program and in its place work with OCHRO to identify ways PCO could increase its support for development programs such as the CAP.

The CAP program objective to identify a representative group of individuals who have demonstrated executive potential and to accelerate their development and advancement is very similar to that of COTM. There are also similarities between the program designs, both are based around secondment assignments; between target populations, both are focused on public servants at the EX minus one, EX minus two, and EX equivalent levels; and both include educational components. The CAP program does not uniquely target members of employment equity groups, but participating organizations are encouraged to consider members of designated groups and preference may also be shown to participating organizations choosing to use CAP to meet employment equity targets. Of the three options presented, this one would be most in-line with renewed HR governance structure.

This option, however, could suffer some of the same problems experienced by COTM because it does not directly address PCO operational requirements. PCO managers may be provided with lists of candidates who do not possess the skills and experience required at PCO. Finally, while PCO offers great challenges and opportunities in the area of policy analysis, the department has limited opportunities for participants to develop skills and abilities associated with managing people and budgets.

5. Management Action Plan

Evaluation of the Career on the Move Development Program

The Assistant Deputy Minister of the Corporate Services Branch (CSB) and the Deputy Secretary to the Cabinet, Operations Branch are jointly responsible for the Action Plan.
Recommendation Actions to be Taken Responsibility Target Date
The results of the evaluation do not support the status-quo as an option moving forward. The evaluation recommends that management consider alternative approaches to achieving the objectives of the COTM program. That said, regardless of the course of action taken by management, it should be made clear that PCO continues to support and encourage advancement of visible minorities, Aboriginal people, and persons with disabilities. Merge the employment equity component of the COTM program with the PCO Analyst Recruitment Initiative, creating one program.

Specific activities to be conducted by Corporate Services Branch and Operations Branch are identified below.
Assistant Deputy Minister, CSB and Deputy Secretary to the Cabinet, Operations Branch are jointly responsible for the Action Plan going forward. April 2010
  Actions to be taken by Corporate Services Branch as the lead, with input from Operations Branch:
  1. Communicate the program changes to Deputy Heads, Heads of HR, OCHRO and the PSC plans for the upcoming merge of the two programs.
  2. Identify the process required to implement an employment equity component to the PCO Analyst Recruitment Initiative which will allow tracking of employment equity representation available in the inventory.






Executive Director, HR, CSB





Executive Director, HR, CSB &
Operations Branch






March 2010







Spring 2010
  Actions to be taken by Operations Branch as the lead in preparation of the annual launch of the revised PCO Analyst Recruitment Initiative:
  1. Issue the Call Letter for the 2011-2012 intake to Deputy Heads announcing a new intake of the revised Analyst Recruitment Initiative.
  2. As part of the preparatory steps prior to issuing the call for nominations, the mechanism for establishing Employment Equity tracking, which includes indicators to be able to assess representativeness and take-up will have been decided on.
  3. Screening and evaluation of candidates by a panel of senior managers, including a written assessment & group evaluation.
  4. Other activities related to the administration of the Analyst Recruitment Initiative
Deputy Secretary to the Cabinet, Operations Branch


Operations Branch






Operations Branch












Operations Branch





Operations Branch







January 2011







Spring 2010













Winter 2011






Spring/Summer 2011 and onwards

Appendix A - Evaluation Matrix

Appendix A - Evaluation Matrix

Appendix B - Budget Variance Analysis

Appendix B presents a comparison of annual budgeted program costs versus actual program costs for fiscal years 2005-2006 through 2008-2009.

  Budget Actual Variance
2005-2006
Program Administrative Support  $ 153,310  $ 119,956  $ 33,354
Travel and Accommodation  $ 233,288  $ 76,796  $ 156,492
Training and Development  $ 58,002  $ 25,189  $ 32,813
Informatic Technology Services  $ -  $ 152,828 -$ 152,828
Professional Services  $ 30,000  $ 43,942 -$ 13,942
Information Services  $ 50,000  $ 29,841  $ 20,159
Other  $ -  $ 3,057 -$ 3,057
2005-2006 Total  $ 524,600  $ 451,609  $ 72,991

2006-2007
Program Administrative Support  $ 163,819  $ 69,062  $ 94,757
Travel and Accommodation  $ 441,970  $ 184,225  $ 257,745
Training and Development  $ 79,470  $ 57,351  $ 22,119
Informatic Technology Services  $ -  $ 21,960 -$ 21,960
Professional Services  $ 30,000  $ 63,029 -$ 33,029
Information Services  $ 25,000  $ 33,691 -$ 8,691
Other  $ -  $ 1,070 -$ 1,070
2006-2007 Total  $ 740,259  $ 430,388  $ 309,871

2007-2008
Program Administrative Support  $ 81,715  $ 68,585  $ 13,130
Travel and Accommodation  $ 254,776  $ 88,868  $ 165,908
Training and Development  $ 66,200  $ 24,086  $ 42,114
Informatic Technology Services  $ -  $ -  $ -
Professional Services  $ 25,000  $ 7,847  $ 17,153
Information Services  $ 20,000  $ -  $ 20,000
Other  $ 1,000  $ 636  $ 364
2007-2008 Total  $ 448,691  $ 190,022  $ 258,669

2008-2009
Program Administrative Support  $ 79,466  $ 56,725  $ 22,741
Travel and Accommodation  $ 215,000  $ 97,953  $ 117,047
Training and Development  $ 44,500  $ 10,912  $ 33,588
Informatic Technology Services  $ -  $ -  $ -
Professional Services  $ 18,000  $ 23,663 -$ 5,663
Information Services  $ -  $ -  $ -
Other  $ 2,000  $ 72  $ 1,928
2008-2009 Total  $ 358,966  $ 189,325  $ 169,641

Appendix C - Comparison with Similar Programs

Appendix C - Comparison with Similar Programs


  1. From the Second Report of the Prime Minister’s Advisory Committee on the Public Service, p. 6, 2008.
  2. Reports on Plans and Priorities - 2008-2009 .
  3. Two of the 30 COTM participants left the program early for other reasons before completing their secondment. For the evaluation, the population was defined as all participants who completed their COTM secondments as well as current COTM participants (N=28).
  4. Human Resources Governance and Accountability Structure.
  5. Responding to the Prime Minister’s Advisory Committee.
  6. The 15th Annual Report to the Prime Minister on the Public Service of Canada, dated April 2008, indicates that almost 50% of public service executives are able to retire by 2012.
  7. Primary source for content, including statistics, in this section is the Employment Equity in the Public Service of Canada 2006–2007 and 2007–2008 report.
  8. Effective behaviours.
  9. PCO, Career on the Move 2008-2009 Information Kit – Human Resources.
  10. Ibid.
  11. Section 2 of the Participant Guide.
  12. Then known as the Public Service Human Resources Management Agency of Canada (PSHRMAC).
  13. Fiscal year 2005-2006 through 2008-2009 (to March 4, 2009).