Appendix A - Evaluation Matrix

Appendix A presents the Evaluation Matrix, which identifies the lines of inquiry that were pursued to gather information for each indicator used to answer the nine evaluation questions under the evaluation issue categories of relevance, performance, and cost-effectiveness.

Evaluation Questions Indicators Document and Administrative Data Review Interviews Group Interviews Survey
Document Review External Stakeholders Senior Officials (PCO) COTM Program (HR) Home Organization Managers Host Managers at PCO PCO Managers (Non-hosts) Past and Present Participants Comparative Analysis
Relevance
R1. Does the COTM program continue to be consistent with PCO's strategic objectives and federal government priorities? R1.1 - Trend in support for employment equity initiatives in the federal government   X   X X   X      
R2. Is there a continued need for the COTM program?   R2.1 - Perception of the continued need for the COTM program     X X X X X X X  
R2.2 - Workforce representation from employment equity designated groups in senior level positions in the federal government   X X              
R2.3 - Number of recently appointed EXs that have central agency experience X                  
Performance
P1. To what extent have secondment positions been filled with suitable candidates? P1.1 - Number of secondment opportunities identified by PCO managers, and reasons for not identifying secondment positions X           X X    
P1.2 - Trends in number of participants per year X                  
P1.3 - Number of candidates identified by Deputy Heads to participate in the COTM program X                  
P1.4 - Perceived barriers to filling positions with suitable candidates and finding the best "fit" of candidate to position          X   X      
P1.5 - Number of secondments that have ended early X                  
P2. How has the COTM program contributed to the achievement of PCOv's operational objectives? P2.1 - Extent to which the graduates contributed to the achievement of PCO's operational objectives       X     X      
P2.2 - Percentage of managers who have offered a secondment position for more than one intake X                  
P2.3 - Extent to which graduates’ performance matches or exceeds objectives set out at the beginning of secondments             X   X  
P3. To what degree have the graduates developed skills and abilities associated with leadership? P3.1 - Extent to which candidates' secondment, learning (via a learning plan) and training objectives were achieved           X X   X  
P3.2 - Comparison between EX competencies possessed by graduates prior to joining the program and competencies possessed after graduation from the program, and the extent to which any changes are attributable to the COTM program            X X   X  
P3.3 - Degree of participant satisfaction with the skills acquired through the program and whether the program met their expectations                 X  
P4. To what extent are graduates better prepared to present themselves for senior level and/or promotional opportunities upon completion of the COTM program? P4.1 - Comparison of the number of times (and success rates) a graduate presented themselves for a senior level position prior to and after completion of the program, and whether any change is attributable to participation in the COTM program                 X  
P4.2 - Number of promotions or prequalification that have been achieved by graduates after graduation                 X  
P5. To what extent have graduates been provided with opportunities to use their newly developed skills and abilities in their home organization or next assignment? P5.1 - Level of communication between PCO and the manager at the home organization or next assignment         X X X   X  
P5.2 - Extent to which the home organization develops a plan for the graduates' return (i.e. could include new opportunities, promotions and/or acting assignments) when applicable           X        
P5.3 - Perception of extent to which graduates are provided with opportunities to use their newly developed skills in their home organization or next assignment           X     X  
P6. To what extent has there been knowledge transfer from PCO to departments and agencies? P6.1 -  Perception of the value of PCO experience for the department           X        
P6.2 - Perception of supplementary benefits to the home organization related to participants’ work experience at PCO           X     X  
Cost-Effectiveness/Alternatives
C1. Is the COTM program being delivered efficiently compared to alternative design and delivery approaches? C1.1 - Cost per graduate X                  
C1.2 - Length of program process steps (e.g. marketing, selection, placement) X       X          
C1.3 - Planned-to-actual resource use (budgets vs. expenditures) X       X          
C1.4 - Perception of whether the COTM program is being delivered efficiently, and potential areas for improvement (i.e. who is delivering it, how it is being delivered, etc.)     X X X X X X X  
C1.5 - Comparison of the COTM to alternative design and delivery approaches (e.g. cost, time, etc.)       X X X X X   X