Evaluation report of CanCode

Audit and Evaluation Branch

June 2024

Table of Contents

• Program context
• Methodology
• Findings
• Summary & Recommendations
• Appendices

1.0 Program context

1.1 Program description

The emergence of CanCode

To help ensure that Canada's youth, kindergarten to grade 12 (K-12), are prepared with the knowledge and skills necessary to function in the evolving digital economy and to support initiatives that provide K-12 teachers with the training and professional development needed to introduce digital skills and coding related concepts in the classroom, the Government of Canada launched CanCode in 2017.

Formerly Coding For Kids, CanCode is a national contribution program funded by Innovation, Science and Economic Development (ISED) Canada aimed at supporting the development of coding and digital skills amongst youth and teachers.

1.2 Program funding

Between 2017-18 and 2023-24, ISED allocated funding of $190 Million to CanCode, including $50 Million in Budget 2017, $60 Million in Budget 2019, and a renewal of $80 Million over 3 years in Budget 2021.

1.3 Program delivery

ISED is responsible for providing overall administration and oversight to the CanCode program. Digital skills and coding training was delivered through not-for-profit third-party organizations funded by ISED through contribution agreements (see Appendix B for funding eligibility criteria). CanCode 3.0 recipient organizations operated in all provinces and territories and differed in size, scope, and geographical reach.

CanCode 3.0 Recipient Organizations:

Geographic Reach	Description	Number of Organizations
National Organizations provided opportunities in most (if not all) provinces and territories.	Large organizations with strong reach and experienced in providing digital learning opportunities to a broad-base of Canadians.	11
Regional Organizations focused on specific areas or regions and can cover between two to nine provinces and/or territories.	Focused on providing training opportunities to certain regions of the country. Funding regional organizations ensured that all parts of Canada had tailored training opportunities fit for their unique circumstances.	11
Local Organizations provide opportunities in one province and/or territory.	Local organizations which serve to provide highly-localized training opportunities. These organizations served provinces, territories or communities not specifically targeted by other organizations.	6
	TOTAL:	28

Number of CanCode 2.0 recipient organizations: 27
Number of CanCode 1.0 recipient organizations: 21

Recipient organizations mobilize a large network to delivery CanCode training, including:

• Volunteers
• Paid staff
• Partnerships with schools and community organizations

1.4 CanCode training delivery

CanCode activities were designed to target two groups: K-12 youth and teachers.

In CanCode 3.0, the program focused on targeting underserved youth, including girls, Indigenous youth, youth with disabilities, and youth living in rural, remote and Northern communities. Many of the organizations delivering CanCode programming offered training and personal development for teachers to support classroom learning and the integration of coding and digital skills into the teaching curriculum.

Each organization has flexibility in determining how they delivered their respective coding and digital skills training activities (the various CanCode delivery methods are listed in Figure 4). These methods were outlined in their contribution agreements with ISED. CanCode training was offered at no cost to youth and teachers.

2.0 Methodology

2.1 Evaluation context

An evaluation of ISED's funding to CanCode is required every five years under the Financial Administration Act. The objective of this evaluation was to assess the relevance, performance and efficiency of ISED funding to CanCode. The scope of the evaluation includes CanCode-funded training and activities, and the efficiency of the delivery model. Long and medium-term outcomes are out of scope given the recent launch of the program. The evaluation covered all ISED funded activities from April 1st, 2017, to Dec 31, 2023.

The evaluation was conducted by the Audit and Evaluation Branch at ISED. A results-based approach was used to examine the achievement of expected outcomes for CanCode (Appendix A). Given the phased approach in program evolution, the evaluation used data from CanCode 3.0 where data from all phases did not align.

2.2 Evaluation questions

Relevance

• To what extent is there a continued need to provide coding and digital skills training for K-12 students and teachers in Canada?

Performance

• To what extent is the program supporting the development of K-12 students digital and coding skills required to participate in the digital economy?
• To what extent is the program supporting the development of teachers' capacity to teach digital and coding skills required to participate in the digital economy?
• Does CanCode have suitable performance measurement in place to demonstrate achievement of outcomes?

Efficiency

• To what extent is the CanCode design and delivery model an efficient and effective approach for developing students' and teachers' digital knowledge and skills?
• To what extent has CanCode design and delivery considered the government-wide policy priorities?

The evaluation produced 7 findings, supported by multiple lines of evidence, and leading to 3 recommendations.

2.3 Data collection methods

The following lines of evidence were used (details in Appendix B):

• Literature and document review
• Performance, administrative and financial data review
• Case studies
• Online survey
• Interviews

3.0 Findings

3.1 Relevance

Finding 1: CanCode addressed a demonstrated need for digital skills knowledge and training to improve the digital skills of teachers and youth across Canada including in rural, remote and indigenous communities and to prepare them for emerging innovative/technological industries.

The need for coding and digital skills training

Within the Canadian context, there is a need to prepare youth with the appropriate skills for the future labour market, support teachers in developing skills and confidence, and to facilitate targeted training to diverse groups of youth that may traditionally face barriers to pursuing career pathways in STEM.

The Canadian labour market is rapidly evolving due to the implementation of advanced technologies, like automation, robotics, and artificial intelligence, and Canada is expected to experience a skills gap.^{Footnote 1} 32% of current jobs will require significantly different tasks and skills.^{Footnote 2}

Barriers to skill development amongst K-12 youth

Evidence indicates that there is a lack of streamlined curricula across Canada for coding and digital skills.^{Footnote 3} While it has been mandated in some provinces, in others, coding remains optional.^{Footnote 4}

By grades 11 and 12, STEM courses are dropped by more than half of students due to the perception they are irrelevant to their career goals or that they are too difficult.^{Footnote 5}

The evaluation revealed that teachers are often uncomfortable teaching coding and digital skills due to lack of experience and knowledge.

Underrepresentation in STEM

Evidence demonstrates that certain sociodemographic groups, such as women, Indigenous people, people from rural and remote communities, and low-income individuals, experience barriers to pursuing STEM education and careers.

Challenges or barriers:

• Lack of encouragement from support system^{Footnote 6}
• Intergenerational trauma due to colonization and discrimination^{Footnote 7}
• Lack of access to learning opportunities, appropriate equipment, and/or infrastructure^{Footnote 8}
• Systemic barriers, leading to poor academic performance^{Footnote 9}

The evaluation revealed that CanCode addressed identified needs through coding and digital skills training offered to diverse K-12 youth and teachers across Canada. 98% of teacher survey respondents Agreed or Strongly Agreed that CanCode training had contributed to developing teachers' and students' knowledge and confidence in coding and digital skills.

Some recipient organizations mentioned that without ISED's funding, they would have added fees to services offered. Others noted that they would not have offered the type of training they provided.

Alignment to federal government priorities

CanCode objectives align with government priorities through encouraging diverse youth to explore STEM career pathways and building coding and digital skills amongst youth and K-12 teachers. The program is linked to other Government Priorities through the Youth Employment and Skills Strategy (YESS).

Innovation and Skills Plan^{Footnote 10}
Position Canada as a world-leader in innovation, create well-paying jobs and strengthen the middle class.

Canada's Digital Charter^{Footnote 11}
First principle: Ensure that all Canadians have equal opportunity to participate in the digital world and the necessary tools and skills to do so.

2022-2023 Departmental Plan^{Footnote 12}
Result: Promote digital innovation, adoption, and literacy.

CanCode complements other federal programs, such as Digital Skills for Youth (DS4Y), Digital Literacy Exchange Program (DLEP), Computers for Schools Intern (CFSI) program, and Let's Talk Science (LTS) which aim to develop Canadians' coding and/or digital skills.

International comparisons suggest that developing youth's coding and digital skills is also a priority in many countries, and several programs like CanCode exist internationally. CanCode enables Canada to grow talent and remain competitive relative to other countries.

3.2 Performance

Finding 2: CanCode has been successful in providing digital training and digital learning activities to a diverse range of youth and teachers across Canada, including underrepresented groups and despite COVID-19 restrictions. The program has increased interest in STEM and digital skills knowledge.

Digital skills training to youth and teachers across Canada led to increased digital knowledge and skills

CanCode has been successful in reaching students and teachers and providing digital skills training. Through the CanCode program, students had access to coding and digital skills learning opportunities in various regions and across the various Canadian provinces. They had the opportunity to learn digital skills, like coding, data analytics and digital content development.

Through the various activities delivered by recipient organizations, CanCode exceeded its targets in providing digital skills training, engaging youth and reaching diverse and traditionally underrepresented groups, including girls; Indigenous youth; youth with disabilities; and youth living in rural/remote/northern communities.

CanCode's reach by provinces/territories (p/t)+

Province/Territory	Students	Teachers
British Columbia	754,627	26,638
Alberta	461,082	19,757
Saskatchewan	557,980	18,636
Manitoba	266,567	9,716
Ontario	3,290,481	129,307
Quebec	577,259	22,712
New Brunswick	376,703	39,601
Prince Edward Island	50,218	3,855
Newfoundland and Labrador	251,436	17,534
Nova Scotia	331,989	32,649
Yukon	21,554	1,122
Northwest Territories	83,056	2,839
Nunavut	95,801	2,961

+ Excludes online participants for which p/t is unknown.

Underrepresented Group	# of Participants
Girls	3,502,129
Indigenous Youth	494,455
Persons with Disabilities	194,337
Rural, Remote, Northern Youth	1,604,803
Total	5,795,724

Participant reach vs targets – all phases:

• Students – actual number: 7,621,161
• Student – target: 6,009,642
• Teachers – actual number: 405,599
• Teachers – target: 270,669

CanCode Promoting Inclusive Digital Skills and STEM Education

The CanCode digital skills training program highlights an achievement in addressing educational inequalities and fostering inclusivity in digital education across Canada. The effectiveness of CanCode in delivering digital skills training to a wide and varied demographic underscores its role in nurturing a digitally skilled and diverse future workforce. This underpins the program's achievements in stimulating STEM interest, promoting equal training opportunities, and reaching underrepresented groups, thereby driving a more inclusive and innovative digital future in Canada.

Stimulating interest in STEM (Science, Technology, Engineering, and Mathematics)

Preliminary indications show that CanCode training is sparking curiosity in students towards STEM education and positively influenced teachers' attitudes towards coding and digital skills. By providing engaging digital skills training, CanCode is cultivating a new generation of innovators and technologists.

Promoting equality in digital skills training opportunities

CanCode's training reach into rural, remote, and Indigenous communities provided digital education where it is most needed. Geographical and socio-economic barriers did not hinder access to essential digital skills training for both youth and teachers.

Reach to underrepresented groups

CanCode training has successfully reached underrepresented groups such as girls, youth with disabilities, newcomer immigrants, and youth in rural and Indigenous communities. CanCode encourages inclusion and diversity in the tech sector, which is vital for a comprehensive and innovative digital future in Canada.

Impact on teachers' coding and digital skills development

Most teachers rated CanCode training as 'Very Good' to 'Excellent' for developing their coding and digital skills.

CanCode increasing digital skills knowledge and sparking digital curiosity

CanCode has made important strides in equipping young minds with essential digital skills. The program has not only improved the digital skills of youth but sparked their interest in STEM education. As a result of their engagement with CanCode's interactive learning activities, some students reported improved digital skills knowledge. Furthermore, the exposure to practical applications of technology has inspired many to consider pursuing education in the STEM fields. This highlights the program's effectiveness in fostering future innovators and tech leaders.

CanCode's positive impact on teachers' STEM education perspectives

Recipients noted that most teachers with no STEM education/background initially felt less confident in integrating coding and digital skills into their curriculum and teaching.

CanCode training reshaped teachers' attitudes towards STEM education and improved their confidence in integrating digital skills into their teaching practices and curriculum.

"This workshop exceeded my expectations! It allowed me to get some hands-on experience with coding, especially as a beginner!"
- Teacher

Empowering teachers: Elevating digital skills teaching with CanCode

The CanCode digital skills training program has showed success in enhancing the digital capabilities of teachers engaged in the program. Its success is attributed to its tailored approach, which resonates with the educator's needs, offering flexible, relevant, and engaging training modules. The program has successfully navigated the common challenges of professional development, providing teachers with not just theoretical knowledge but also practical digital skills that can be seamlessly integrated into their teaching practices.

CanCode's influence is demonstrated in the increased numbers among teachers to incorporate digital tools and methodologies into their curricula, enriching the learning experience for their students.

CanCode training improved digital skills:

• Exceeded its training targets
• Greater willingness to incorporate digital skills into the curriculum
• Empowered teachers to explore and adopt innovative teaching methods
• Positive impact on teachers' attitudes toward digital skills teaching
• Free of charge training broadened accessibility
• Engaging content resulted in active participation

Utilization of CanCode resources:

• Teachers that have access to tools and resources to support the curriculum: 576,233
• Teachers engaged with CanCode training: 405,599

Navigating the Covid-19 era successfully

The outbreak of COVID-19 presented unprecedented challenges. Physical distancing measures and lockdowns disrupted traditional in-person learning models. The adaptability of the CanCode digital skills training program was demonstrated during this period. The program maintained its reach, performance, and adopted innovative methods to continue delivering digital skills training.

• Transition to online platforms: Recipients migrated their training modules to online platforms. Utilizing various video conferencing tools like Zoom and Microsoft Teams, they facilitated interactive virtual classrooms, ensuring uninterrupted digital skills training
• Social media engagement: Social media was employed to share educational content and engage with students and teachers. This approach expanded the reach to a broader audience
• Mail-based learning kits: To ensure inclusivity, particularly for areas with limited internet connectivity, some organizations sent out physical learning kits. These kits included printed materials, USB drives and physical STEM education activities
• TV broadcasts: Leveraging television networks, educational content was broadcasted using cable and satellite TV, along with community programming to ensure that learning continued even for those without internet access. This method proved particularly effective in reaching remote and rural communities
• Asynchronous learning materials: Provided pre-recorded video tutorials, downloadable instructional materials, and online activities

Finding 3: CanCode delivered digital skills learning and training activities using innovative, multimodal, and inclusive approaches, which is contributing to building a creative and thriving digital ecosystem.

Cultivating a digital future: CanCode's innovative approach

The CanCode program has employed innovative, multimodal, and inclusive strategies. These have included diverse, hands-on activities like robot building workshops and coding sessions, creatively tailored to engage a wide array of learners with various needs.

The program's adaptability in delivering both virtual and in-person training, along with its focus on culturally relevant content, particularly for Indigenous communities, underscores its commitment to inclusivity. By doing so, CanCode has contributed to nurturing a forward-thinking digital ecosystem, empowering both students and educators with essential 21st-century skills.

Examples of the various approaches to digital skills learning

• Engaging youth with disabilities: Digital literacy lessons and assistive technology skills development for students with disabilities (e.g. Using digital tools to make films, take professional quality photographs and edit them, produce albums, and graphic design)
• In person and virtual ventures: In person workshops, after school clubs, collaborative in-person activities (e.g. Video calls; live streamed events; virtual clubs and online support; integrating physical materials with virtual instruction)
• Inclusive, culturally relevant content: Content designed for a specific audience (ex, Indigenous communities) (e.g. Activities connecting Western and Indigenous STEM; Creation of prototype projects involving technology and integrating Indigenous Traditional Knowledge and culture)
• Coding workshops and robot building: Hands-on technology engagement using diverse tools, combining physical building with digital coding (e.g. Activities focusing on AI and Machine Learning, 3D/4D Design, and Robotics. Engaging students with tools like Ozobots, Microbits, Makey Makeys, and Lego NXT Robots)
• Play and art into coding: Creative activities that blend play and art with coding, appealing to a wide range of interests (e.g. Project based-learning activities. Youth creating projects using digital creation, AI, coding, and robotics (creating DIY wearable technology, developing games))
• All-girls programs: Hosting virtual all-girls conferences, engaging underrepresented groups in STEM (e.g. Development of marketing materials and workshops led by female (Bluecoats); All-Girls clubs and camps)

Finding 4: The evaluation did not find a harmonized approach to data collection across the phases of the program. The varying methods in data collection led to a range of data quality, impacting the uniformity of performance data. There were limitations in gathering data on GBA Plus/EDI, suggesting a need for a more cohesive approach in these areas.

Navigating data diversity: Challenges in data collection and monitoring progress

The evaluation faced challenges in assessing the quality of performance data, primarily due to variations in data collection methods among different recipient organizations. The variation in methods used by different organizations for collecting performance data presents an opportunity for enhancing the consistency and depth of information gathered. It indicates a need for more uniform and structured approaches to ensure data quality.

Variability in data collection: Improvements in data collection across the phases of CanCode resulted in various performance data gathering approaches using different tools and methods leading to challenges in uniformly assessing the robustness of the program's data.

• Manual data entry: Manual data entry is prone to errors like typos, omissions, and duplications
• Double-counting: Joint delivery of program activities led to some double counting of participants, inadvertently increasing the total participation number
• Challenges in GBA+/EDI Data Collection: Difficulties in collecting demographic data including GBA Plus and EDI due to privacy, ethical concerns, and sensitive nature of questions
• Opportunities for the use of qualitative insights: While qualitative data was part of the data collection, the reporting emphasis was on quantitative metrics. This may have missed capturing the nuanced changes in attitudes, perceptions, and experiences

Methods used by recipient organizations for data collection:

• Registration surveys
• Exit surveys
• Attendance sheets
• Website visitor tracking
• Tracking participants through the hub
• Participant counts

Recommendation 1: ISED Connected Canada Branch, Spectrum Telecommunications Sector should explore approaches to enhance the consistency of recipient-submitted data. Reliable and simple data tools as well as consistent methodologies would allow the program to better track its progress against outcomes, alleviate reporting requirements for recipient organizations and identify gaps in program reach relative to equity-deserving groups.

Finding 5: Over the three phases of CanCode, progressive enhancements were made to the reporting requirements; however, reporting mechanisms and requirements remained onerous and challenging for recipient organizations.

Reporting on CanCode's progress

Since its inception, the CanCode digital skills program has made improvements to performance reporting across its three iterations. Over time, the program has taken measures to streamline reporting processes while ensuring compliance with TBS requirements. It has also refined quarterly progress reports and final implementation reports, leading to more consistent reporting on performance measures. Despite, these advancements, the program's approach to reporting is considered time consuming by the recipient organizations.
Recipient organizations noted the following:

• Onerous reporting requirements: Recipient organizations found the reporting process laborious and time-consuming, especially with four required Quarterly Progress Reports (QPRs) and a final implementation report and financial reporting requirements further added to this
• Lack of tailored reporting approaches: Despite some organizations focusing on digital skills training for both teachers and youth, and others exclusively for students or for specific underrepresented groups, all were required to use the same reporting structure
• Unclear guidelines on financial reporting: Lack of clear guidelines on eligible expenditures led to confusion and rejections of certain costs, adding to the administrative burden
• Heavy administrative burden: Organizations, particularly the small ones, felt the strain of the reporting requirements due to limited resources
• Adjustments to reporting system: Recipient organizations experienced multiple changes in reporting templates, procedures, and policies over multiple phases, causing confusion and increased workload on them.

Recommendation 1: ISED Connected Canada Branch, Spectrum Telecommunications Sector should explore approaches to enhance the consistency of recipient-submitted data. Reliable and simple data tools as well as consistent methodologies would allow the program to better track its progress against outcomes, alleviate reporting requirements for recipient organizations and identify gaps in program reach relative to equity-deserving groups.

3.3 Efficiency

Finding 6: CanCode funding contributed to increased partnerships and leveraging of resources to advance program objectives. Continued efforts to enhance partnerships, increase knowledge sharing and networking among recipients through a Community of Practice would further advance the reach of the program.

Partnerships in CanCode's journey

The CanCode recipient organizations were able to leverage private and public partnerships which increased its impact. The multifaceted partnerships and collaborations fostered by CanCode recipient organizations contributed to the successful implementation and outreach of the Program and to achieving and even surpassing program objectives.

Diverse partnerships and their impacts

Private and public sectors that CanCode has partnered with:

• Educational institutions (universities)
• Community groups
• Collaborations with underrepresented groups
• Schools and school districts
• ICT industry players

These partnerships have allowed for resources to be leveraged:

• Securing additional funding and in-kind contributions
• Pooling expertise

These partnerships have also facilitated enhanced reach and effectiveness:

• Program met and exceeded targets
• Variety of stakeholders enabled reaching diverse resources and expertise
• Ensured a more inclusive approach to digital skills training
• Broadened program's impact

Best practices: Establishing a Community of Practice (CoP)

The establishment of a Community of Practice (CoP) was described by recipient organizations as one of the Program's best practices and its benefits were noted. In the short period that the CoP was in place, recipients had the chance to share experiences in delivering their program's activities, solutions for optimizing challenges, best practices in content development, and outreach, particularly to under-served communities. The evaluation identified potential areas for improvements in collaborative efforts, particularly enhancing the structure and coordination of the Community of Practice.

In the meetings of the CoP recipients had the chance to:

• discuss approaches to address and improve situations (data collection, reporting…) and some program delivery challenges
• exchange practical experiences, knowledge, and insights related to implementing their program activities, conducting workshops, and engaging more participants

The program promoted the exchange of tools among recipient organizations, enhancing collaboration. Additionally, these meetings led to the formation of new partnerships.

The CoP provides a platform for exchanging ideas, strategies, and experiences, and fosters collaboration and contributed to addressing common challenges encountered by some organizations. Recipient organizations expressed a desire for more frequent meetings of the Community of Practice. They noted that such structured support activities would:

• Create a support network that is especially valuable for smaller organizations
• Ensure ongoing learning and adaptation of best practices

Recommendation 2: ISED Connected Canada Branch, Spectrum Telecommunications Sector should explore strategic approaches and further leverage the Community of Practice forum to enhance knowledge sharing, networking initiatives and collaborative learning among recipient organizations. Such approaches could further enhance innovation, promote best practices and advance the goals of the program.

Finding 7: CanCode's use of various delivery models and implementation of free training across regions and organizations contributed to increased participation and engagement. Some challenges were identified in the application process, project approval and funding allocation processes.

Efficient use of resources

CanCode successfully leveraged its resources to enhance digital skills and promote digital literacy, demonstrating an effective approach in its programming. Data, however, was not available across all phases to fully assess efficiency of activities and costs per participant.

• Free of charge training model removed financial barriers and broadened reach and accessibility
• Developing bilingual digital content; culturally sensitive instructors and digital content
• Up to 10% of the funding is used to cover administrative costs

Various delivery models representing adaptability

By employing various delivery models of the program activities, the program successfully reached a broad audience and participants from various backgrounds and regions had access to CanCode digital skills training.

Challenges in the application, approval, and funding process: Implications for program efficiency

CanCode recipient organizations expressed gratitude for the ISED funding provided for delivering digital skills training, acknowledging its important role in facilitating those important learning activities for enhancing youths and teachers' digital literacy. The funding not only enabled them to reach a wider diverse audience, but it also contributed to expanding their partnerships and networks. However, recipients encountered challenges in the application and approval processes, as well as in funding allocation.

Application process

• Need for support during the application process: Recipients noted their need for additional guidance and assistance to navigate through and address the requirements of the funding application

Project approval and selection process

• Opportunities for enhanced feedback on unsuccessful applications: Recipients noted a need for feedback and additional insights on unsuccessful applications and/or parts of their proposals to help them improve in future submissions

Funding allocation

• Continued need for transparency: Recipients noted that allocation of funds and how funds are distributed among small new, and bigger organizations should be understood by all recipient organizations
• Limited or unapproved funding for some groups: Some organizations did not get their proposals fully funded due to limits in available program funding. This created challenges for some organizations to reach their intended communities or fully implement their activities.

Recommendation 3: ISED Connected Canada Branch, Spectrum Telecommunications Sector should explore opportunities to enhance the program application process to support applicants and communicate decisions.

4.0 Summary & recommendations

Conclusion

CanCode met the growing need for digital skills knowledge and training to enhance the digital literacy of youth and teachers across Canada. It reached a diverse range of participants, including underrepresented groups, by employing innovative and adaptable delivery methods that were successfully adapted to COVID-19 restrictions. While the program succeeded in its outreach and inclusivity; areas such as data collection, reporting, and the administrative burden on recipient organizations require enhancements. The establishment of a structured Community of Practice and improvements in the application, approval, and funding processes were identified as areas for further development. Overall, CanCode's contribution to fostering digital skills among Canadian youth and teachers is setting the foundation for a digitally proficient future workforce.

Best Practices: Establishing a Community of Practice was one of the program's best practices as it facilitated exchanging ideas, collaboration and addressing common challenges among recipient organizations.

Findings

• Finding 1: CanCode meets the demonstrated need for digital skills knowledge and training to improve the digital skills of teachers and youth across Canada including in rural, remote and indigenous communities and to prepare them for emerging innovative/technological industries.
• Finding 2: CanCode has been successful in providing digital training and digital learning activities to a diverse range of youth and teachers across Canada including underrepresented groups and despite COVID-19 restrictions. The program has increased interest in STEM and digital skills knowledge.
• Finding 3: CanCode delivered digital skills learning and training activities using innovative, multimodal, and inclusive approaches which is contributing to building a creative and thriving digital ecosystem.
• Finding 4: The evaluation did not find a harmonized approach to data collection across the phases of the program. The varying methods in data collection led to a range of data quality, impacting the uniformity of performance data. There were limitations in gathering data on GBA Plus/EDI, suggesting a need for a more cohesive approach in these areas.
• Finding 5: Over the three phases of CanCode, progressive enhancements were made to the reporting requirements. Despite these improvements, reporting mechanisms and requirements remained onerous and challenging for recipient organizations.
• Finding 6: CanCode funding contributed to increased partnerships and leveraging of resources to advance program objectives. Continued efforts to enhance partnerships, increase knowledge sharing and networking among recipients through a Community of Practice would further advance the reach of the program.
• Finding 7: The CanCode program has demonstrated efficient use of resources to produce outcomes. The use of various delivery models, and implementation of free training across regions and organizations contributed to increased participation and engagement. Some challenges were identified in the application process, project approval and funding allocation processes.

Recommendations

• Recommendation 1: ISED Connected Canada Branch, Spectrum Telecommunications Sector should explore approaches to enhance the consistency of recipient-submitted data. Reliable and simple data tools as well as consistent methodologies would allow the program to better track its progress against outcomes, alleviate reporting requirements for recipient organizations and identify gaps in program reach relative to equity-deserving groups.
• Recommendation 2: ISED Connected Canada Branch, Spectrum Telecommunications Sector should explore strategic approaches and further leverage the Community of Practice forum to enhance knowledge sharing, networking initiatives and collaborative learning among recipient organizations. Such approaches could further enhance innovation, promote best practices and advance the goals of the program.
• Recommendation 3: ISED Connected Canada Branch, Spectrum Telecommunications Sector should explore opportunities to enhance the program application process to support applicants and communicate decisions.

Appendix A: Expected outcomes

Through the eligible activities funded under the ISED funding agreement, CanCode is expected to achieve the outcomes below.

Short-term outcomes:

• Canadian youth (K-12) including youth from underrepresented groups have access to coding and digital skills learning opportunities through the CanCode 3.0 Program
• Canadian teachers have access to tools and training to teach digital skills and coding to their K-12 students through the program

Medium-term outcomes:

• Canadian students increase their capacity to effectively code and use digital skills
• Canadian teachers increase their capacity to incorporate and teach coding and digital skills in their classroom

Long-term outcomes:

• Canadians graduate from STEM fields in Canadian post-secondary institutions

Appendix B: Funding eligibility criteria

For CanCode 3.0, applicants needed to meet the following criteria to be eligible for ISED funding:

• Your organization or partners has a minimum of 3 years' experience in the delivery of coding and digital skills
• Your proposed initiative will target at least one traditionally underrepresented group in the STEM field
• The proposed initiative will be delivered at no cost to participants (i.e. no participant user fees)
• Your organization will have the resource capacity and expertise with program funding, either internally or through partnerships, to successfully deliver the proposed initiative within the program time frame September 1, 2021, to March 31, 2024, and agree to accurately measure and report on success
• How your organization plans to be fully self-sustainable at the end of the program (exception for organizations serving primarily underrepresented groups which are still expected to demonstrate progress to self-sustainability)

Appendix C: Methodology

The evaluation was based on five data collection methods, including qualitative and quantitative sources. In order to answer the evaluation questions, information from multiple lines of evidence was collected and triangulated.

Literature and document review

The literature review was comprised of pertinent literature to gain a thorough understanding of the current and continued
need for CanCode from the perspectives of increasing coding and digital skills amongst Canadian youth and teachers. An international comparison was also performed of other similar programs in other countries to understand CanCode's relevance from an international lens. The document review included key government priority setting documents and program reporting documents to support the assessment of performance and efficiency.

Performance, administrative and financial data review

CanCode performance data, collected as part of the CanCode's performance measurement requirements, was reviewed in order to assess the extent to which progress has been made towards achieving the short-term, medium-term, and long-term outcomes outlined in the logic model. An analysis of the administrative and financial data for CanCode was also performed to get a comprehensive understanding of the program's operations (e.g Resource allocation and management, and transparency).

Case studies

One case study was conducted to support the overall evaluation of CanCode. The case study cross-examined 12 recipient organizations of various sizes. The case studies relied on document review and administrative data.

Online surveys

Online surveys were conducted to gather perspectives on the impacts of CanCode, in addition to the potential improvements to enhance the program's effectiveness. The surveys targeted K-12 teachers that had participated in CanCode training activities to develop their skills and staff with recipient organizations which were not selected for interviews. Surveys resulted in responses from a total of 89 teachers, of which 28 respondents' answers were removed due to various reasons, and 8 recipient organization staff.

Virtual interviews

A total of 21 interviews were conducted using MS Teams to gather diverse perspectives on the relevance, performance and efficiency of CanCode. Interviews were facilitated with recipient organizations and ISED staff.

Appendix D: Challenges and mitigations

The evaluation encountered 3 limitations and evaluators applied related mitigation strategies.

1. Respondent bias
   Challenge: Interview and survey participants were either involved in program delivery or direct beneficiaries and, as a result, responses may have been positively biased.
   Mitigation: The purpose of the interview and its strict confidentiality was communicated to participants. Responses were validated across stakeholder groups as well as other lines of evidence.
2. Survey response rate
   Challenge: The response rate for the evaluation survey administered to participating CanCode teachers was low (36%). This was due to challenges in obtaining teachers' contact information from CanCode recipient organizations, and the launch of the survey coincided with the beginning of the school year when teachers were busy.
   Mitigation: Other lines of evidence were leveraged to support evidence from surveys.
3. Data quality
   Challenge: Given that CanCode is a newer program, reporting requirements have changed throughout the evaluation period. This created challenges in evaluating certain criteria that was not consistently reported on.
   Mitigation: The evaluation focused on data that was consistently reported on throughout the evaluation period and leveraged other lines of evidence where data was unavailable