Evaluation of Computers for Schools Plus (CFS+) and Computer for Schools Intern (CFSI) programs

From: Innovation, Science and Economic Development Canada

June 2023

1.0 Program Context

1.1 Program Descriptions

Computers for Schools Plus* (CFS+) program

The Computers for Schools Plus program (formerly Computers for Schools) works with CFS+ Affiliates and partner organizations across Canada to collect, refurbish, and distribute donated computer equipment and electronic devices to schools, libraries, not-for-profit organizations, Indigenous peoples, and eligible low-income Canadians around the country. In 2017, the program expanded its reach to provide additional refurbished computers to low-income Canadians through the Connecting Families Initiative (CFI).

The CFS Intern (CFSI) Program

Delivered by CFS+ Affiliates, the CFSI program aims to enhance the employability of youth (15 to 30 years old) by providing employment internships. CFSI is the in-house workforce for the CFS+ refurbishment workshops and is part of the Government of Canada's Youth Employment and Skills Strategy (YESS).

1.2 Program Funding

The CFS+ had average annual funding of $5.4 million. Additional top up of $1.0 million to CFS+ in both 2016-17 and 2017-18 provided an additional 10,000 computers in each of those years and extended the program's reach. (Figure 1)

The CFSI had average annual funding of $3.7 million and received an additional $1.13 million in 2019-20 under YESS to support more youth internships. In 2020–21, the CFSI program was also provided with an additional $5.2 million in funding to help support more internships as part of the government response to the COVID-19 pandemic. (Figure 2)

The combined budget for both programs during the evaluation period (April 1, 2016 and March 31, 2022) represents $76.0 million, of which $70.3 million was for Grants and Contributions (G&C) and $5.6 million was for operating expenditures (O&M).

1.3 Program Delivery

ISED

The Connected Canada Branch within ISED's Spectrum Telecommunications Sector is responsible for oversight and ongoing management of CFS+ and CFSI.

CFS+ Affiliates

CFS+ and CFSI are delivered through a network of 13 not-for-profit organizations and the Government of Prince Edward Island (CFS+ Affiliates) under contributions agreements with ISED (Figure 3). CFS+ Affiliates are independent, self-governing organizations, responsible for overseeing the refurbishment and distribution activities in their jurisdiction and for hiring youth and providing them with meaningful internships under CFSI.

CFS+ Affiliates assess donated computers against the minimum specifications. Those that meet the specifications are refurbished for the program's beneficiaries and those that do not are sent to a certified recycler.

National CFS+ Organizations

CFS+ Affiliates are supported by 2 National CFS+ Organizations:

• National Technology Centre (NTC): distributes devices and parts, develops refurbishment standards and helps deliver devices for national initiatives.
• Computers for Success Canada (CFSC): facilitates coordination between CFS+ Affiliates, the NTC, and ISED; leads national promotion initiatives; solicits donations; supports CFS+ in Nunavut, and delivers the CFI.

2.0 Methodology

2.1 Evaluation Context

An evaluation of ISED's funding to CFS+ is required every 5 years under the Financial Administration Act.

The objective of this thematic evaluation was to assess the relevance, performance and efficiency of ISED funding to the CFS+ jointly with the CFSI program, given the integral support that the CFSI provides CFS+.

The scope of the evaluation includes the CFS+ and CFSI, as well as their efficiency in delivering the CFI. The evaluation covered all ISED funding to the CFS+ and CFSI during the period from April 1, 2016, to March 31, 2022.

The evaluation was by the Audit and Evaluation Branch at ISED. A results-based approach was used to examine the achievement of expected outcomes for the CFS+ and CFSI, as identified in the logic model (Appendix A).

The thematic evaluation questions were designed to look at both programs through a common frame and leverage indicators to ensure a relevant and useful assessment. Although the evaluation of both programs was conducted jointly, each program was also independently assessed.

2.2 Evaluation Questions

Relevance

1. To what extent is there a unique and continued need for the CFS+ and the CFSI?

Performance

2. How has the development of a network of partners contributed to the achievement of the programs' expected outcomes?
3. To what extent have the programs increased accessibility to computer technology for Canadians and reduce the environmental footprint associated with surplus electronic equipment?
4. To what extent has the program contributed to learning opportunities for youth of diverse backgrounds and genders, and enhanced the employability of youth through these internships?

Efficiency

5. To what extent do the programs and the CFS+ Affiliate delivery models demonstrate operational efficiency?

The evaluation produced 10 findings, supported by multiple lines of evidence, and leading to 2 recommendations.

2.3 Data Collection Methods

The following lines of evidence were used (details in Appendices B and C):

1. Literature and Document Review
2. Performance, Administrative and Financial Data Review
3. Virtual Interviews
4. Online Surveys
5. Case Studies

3.0 Findings

3.1 Relevance

Access to technology critical to participating in today's digital economy is addressed through CFS+.

Technology has revolutionized the way the world operates including how people communicate, think, and learn. Digital technologies are associated with economic and social benefits, which continue to shape the lives of individuals, communities, and societies.^{Footnote 1}

Access to the internet is important given that it is used by Canadians to find jobs, conduct business, further their education, keep informed on matters of public concern, consult health care professionals, and interact with all levels of government. Similarly, in interviews, technology was perceived to be synonymous with greater access to health services, financial services, communication, and information.

To thrive in the digital economy, citizens must also possess digital skills, without which they are limited in the benefits they can expect to receive and may face barriers to their full participation in society.^{Footnote 2} The organizations and individuals served by the CFS+ program do not have the financial means to purchase computer technology, and many depend on donated computer equipment to address their technology needs or risk going without.

The CFS+ program is designed to address the gap in access to technology for Canadians through the refurbishment and distribution of end-of-life electronic devices (computers, tablets, smartphones) at little or no cost to the end user. Internet access for low-income families is also provided through the Connecting Families Initiative. The evaluation found that the CFS+ program provided Canadians with the computer equipment necessary to work and live in a digital knowledge-based economy.

The COVID-19 pandemic increased the need for access to technology and highlighted a gap in access for certain groups.

While access to technology and the Internet were considered important prior to the COVID-19 pandemic, the lockdowns in 2020 increased society's reliance on technology. Findings from the 2021 Households and the Environment Survey (HES) showed Canadian households increased their screen time during the pandemic. Individuals with access to the Internet could continue to work, run a business, engage in civic activities, pursue education, and socialize with friends and family from their homes.^{Footnote 3} The pandemic also revealed that Canadians were not equally well positioned to cope, adapt or thrive in an online environment.^{Footnote 4}

The gap in access to technology and the Internet became significantly more pronounced during the pandemic as many individuals were not well equipped to transition to a largely online environment. Documents and literature suggested that Indigenous peoples, those that live in rural or remote areas, seniors, and those in the lowest income quintile faced greater barriers in accessing technology and the Internet in Canada^{Footnote 5Footnote 6Footnote 7}; these were key client groups of the CFS+. The access gap for these groups resulted from barriers such as income, literacy, geography, and ability. Despite investments by the federal government in these areas, many Canadians were unable to fully participate in the digital economy.^{Footnote 8}

Interviews with CFS+ Affiliates revealed that during the pandemic there was an even greater demand for devices equipped with a webcam and Wi-Fi connection. These enabled students, teleworkers, and seniors to maintain social contacts, access services, or to perform tasks for school or work from home. Literature indicated that family members who were forced to share space, computer devices, and Internet bandwidth struggled to keep up with work and school.^{Footnote 9} It was noted that demand for portable devices, such as laptops (see Figure 4) and tablets also increased. These devices allowed recipients the flexibility to move to quieter spaces within their home or for the device to be tucked away when not in use.

The CFS+ program's goal of increasing access to technology for Canadians aligns with federal government priorities.

The CFS+ program is aligned with the Innovation, Science and Economic Development Canada's Innovation and Skills Plan (ISP), announced in Budget 2017. The ISP was an effort to make Canada a world-leading centre for innovation, to create well-paying jobs, and to help strengthen and grow the middle class. The CFS+ program aligns with the ISP by increasing Canadians' access to technology and the development of digital skills. Accordingly, these activities address the ISP's strategic outcome of ensuring that Canadian businesses and communities are competitive.

The objectives of CFS+ are aligned with the federal government's priorities of supporting Canada's most vulnerable individuals, who have been significantly impacted by the COVID-19 pandemic. Given society's increased reliance on technology since the pandemic, more supports are required to reduce barriers for Canadians in accessing technology and the Internet.

CFS+ also contributes to Canada's Digital Charter, especially the first principle of universal access. This includes providing Canadians with equal opportunities to participate in the digital world, as well as access to the necessary tools to do so.

The evaluation found that government funding was perceived by stakeholders as very important. This perception has probably increased due to the recent rise in inflation and the impacts of the pandemic, which have led to increased costs and reduced in-kind contributions to Affiliates over the years. The program has experienced increased pressure on its operations due to these changing financial circumstances.

The Innovation and Skills Plan (ISP) pillars:

• Equipping workers with the tools, skills and experience they need to succeed.
• Fostering a nation of innovators through financial assistance and improved innovation.
• Promoting key sectors in Canada's innovation economy such as clean technology, digital industries and agri-food.

While Government of Canada programs shared similar goals, the CFS+ was unique in operation and scale.

Among the programs examined, CFS+ has been in existence the longest, having been established in 1993. The program operates on a national scale, with CFS+ accepting donations and distributing refurbished devices in all provinces and territories. Nunavut's delivery of the program is supported through CFSC's partnership with the Pinnguaq Association. Refurbishment of devices does not take place in Nunavut. Instead, devices are shipped from other CFS+ Affiliates to the territory.

No programs or organizations were found to duplicate the activities or scope of the CFS+. Some initiatives had similar goals to CFS+ (providing refurbished computer equipment), such as reBOOT Canada, but operated on a smaller scale, charged for equipment, or were specific to the province or region. Government programs, such as the Digital Literacy Exchange Program and Digital Skills for Youth (DS4Y), contributed to Government of Canada goals similar to those of CFS+. These included digital skills and literacy development for Canadians, but did not duplicate the CFS+, either in terms of scope or target beneficiaries.

Canadian youth face lower rates of employment than other age groups.

Canadian youth between the ages of 15 and 30 are less likely than older Canadians to have a job (see Figure 5). Canadian youth face lower rates of employment. Reasons include being in school full-time, discrimination by employers who perceive youth as less qualified or valuable, lack of work experience and a decrease in low-wage positions in 2019-2020, which are typically held by youth.^{Footnote 10Footnote 11}

Research revealed that learning opportunities for youth have proven to be very valuable for employability and for bridging the gap between the school curriculum and working life.^{Footnote 12} This was echoed by a CFS+ Affiliate who described seeing many youth graduating from IT programs with a strong theoretical background but lacking first-hand experience. It was also noted that with the shift to a knowledge-based economy employers place more emphasis on soft skills, such as problem-solving, critical thinking, and communication. These are not taught effectively in educational institutions^{Footnote 13} but were benefits provided through the CFSI program.

Diverse youth encounter additional barriers to entering the Canadian labour market.

Additional supports are necessary for diverse youth (i.e., youth from various languages, genders, ethnic origins, and ability levels) who face unique challenges in obtaining employment. A 2017 report by the Expert Panel on Youth Employment highlighted that Canada's diversity was not reflected in the culture of its workplaces or in the programs offered to support employment.^{Footnote 14} Youth who were not in school or in the labour market saw their skills erode over time.^{Footnote 15} In addition, Indigenous youth, female youth, recent immigrant youth, and youth with disabilities encountered barriers when entering the labour market or in the workplace.^{Footnote 16Footnote 17Footnote 18}

The nature of the barriers that diverse youth experience included:

• Employment and training opportunities for rural or remote youth tend to be limited. As a result, youth must often leave their communities to advance their careers or education.^{Footnote 19} Moreover, a lack of available transportation is a challenge for rural and remote youth, leading many to rely on private resources for transportation, which can be costly.
• Systemic and indirect discrimination creates barriers for not only obtaining employment, but also with regards to advancement in the workplace. Youth that face discrimination often include Indigenous youth, recent immigrant youth, LBGTQ2+ youth, those who have disabilities, those with mental illnesses, racialized youth, those with criminal backgrounds, and youth experiencing homelessness.^{Footnote 20}
• Highly educated young individuals had a triple advantage in the Canadian labour market over less-educated young individuals as they were more likely to have a job, to hold a full-time permanent position, and to receive a higher wage.^{Footnote 21} Therefore, youth without higher education may face barriers in obtaining employment that meets their needs.
• Employers' inability to provide suitable accommodations can create additional barriers for youth with disabilities seeking to secure useful work experience.^{Footnote 22}

The CFSI program supported youth in developing skills and gaining practical work experience to increase their employability, which is aligned with federal government priorities.

The CFSI was successful in providing practical work experience to youth. It was noted during interviews that the CFSI had increased the employability of youth interns through participation in internships, which positioned them advantageously to move forward in their careers with other employers. Evaluation evidence, interviews, and case studies indicated that the work performed as part of the CFSI program was critical for the functioning of the overall CFS+ program as interns performed most of the refurbishment activities. Stakeholders indicated that without government funding for the CFSI program, the objectives of the CFS+ program would not have been met.

The CFSI program aligns with several government initiatives. It contributes to the People and Skills pillar of ISED's Innovation and Skills Plan by helping the next generation of workers to be well equipped to participate in and contribute to the evolving Canadian economy. The CFSI objectives are in line with the Government of Canada's Youth Employment and Skills Strategy (YESS) and, in general, addressing systemic inequalities.

Providing employment opportunities to youth was considered by CFS+ Affiliates to be very important and there was a strong perceived need for the Government of Canada to support diverse youth in obtaining learning opportunities. Contribution agreements for CFSI funding encourage Affiliates to diversify the youth hired through the program to increase employment for underrepresented groups and women. Youth that participated in the CFSI gained digital skills, developed interpersonal competencies, and received meaningful work experience and mentoring—all factors contributing positively to their future employability. By providing support to youth employment, the Government of Canada is helping to ensure that future generations of workers have the skills needed to maintain a strong Canadian economy.

The accelerated growth of e-waste is an issue that warrants increasing attention.

According to the literature, e-waste is the fastest-growing waste stream in the world.^{Footnote 23} Each year, around 50 million metric tons of e-waste are generated globally^{Footnote 24}, and by 2030, current volumes are expected to double^{Footnote 25}. Furthermore, only 20% of the global e-waste is properly recycled^{Footnote 26}, suggesting that current methods of e-waste management are not effective.

Consumers are constantly seeking the newest, most innovative technologies, forcing outdated technologies to be disposed of in the waste stream^{Footnote 27}. The consumer electronics market (e.g., laptops, cell phones, tablets, and TVs) is a 1 trillion-dollar market, and the market itself is expected to grow by 7% each year between 2020 and 2026^{Footnote 28}. In 2016, half the global e-waste was sourced from personal devices, such as smartphones, tablets, laptops, computers, and TVs.^{Footnote 29} Given the growing consumption of electronic devices, paired with the increased pace of electronic obsolescence, the management of e-waste is becoming an issue that warrants increasing attention.

In a 2021 study published by the Council of Canadian Academies, Canada had an estimated circularity rate (the share of material resources used in the economy which come from recycled products and recovered materials) of 6.1%. This value is low when compared to that of EU countries, where the circularity rate is 14.4% in the EU-27. Therefore, Canada's low circularity rate suggests that more could be done to increase the use of recovered materials

E-waste is an international priority as it is extremely harmful to the environment, human health, and wildlife.

Environmentally sound e-waste management is an international priority that has resulted in the development of policy tools (e.g., the Basel Convention and the European Waste Electrical and Electronic Equipment (WEEE) Directive). Canada does not have specific federal regulations for e-waste management^{Footnote 30}, but there are programs (e.g., the Extended Producer Responsibility (ERP) program and the Electronic Product Stewardship (EPS) program) which aim to encourage electronics producers to be more accountable for their used or end-of-life products.

As e-waste grows, it poses significant risks to the environment and the health of people and wildlife. E-waste is extremely harmful to the environment when it is sent to landfills or incinerated. The hazardous materials that are present in electronic devices, such as mercury, lead, chromium, cadmium, and (poly)brominated flame retardants, can leach into the soil or create harmful emissions in the air. Effects of hazardous wastes on health include, but are not limited to, irreversible effects on the nervous system, effects on the immune system, interference with brain development, and damage to kidneys.^{Footnote 31}

Sustainable management of e-waste aligns with government priorities.

The CFS+ program's activities specifically target the issue of e-waste through the refurbishment of end-of-life electronic devices, and therefore addressing e-waste through the lens of a circular economy model (Figure 6). By refurbishing electronic devices, CFS+ diverts many devices from the landfill and, in turn, working devices are provided to Canadians, further reducing the need to manufacture new devices. The CFS+ and CFSI programs are aligned with government priorities as refurbishing electronic devices supports the Federal Sustainable Development Strategy of reducing Canada's carbon footprint, promoting responsible consumption and production, greening of government operations, and the Government of Canada's commitment to increasing green jobs.

3.2 Performance

The CFS+ Affiliates have implemented procedures in order to get the most use out of the donated equipment.

The evaluation found that the CFS+ and the CFSI were effective at maximizing the use of donated equipment to help promote reuse and recycling, and ensure residual e-waste was handled in an environmentally friendly manner. Figure 7 provides an overview of the process that takes place once equipment is received. Overall, the goal is to refurbish and reuse as much as possible, with any unusable equipment or parts being responsibly recycled.

Interns were primarily involved in processing and refurbishing the donated equipment. During interviews, some interns noted a sense of relief in seeing the large amount of equipment being diverted from landfills. Many interviewees noted that there should be more recognition of the environmental benefits resulting from the CFS+. Some also spoke of the need to increase awareness among the general population regarding the proper disposal of e-waste. 2 interviewees noted that before becoming involved with CFS+, they were unaware of the environmental issues associated with e-waste.

One Affiliate noted that for some donors, it would be easier to just throw out the old equipment, so CFS+ is effective at keeping e-waste out of landfills. It was mentioned that there is competition for e-waste, and some companies offer a full-service option where they will pick up the old equipment from organizations and pay them for the e-waste. These full-service companies later sell devices and parts for profit.

Environmentally, reuse and recycling of electronic devices are preferred options to disposal.

The evaluation found that refurbishing computers is environmentally sound as the manufacturing of a new device includes several processes, such as mining and global shipping of parts, that are harmful to the environment. Extending the life of a computer not only helps to reduce greenhouse gas emissions, but also helps to preserve natural resources and conserve more energy when compared to recycling computer equipment.

According to the waste management hierarchy (Figure 8) developed by the European Union^{Footnote 32}, reuse is preferable to recycling from an environmental perspective. Recycling and re-manufacturing new devices requires a large amount of energy, which is saved when devices are reused.

Reuse and recycling of electronics decrease impact on natural resources

Recycling is preferable to disposal without any further processing (i.e., computers being sent to landfills). Recycling of desktops and laptops saves up to 87% of natural resources^{Footnote 33}, in comparison to when these devices are sent to the landfill and resources are not recovered.

A typical desktop computer is composed of many elements and metals (Figure 9). The literature suggests that there is a scarcity of raw earth materials used in the production of electronic devices and that refurbishing and recycling are increasingly important to ensure that we are not depleting natural resources beyond sustainability.^{Footnote 34}

Through the CFS+, a significant number of devices were responsibly disposed of and kept out of landfills.

Between 2016 and 2022, the Affiliates responsibly disposed of 539,627 devices which could not be refurbished and miscellaneous e-waste, totalling 12.7 million lbs (5,748 metric tons).

Available data indicated that desktops, laptops, monitors, and printers accounted for 65% of the total tonnage of e-waste. The remaining 35% consisted of miscellaneous items such as parts, keyboards, and peripherals for which there was no need. It was observed that the largest amount of e-waste disposal was in 2021-22, accounting for 24% of the total over the evaluation period. The data also illustrated that 53% of e-waste was sourced from equipment donated by the federal government, compared with 47% of e-waste from organizations other than the federal government.

Over the 6-year period, expenses related to disposal of e-waste totalled $60,283.36, while profits generated from the sale of e-waste to recyclers totalled $511,148.21 Data relating to revenue and expenses of e-waste disposal were not reported by all Affiliates; of the 14 Affiliate organizations, 9 had data on both expenses and profits, 3 reported only one or the other, and 2 did not report data on this subject.

The CFS+ had many partnerships managed at different levels. Data gaps were noted relative to large-scale partners.

Partners support CFS+ by donating equipment, providing cash and in-kind contributions, and connecting clients with devices. CFS+ Affiliates and Partners spoke to the importance of the partnerships in achieving the program's goals. Some partnerships exist at the local level and are managed by the respective CFS+ Affiliates, while others exist at the national or provincial level and are managed by ISED, CFSC, and NTC. These are sizable partnerships whose operations span multiple provinces and/or territories (e.g., national retailers).

While performance data was available to quantify the partnerships at the local level, data regarding the number and types of large-scale partnerships was not available. CFSC estimated that the development and maintenance of corporate sector partnerships comprised one-third of the organization's work; however, no partnership data was reported in the annual reports to ISED. It was also noted during interviews that some of these large-scale partnerships had ceased over time, but without performance data, it was difficult to assess the extent to which this had occurred and the impact those partnerships may have had. The evaluation further found that partnerships with the provincial and territorial governments varied, with 1 CFS+ Affiliate noting they had no existing partnership with their provincial government and another Affiliate indicating the important role that their provincial government plays in the regional delivery of the program. This was identified as an area where the CFSC could play a greater role. The fluctuation in donors was also reflected in the donation data, with some Affiliates relying more on provincial donations, while others relying more on federal donations.

Federally, there is an opportunity to better leverage CFS+ when developing digital skills initiatives.

Section 4.2.16 of the Federal Government Directive on the Management of Materiel^{Footnote 35} specifies that CFS+ must receive first right of refusal on all surplus government computer equipment that cannot be relocated within the department. ISED works with departments to ensure they are aware of this obligation, while NTC works with departments in the National Capital Region to receive and refurbish the donated equipment. Some interviewees noted that there is an opportunity to better leverage the CFS+ when developing other government initiatives targeted to digital skills (for example coding for kids or digital literacy training). Since access to devices is key, some consideration should be given at the program development stage to the manner in which CFS+ could work with the programs to ensure they have the equipment they need to deliver their initiatives.

CFS+ program relies heavily on federal and provincial government donations of surplus computers and equipment.

Between 2016 and 2022, a total of 1,366,332 electronic devices were donated to the CFS+ program. Desktop computers comprised 45% of the total donations while laptops accounted for 24%. Donations were received from various types of organizations, with the federal government being the largest donor of all equipment, except for tablets and smartphones which were donated primarily by provincial/territorial governments. At the Affiliate level, donations from the provincial and territorial governments varied, with some regions like Quebec and Prince Edward Island receiving their largest share of devices from the province, while other regions like British Columbia and Ontario receive more from the federal government.

Computer donations generally increased from year to year, except for 2020-21 when the donations decreased across all groups. Interviews revealed that this was largely due to the COVID-19 pandemic, which shifted many to working remotely and led to the temporary closure of many workplaces. Some donors also experienced challenges during the pandemic, such as delays in getting new equipment, which impeded their ability to donate old devices.

Adequate supply of equipment donations is key to serving clients.

Interviewees noted there is a need for continued promotion of the program to maintain its profile with partners and ensure a steady stream of equipment donations.

The CEO pledge through CFSC was noted as an initiative that helped to increase donations. Launched in June 2021, 49 companies signed up within the first year, with an estimated 10,000 new devices being donated as a result.

The biggest challenge is not people needing devices, but rather having devices available to meet their needs.

The number of unique donors has increased, with the highest number of donors observed in 2021-2022.

From 2016 to 2022, a total of 3,230 unique donors were identified as having donated equipment to the CFS+ program. The number of unique donors may have been slightly overstated as there were some inconsistencies in how data was reported, leading to some donors being counted more than once (e.g., different spelling, or use of acronyms). Annually, the number of unique donors ranged from 482 to 1,105. While the number of unique donors fluctuated, it nearly doubled over the evaluation period. The CEO pledge, initiated in 2021, was a likely contributor in the last year.

There were inconsistencies in the donor-related data collected from the CFS+ Affiliates. Some donors were assigned different organization types over different years, and 7% of the data regarding organization type was absent. These inconsistencies make it difficult to assess the full impact of donors to the CFS+ and to possibly leverage their participation.

Most donors provided equipment once during the evaluation period.

The majority of donors (70%) provided equipment once during the evaluation period. While only 5% of donors provided equipment 6 times or more during the 6-year period, these more frequent donors were mostly large federal government departments or large corporations. Many interviewees noted the importance of continued outreach and promotion to ensure the program is not forgotten. The performance data reinforces the importance of ISED's role in ensuring that departments are aware of the CFS+. It also supports the efforts of the CFSC to secure increased numbers of large partners through the CEO pledge.

Contributions from various types of organizations represent an important source of funding for the CFS+ Affiliates.

During the evaluation period, CFS+ received a total of $30.4 million in contributions from various sources, which covered expenses such as office and warehouse space, volunteers, software donations, transportation costs, etc. A total of $15.1 million was received as cash contributions, while $15.3 million was received as in-kind contributions. Contributions were largely provided by provincial/territorial governments, who provided $8.8 million (or 29%). Non-profit organizations also provided $7.7 million in contributions (or 25%). At the Affiliate level, the contributions varied greatly, with 1 single Affiliate accounting for 50% of contributions during the evaluation period.

Contribution amounts varied and related performance measurement targets were absent.

Contribution amounts per year ranged from a low of $3 million in 2019-20 to a high of $6.7 million the year prior (2018-19), with an average of $5.1 million. While the COVID-19 pandemic impacted donations, it is difficult to assess the success on this indicator in the absence of an established program target.

Inconsistencies were noted in how data was reported, with some contributions being grouped together as "Various." The descriptions of contributions also varied between Affiliates, making it difficult to compare the different types of contributions (e.g., cost recovery). It should also be noted that there were gaps in the availability of data (missing data for some years, no data for CFSC).

The CFS+, supported by the CFSI, distributed an average of over 80,000 devices per year during the evaluation period.

Between 2016-2022, CFS+ Affiliates refurbished and distributed over 475,000 devices to beneficiaries. The largest number of devices was distributed by CFS+ in 2016-17, with lower numbers in subsequent years. In 2021-2022, CFS+ distributed the lowest number of devices during the evaluation period, at 62,509. The average number of devices distributed per year was 80,428.

While the number of devices distributed has been decreasing, it should be noted that these numbers do not include the approximately 50,000 devices distributed by the CFS+ Affiliates as part of the CFI during the last 2 years of the evaluation.

While desktops remain the most distributed device, there is growing demand for more mobile options.

The most distributed type of devices were desktop computers, representing 48% of total distributed devices. Laptops were the next most distributed device at 30%. The number of laptops distributed increased by 42% during the evaluation period, and 2021-2022 was the first year where the number of laptops distributed exceeded the number of desktops.

Evidence showed that there was a greater demand for laptops during the COVID-19 pandemic, which obliged CFS+ Affiliates to either restrict or limit customer orders based on available inventory, decline orders, or offer replacements or comparable devices to customers. The CFS+ Affiliates noted that there was a supply issue, as many had had too many desktops and not enough laptops.

CFS+ Affiliates were effective at meeting their distribution targets, with devices being distributed across the country.

The CFS+ program met the distribution target for 5 of the 6 years covered during the evaluation period. The exception was 2017-2018, when results were 1,089 below target. At the Affiliate level, progress against established targets has varied. The number of Affiliates meeting their annual distribution targets has ranged from a high of 12 in 2016-17 to a low of 6 in 2021-22. The activity in the latter years of the evaluation period was, however, impacted by the COVID-19 pandemic.

Amongst the CFS+ affiliates, Ontario distributed the largest number of devices to beneficiaries, representing 19% of total devices distributed, followed by Quebec, with 17% of total distributed devices. Figure 19 shows the distribution of devices across Canada for the evaluation period.

While the CFS+ Affiliates have been successful in distributing devices across the country, activity in the northern part of some provinces is low. Interviews revealed that this was due to the high costs of shipping devices to these regions, and suggested this could be alleviated by seeking more partnerships with shipping companies.

Schools were an important program beneficiary, but there was an increase in devices being distributed to others.

Among CFS+ beneficiaries, schools received 62% of total distributed devices, while not-for-profits received 28%.

While schools receive the largest number of devices every year from CFS+, the rate at which they are receiving devices is decreasing by an average of 12% annually, and in 2021-2022, almost half of all devices were distributed outside of schools (48%). Schools are increasingly supplying their students with devices as part of their curriculum and in response to COVID-19.

The socio-demographic group that benefited most from the program was youth, who received over 40% of total devices distributed, followed by the general population (16%). While eligible low-income Canadians accounted for only 5% of the total, they were only added as a target group in 2018-2019. Their numbers more than tripled from 2018 to 2022.

It was also noted that the number of devices going to low-income households was somewhat understated, as devices distributed to low-income families under the Connecting Families Initiative (CFI) program did not count towards the Affiliates' progress against targets.

Discrepancies in data reporting made it difficult to get a complete picture of socio-economic distribution. In 4% of cases, the group category was left blank, with a further 20% unspecified as other. It was unclear whether Affiliates lacked the correct category definitions to complete data reporting, or if the information had not been collected from beneficiaries.

Computer recipients were satisfied with the program and without access to devices would have experienced negative consequences.

The evaluation survey showed that the CFS+ program met the needs of computer recipients. Respondents were asked to rate the CFS+ program's ability to meet their needs, based on their most recent request. There was a high level of satisfaction, with the majority rating the program as Excellent (63.5%) or Very good (24.4%).

Had they not been able to access the computer equipment requested from CFS+ during their most recent request, the evaluation survey respondents reported there would have been many negative impacts. The most common impact would have been the inability of students to complete schoolwork (50.9%), followed by inability to participate in training (31.1%), social isolation (26.3%), inability to access government programs or services (24.4%), inability to search for jobs (21.7%), and inability to work (21.2%) (responses were not mutually exclusive).

The devices provided through CFS+ have a significant second life, with many being used for 2 to 3 years.

In order to assess the longevity of the equipment provided through CFS+, the evaluation survey asked participants how long the equipment they received remained in use. For computers, laptops, and monitors, between 42% and 47% were still in use at the time of the survey.

These results represent a significant second life for all 3 equipment types given that the government assets were originally used for 2 to 3 years prior to donation. Printers and other equipment were the exception, where respondents did not know the length of use. However, more than one-quarter of these assets were reported as still in use (27.0% and 27.7% respectively). When asked for a reason, those respondents who reported that the equipment remained in use for less than 1 year indicated that the equipment was inadequate (39.5%), that it had broken (25.6%), or that it was no longer required (16.3%). It should be noted that this represented a small proportion of total computer, laptop, and monitor devices.

Almost 300 youth per year gained experience and developed skills as a result of their internships through the CFSI.

Interns hired through the CFSI have been important sources of labour for the CFS+ Affiliates. Evidence showed that these internships provided youth with practical and technical experience, ICT skills, as well as critical soft skills, such as time management and good working habits. During the evaluation period, 1,674 youth interns were hired as part of the CFSI program, ranging from a low of 248 in 2018-19 to a high of 331 in 2020-21, with an average of 279 interns per year hired over the 6 years of the evaluation period.

The number of interns hired annually increased in 2020-2021 (up 63 from the previous year), which resulted from additional COVID-19 funding. Many CFS+ Affiliates interviewed noted that they could accommodate more interns but that they were limited due to the available base funding. Although the Affiliates have benefited from time-limited funding, the ad hoc nature of top-up funding posed challenges for the planning of human resources.

Quebec hired the most interns during the evaluation period (361 interns), followed by Ontario (293 interns). Quebec figures do not include interns hired by CFSC (114) and STLR (151), which are both in that province. Over the 5-year period, 440 interns dropped out of the CFSI program, almost 30% of which were in Ontario. One-third of the interns surveyed who did not complete the full internship indicated leaving because they had found other employment (35.4%). Affiliates noted that it was difficult to retain youth, partly due to labour shortages and the relatively low wages offered, which is a particular challenge in large urban centres. The retention issue posed challenges as Affiliates had to constantly recruit and train new staff.

While the program has implemented measures to increase the diversity of interns, intern diversity remains a challenge for some regions.

Data regarding the diversity of interns was only available from 2018 to 2022, while targets were only established in 2020-21. From 2018 onward, more than one-third of interns hired identified themselves as racialized (36%), which was higher than the 30% target for significant representation used for initiatives such as the 50-30 Challenge, and higher than the established targets from 2020 to 2022.

A total of 19% of interns identified as female, just shy of the 20% target established for the CFSI. This represents an improvement over the 15% noted during the last evaluation. Documents show that CFS+ has always had a low number of female interns, and efforts to reduce the gender gap have included removing the heavy lifting requirement displayed in recruitment posters. Some Affiliates also mentioned modifying their marketing strategy to attract more female youth. Other noted changes include removing post-secondary educational requirements, allowing part time hours for single parents, and covering transportation costs, such as bus passes. The evaluation found that diversification of participants in the CFSI program remains a priority.

Among the diversity dimensions measured for the CFSI, official language minority and persons with a disability both represented 6% of interns, while Indigenous interns accounted for 4%. The CFS+ Affiliate in the Northwest Territories had 43% of their interns identifying as Indigenous, the largest among all Affiliates. The largest percentage of interns identifying as a racialized person were in Manitoba (41%), Alberta (40%), and Ontario (38%). Many Affiliates noted that broadening the eligibility criteria to accept persons in Canada on a work permit could be an opportunity to increase diversity, as many had applied but were not eligible given that the focus of the program is to prioritize Canadian youth.

The evaluation found that some CFS+ Affiliates were successful in leveraging other programs in order to hire additional interns or bring in co-op students, although this was not reflected in the CFSI performance data. 2 of the CFS+ Affiliates interviewed noted using funding from other sources, in addition to CFSI, to hire persons with disabilities. 1 Affiliate indicated that based on their experience in providing internships to neurodiverse individuals, additional support was required, and that being able to expand the CFSI to provide peer mentors for interns with disabilities could be advantageous.

The CFSI program helped interns develop skills and secure employment, leading to careers in a digital technology field.

Interviewees noted that the program provides a good opportunity for youth to further their career in IT. The evaluation survey results showed that the CFSI helped increase the employability of youth.

In a survey, interns who were no longer in the CFSI program were asked to describe their activities after the program. Most interns reported that their experience with the CFSI program had a major (28%) or moderate (33%) influence on their career path.

The program provided interns with practical experience and increased their digital and career skills.

Interviewees agreed that CFSI was effective in supporting learning opportunities, skill development, and employability for youth. It provided hands-on experience working with computers, as well as some career skills.

The evaluation survey also suggested that there was potential to further help interns make contacts with potential employers. While a slight majority of the interns surveyed reported that the program put them in a better position to make contacts with potential employers (57.6%), almost 1 in 4 interns indicated this was not the case (22.9%), and the remainder were unsure (19.4%).

Interns agreed that the CFSI met their needs, but some pointed out the non-competitive wages and the short duration of internships.

The evaluation survey showed that interns were either very satisfied (42.8%) or satisfied (38.4%) with the application process, with 8 out of 10 interns rating it positively. While many interviewees noted that the internship duration was too short, almost half of the interns surveyed reported that the length of the internship was just right (48.1%), one-quarter indicated it was somewhat short (25.9%) and close to one-fifth indicated it was too short (17.3%).

While the CFSI did allow for interns to complete a second internship, this was the exception and not the norm. Some CFS+ Affiliates interviewed thought there should be more flexibility with respect to duration to allow for longer internships, noting that in some instances the duration of the term should be commensurate with the needs of the individual intern.

Interns were less satisfied with the hourly wage, with only 50% of interns surveyed reporting they were satisfied (32.3%) or very satisfied (18.6%). Slightly less than half of the interns either agreed (30.6%) or strongly agreed (16.3%) that the internship salary was competitive with similar jobs in their local labour market, while more than one-quarter of interns either disagreed (16.9%) or strongly disagreed (10.0%). CFS+ Affiliates noted that offering competitive wages was a challenge, as there was a single amount of funding allocated per intern, and offering a higher wage meant reducing the duration of the internship.

Interns felt that more could be done to help improve access to professional development and skills acquisition.

Overall, interns surveyed were satisfied with the CFSI program, with the majority strongly agreeing (46.9%) or agreeing (36.9%). Most also strongly agreed (66.7%) or agreed (22.0%) that there is a need to continue providing this kind of experience to youth, and further agreed that they developed new skills through their experience with the CFSI program (38.3% strongly agreed, 41.3% agreed).

Interviewees noted that for some youth, the CFSI internship is often their first job after leaving high school or post-secondary. In some instances, Affiliates can offer the interns additional training, for example resume-building and interviewing skills; however, access to this type of training across Affiliates is not consistent. One Affiliate noted that this is a balancing act as the interns are there for a limited period, and the CFS+ relies on them for its operations. Similarly, if an intern wants to take a course, it can be covered through CFSI, but this reduces the amount of funding available to support the internship. Many of the Affiliate officials interviewed (60%) noted that the funding limit impedes their ability to provide training.

Interviewees noted that, with the growing distribution of devices to non-school partners, it might be possible to extend intern activities to help with setting up equipment, networking, or providing training to new users, thus expanding the skills to be acquired. While CFS+ Affiliates did aim to provide the interns with varied experiences and expose them to various aspects of the process—e.g., sorting, refurbishing, shipping—a few interviewees noted that the work can get monotonous and that some technical work mostly involved clicking through pop-up menus and installing software. Interviewees and Affiliates also noted that they received many types of requests from partners for assistance that they are unable to accommodate, such as providing computer use and related training workshops to recipients.

3.3 Efficiency

Both programs demonstrated operational efficiency; however, variations exist based on the different delivery models.

The evaluation found that the program delivery models were operating efficiently. Base funding for both programs has been unchanged since 2016, with many CFS+ Affiliates exceeding their targets despite the pandemic and rising operational costs due to inflation; some interviewees questioned whether this was sustainable. Among Affiliates, the difference in how CFS+ is delivered varies according to regional needs, with different regions having different delivery models (e.g., number of locations, size of partner network, participation of provincial or territorial governments, use of volunteers, etc.).

The cost per device distributed varies greatly, as illustrated in Figure 26 (the cost per device in Nunavut is excluded due to data gaps). The overall cost at the program level is $99. This is lower than the cost per device noted in previous evaluations ($105), suggesting efficiency has increased. While the cost per device varied, 57.14% of Affiliates had a cost per device of $100 or less. The Affiliates operating in Northern Canada (Yukon, NWT, and Nunavut) had higher costs per device, mostly due to higher shipping costs. No correlation was observed between the cost per device and average annual volume of devices distributed. However, it was noted that the 2 Affiliates serving the NWT and Yukon had the lowest annual volume of distribution (less than 1,000) and the highest cost per device ($396 and $347 respectively).

While partner contributions were significant, it was difficult to gauge success on this measure in the absence of a metric.

Partner funding, cash contributions, and in-kind contributions increase the program's reach, but no metrics were in place to assess its extent (leverage ratio). During the evaluation period, ISED funding for both programs totalled $70.3M, while recipients reported contributions of $30.4M and profit from disposals representing $511K (some data was not available). This represented a leverage ratio of $0.48 for every ISED dollar invested at the program level. Looking at leveraged funding at the Affiliate level, the leverage ratio ranged from $0.21 to $1.43. Quebec had the highest leverage ratio, driven largely by partner contributions ($15.4M). Most partner contributions were cash ($11.1M or 73%), and the largest share of contributions came from not-for-profit (41%). The largest share of NFP contributions was from Ordinateurs pour écoles du Québec (OPEQ) and covered administrative costs (salaries, telephone, rent, transportation). PEI had the second highest leverage ratio (1.39), which was also driven by partner contributions, all of which ($1,786,351) came from the provincial government for this Affiliate. The evaluation encountered data quality issues, with the same contributors being reported in different categories across different fiscal years.

CFS+ administrative costs were lower than forecasted, and measures were implemented to minimize lapses in G&C funding.

G&C spending was lower than expected in 2016-17 and 2017-18 as the CFS+ received top-up funding late in the year, leading to lapsed funds. The variance for G&C spending for the remaining years was minimal and represented 1.0% or less of the budget.

Following the lapsed funding in 2016-17 and 2017-18, ISED implemented a cost-forecasting tool to help mitigate the risk of lapsing funds, and a financial outlook forecast exercise conducted at the end of Q3 to reallocate funds and avoid lapses. Previously, ISED would often get notified late in the fiscal year that an Affiliate would be lapsing funding without sufficient time to reallocate. By canvassing Affiliates in Q3, they were proactive in forecasting unused funds and reallocating.

At the program level, total ISED administrative costs for the CFS+ were forecasted to be 12.3%, based on the original total funding amounts allocated in the fiscal framework.

As of March 2022, the actual administrative costs were 9.5%, with the average actual administrative costs remaining fairly consistent over the evaluation period.

For CFSI, the administrative costs were lower than forecasted and there were challenges with spending the full G&C budget.

CFSI and CFS+ were supported by the same administrative staff and these costs were proportioned to each program for reporting purposes. Some additional operating costs not forecasted were incurred in the first 2 years of the evaluation period, for the translation of reports and documents and the hiring of a consultant to revise the reporting tools.

Total ISED administrative costs for the CFSI were forecasted to be 6.6% of the total program budget allocated in the fiscal framework. As of March 2022, the actual administrative costs were lower than forecasted (4.7%) as additional funding was provided to CFSI in response to COVID-19, leading to a lower proportion of administrative cost to total budget.

For the last 2 years of the evaluation period, significant G&C top-up funding for CFSI was received, representing more than double the original CFSI budget.

Except for 2018-19, G&C spending was lower than available funding for all years, in part due to the timing of funding being received by Affiliates and the delays and challenges related to the hiring of interns such as COVID-19, roll-over of interns, time required to hire and train, as well as labour availability in some regions.

Measures were implemented at the program and Affiliate levels to increase operational efficiencies.

Efficiency measures were introduced by the ISED program and at the Affiliate levels. The annual general meetings of the CFS+ Affiliates, hosted by ISED, was identified as a mechanism where promising practices were shared to help increase efficiency.

Nationally, CFSC implemented the CEO Pledge to help increase donations from large-scale partners, and facilitated bulk purchasing of operating system licenses, which lowered the cost per licence for the Affiliates. The NTC also worked on surveying federal departments to gauge the incoming inventory to help with planning.

Some Affiliates continue to face challenges to operational efficiency specific to the jurisdictions they serve.

Some Affiliates noted challenges to operational efficiency that were specific to their regions. For instance, 1 Affiliate mentioned that in their province, legislation for reuse and recycling changed twice since 2016 and this resulted in the need for more security and more information regarding the assets for donors and governments. These were unanticipated changes which needed to be accommodated with existing funding levels.

Efficiency was noted as a challenge for Affiliates serving larger regions. The fewer the number of service points, the greater the challenge in providing geographic reach and distribution and in securing interns, not to mention the additional costs for shipping. However, the increased shipping costs are generally lower than the cost of operating additional centres.

Affiliates noted that national organizational partnerships and initiatives (bulk purchasing) could increase efficiencies, expand geographic reach, and reduce costs, particularly for distribution.

The Government of Canada was able to leverage the CFS+ Network to successfully deliver the CFI.

While the Connecting Families Initiative (CFI) was not included in the scope of the current evaluation, the evaluation did seek key informant views on the efficiency of delivering the CFI through the CFS+ program. The number of computers to be distributed was significant (50,000), and the refurbished devices had to be delivered within a short timeframe. The CFS+ Affiliates were able to deliver devices, but the exact numbers were not reported through CFS+.

Affiliates generally agreed that CFI was a great success, and many pointed out that receiving a device and an Internet connection can expand opportunities for these families. CFI helped to demonstrate the agility of the CFS+ Affiliates and Network. They were able to quickly pivot and adapt their operations to meet the needs of individual families and have the program delivered across the country.

Thanks to the CFI, ISED was able to leverage the CFS+ Network to help meet a Government of Canada priority of promoting the inclusion of under-represented groups in the digital economy. This was the second time the CFS+ program has been leveraged to help reach a dedicated group, the first being a smaller-scale initiative launched in 2015 that provided over 7,500 computers to Syrian families.

Fulfilling the CFI meant Affiliates needed to adjust their processes to accommodate distribution to individual households. The CFI placed some pressure on the delivery models as individual orders took longer to process, which impacted on productivity. In recognizing the additional stress that the initiative placed on CFS+ Affiliates, CFSC developed a Shopify platform to manage demand and orders for the Affiliates. Some Affiliates expressed disappointment that the devices distributed through CFI were not counted as part of the CFS+ progress reports, as in some regions thousands of devices were distributed through this program. The reporting of devices through CFI only did not paint an accurate picture of support provided by CFS+.

4.0 Conclusions

Relevance:

The evaluation led to 3 findings demonstrating that there is a continued need for the CFS+ and CFSI

Finding 1: There is an ongoing need for the Government of Canada to support Canadians through CFS+ in obtaining the necessary tools and skills to effectively participate in the digital world.

Finding 2: The CFSI program supports youth in developing important skills and gaining practical work experience to increase their employability, which aligns with federal government priorities.

Finding 3: The CFS+ and the CFSI have been effective in reducing the environmental footprint associated with surplus equipment. Through the refurbishment, reuse, and recycling of electronic devices, a significant number are being diverted from landfills and given a second life, leading to environmental benefits.

Performance:

The evaluation led to 5 findings which demonstrated that the programs are meeting established objectives, but opportunities exist for improved data collection and program enhancements.

Finding 4: The CFS+ and the CFSI have been effective in reducing the environmental footprint associated with surplus equipment. Through the refurbishment, reuse, and recycling of electronic devices, a significant number are being diverted from landfills and given a second life, leading to environmental benefits.

Finding 5: The CFS+ has developed a large network of partners that increase the reach and resources available to the program and are integral to its success. Inconsistencies and gaps in performance data related to partnerships and donations were identified.

Finding 6: The CFS+, supported by the CFSI, have been effective in increasing access to computer technology for Canadians. Program beneficiaries have been shifting towards non-school groups and individuals. Beneficiaries were satisfied with the CFS+ program and would have experienced negative consequences in the absence of the program.

Finding 7: The CFSI provided learning opportunities and enhanced employability for diverse youth. Interns developed digital skills and secured employment in a technology field. Helping interns to connect with potential employers was identified as an area of opportunity, while broadening eligibility was identified as a way to improve diversity.

Finding 8: Interns expressed high satisfaction with the CFSI program, although some issues were noted concerning the internship duration and the competitiveness of the wages. Opportunities for improvement included more support for access to training and skills acquisition.

Efficiency:

In assessing the efficiency of program delivery, the evaluation led to 2 findings which highlighted that both CFS+ and CSI were delivered efficiently and that the CFS+ could be successfully leveraged to support other federal initiatives.

Finding 9: The evaluation found that the CFS+ and the CFSI were delivered efficiently, with delivery models varying across the country. At the program level, administrative cost ratios were lower than forecasted due to additional G&C top-up funding. Both ISED and the CFS+ Affiliates were found to have implemented measures to improve the program's operational efficiency.

Finding 10: The delivery of the Connecting Families Initiative through the CFS+ Network has demonstrated the agility with which the CFS+ can be leveraged to help further targeted priorities. The CFI addressed gaps that individuals faced in accessing the CFS+ Program.

The evaluation identified 2 recommendations.

Recommendation 1: ISED Connected Canada Branch should implement approaches to enhance the consistency of data submitted by CFS+ Affiliates. The availability of consistent and complete data for CFS+ in the areas of donor types, contributions, and device distribution by socio-economic group would ensure progress against CFS+ expected outcomes and position the program to continue responding to the evolving technology needs of Canadians.

Recommendation 2: ISED Connected Canada Branch should investigate further opportunities to increase donor contributions and partnerships to meet program needs, expand the reach of the CFS+ program, and support operational efficiencies for CFS+ Affiliates.

Appendix A: Logic Model

Through the eligible activities funded under the ISED funding agreement, the CFS+ and CFSI programs are expected to achieve the results in the logic model below, along with related short-term, medium-term and ultimate outcomes. The logic model was derived from the CFS+ program's Performance Measurement Strategy submitted to ISED in 2016.

Appendix B: Methodology

The evaluation was based on 5 data collection methods, including qualitative and quantitative sources. In order to answer the evaluation questions, information from multiple lines of evidence was collected and triangulated.

Literature and Document Review

The literature review was comprised of pertinent literature to gain a thorough understanding of the current and continued need for CFS+ and CFSI from the perspectives of increasing access to technology to Canadians, providing employment opportunities to diverse youth, and addressing the issue of e-waste. The document review included key government priority setting documents and program reporting documents to support the assessment of performance and efficiency.

Performance, Administrative and Financial Data Review

The CFS+ and CFSI performance data, collected as part of the CFS+ program's Performance Measurement Strategy, was reviewed in order to assess the extent to which progress has been made towards achieving the short-term, medium-term, and long-term outcomes outlined in the CFS+ logic model. An analysis of the administrative and financial data for CFS+ and CFSI was also performed to assess efficiency.

Case Studies

3 case studies were conducted to support the overall evaluation of CFS+ And CFSI. The case studies examined 2 CFS+ Affiliates and 1 National CFS+ Organization. The case studies relied on document review, administrative data, and a total of 3 interviews with management staff of the 3 CFS+ organizations.

Online Surveys

Online surveys were conducted to gather perspectives on the impacts of CFS+ and CFSI, in addition to the potential improvements to enhance the programs' effectiveness. The surveys targeted device recipients and current and past interns who participated in the CFSI. Surveys resulted in responses from a total of 850 recipients and 172 interns.

Virtual Interviews

A total of 16 interviews were conducted using either MS Teams or by telephone across the following stakeholder groups to gather diverse perspectives on the relevance, performance and efficiency of CFS+ and CFSI:

• ISED management and staff
• Other government departments
• National CFS+ Organizations
• CFS+ Affiliates
• Network Partners
• CFS Interns

Appendix C: Challenges and Mitigations

The evaluation encountered 3 limitations and evaluators applied related mitigation strategies.

1) Attribution

Challenge:

The presence of other funding partners made isolating and measuring the direct impact of the federal government's contribution challenging.

Mitigation:

Interview questions were designed in a way that respondents could answer, to the extent possible, the incremental impact of ISED's funding to CFS+ and CFSI.

2) Respondent Bias

Challenge:

Many interview participants were either involved in program delivery or direct beneficiaries and, as a result, responses may have been positively biased.

Mitigation:

The purpose of the interview and its strict confidentiality was communicated to participants. Responses were validated across stakeholder groups as well as other lines of evidence.

3) Data Quality

Challenge:

Self-reported data collected quarterly by CFS+ from Affiliates is the basis of the program's Performance Measurement Strategy. During the evaluation, data quality issues were encountered, such as inconsistent labeling of donor organizations and primary target groups.

Mitigation:

Where data was unavailable, other lines of evidence were leveraged to confirm or support evaluation findings.

Appendix D: Bibliography