Evaluation of the Digital Research Infrastructure (DRI) Strategy

Program context

The Digital Research Infrastructure (DRI) Strategy was created in 2018.

Digital Research Infrastructure (DRI) is an essential foundation for scientific and academic research. It is the collection of computing infrastructure (tools and services) required to analyze significant amounts of data to produce new scientific insights. A growing number of researchers rely on DRI to open new scientific possibilities and conduct leading-edge research.

The Canadian DRI system comprises four pillars:

1. Digital network that allows researchers to send data between institutions and around the world.
2. Advanced Research Computing (ARC) or supercomputers used for big data analysis and simulation.
3. Research Software (RS) that allows researchers to access, share and utilize data.
4. Research Data Management (RDM) ensures the management, publication and preservation of datasets to promote data sharing, security, accessibility and reuse.

There are two national organizations that provide support to the four pillars.

CANARIE is a non-profit corporation that was established in 1993. It is the national organization responsible for coordinating the digital network for research in partnership with regional advanced networks and post-secondary institutions (pillar #1). CANARIE's core purpose is the advancement of Canada's knowledge and innovation infrastructure, while its mission is to design and deliver digital infrastructure and drive its adoption for research, education, and innovation.

The Digital Research Alliance of Canada (DRAC) has been in operation since 2019-20 and addresses pillars #2-4 (ARC, RS and RDM services). Services involving ARC, RS, and RDM all require almost daily interfaces with researchers and high levels of personal assistance to optimize their effectiveness as tools. Consolidating these functions within one organization aims to provide a holistic view of the researcher-facing pillars of the DRI system and the development of a strategic national roadmap for developing efficient and coordinated services.

The DRI Strategy is making investments to achieve the following objectives:

• Fund DRAC to consolidate and oversee the delivery of ARC, RDM and RS for Canada's academic research community.
• Streamline access and deliver more open and equitable access to advanced computing and big data resources for researchers across Canada.
• Improve the availability of more digital tools and services and increase the optimum use of DRI resources.

Governance

CANARIE's governance structure includes a Board of Directors that is comprised of 15 representatives from Canada's research, education, and innovation communities. Directors are elected for one-year renewable terms, and all Directors serve on at least one standing committee of the Board. CANARIE Board members are involved in planning strategic initiatives, reviewing and approving initiatives that set CANARIE's direction within funding mandates, making final decisions on programs to be funded, and approving corporate policies and overseeing compliance.

DRAC's 17 representatives on their Board of Directors oversee the management of the organization's activities. Directors are elected for renewable terms of up to three years. Directors are recruited based on an established skills and competency model, which includes knowledge of the researcher community; expertise in organizational governance, management, planning or oversight; as well as expertise in at least one of the DRI pillars (ARC, RDM, and RS).

Target population

CANARIE's primary target population is the research and education (R&E) community in Canada, including researchers and scientists working at universities, colleges, research hospitals, and non-profit research institutions. DRAC awards funding and/or services to ultimate recipients, which includes:

• Institutions or organizations situated in Canada capable of carrying on meaningful research including a university, post-secondary college, or hospital;
• Other corporations with a mandate in ARC, RDM or RS; and
• Provinces or territories implementing a project or activity to support ARC, RDM or RS.

Program funding

Budget 2018 committed $572.5M to the DRI Strategy to ensure that Canadian researchers have the digital tools they need to support scientific excellence, which included $375M towards DRAC, $50M to Host Site institutions towards the ARC Expansion Program through a separate Contribution Agreement directly to the Host Site Institutions, and $145M towards CANARIE (from 2019-20 to 2023-24). An additional $37M was allocated to CANARIE via a Contribution Agreement amendment for 2024-25.

Evaluation context

An evaluation of ISED's funding to CANARIE and DRAC via the DRI Strategy is required in accordance with the Financial Administration Act.

The objectives of this evaluation are to examine the relevance, performance and efficiency of the DRI Strategy. The evaluation also examined the implementation of DRAC and assessed early findings for DRAC's intermediate outcomes.

The scope of the evaluation includes all ISED funding to CANARIE and DRAC during the period from April 1, 2018 to March 31, 2023.

The evaluation was conducted in-house by the Audit and Evaluation Branch at ISED. A results-based approach was used to examine the achievement of expected outcomes, as identified in the logic model (Appendix A).

Evaluation questions

Relevance

• To what extent is there a demonstrable need for Canada's DRI Strategy?

Implementation (DRAC only)

• To what extent has DRAC been implemented as planned? Are there lessons learned to date? (this will include an assessment of DRAC's short-term outcome of a national non-profit DRI governing corporation with clear roles and responsibilities)?
• To what extent is the performance information available through DRAC effective in helping assess the success of DRAC?
• To what extent are relationships between and among DRI program stakeholders effectively contributing to its implementation?

Performance

• Capacity building: To what extent is the DRI Strategy contributing to increased technological capacity and access to networks, tools, services, and platforms within the DRI ecosystem?
• Cybersecurity: To what extent is the DRI Strategy contributing to a more secure DRI ecosystem?
• Recognition and adoption: To what extent is the DRI Strategy contributing to increased awareness and adoption of the platforms and tools within the DRI ecosystem? To what extent are end users aware of DRI services that are available to them?
• Skills development: To what extent is the DRI Strategy contributing to Canada's researchers having improved skills and enhanced opportunities for collaborative knowledge creation and innovation?

Efficiency

• To what extent are the resources allocated to the DRI Strategy being utilized in an economical manner in producing outputs and progressing towards expected outcomes?
• To what extent are DRI funding arrangements appropriate in supporting effective service delivery?

Evaluation methodology

Multiple lines of evidence were used to address the evaluation questions (see Appendix C for more detail.)

• Document review
• Data review
• Case studies
• Virtual interviews

Findings

Relevance

Unique and continued need

Need for DRI is increasing

The increased complexity and size of datasets and computational workflows has resulted in a growing demand for DRI infrastructure. Interviewees and the document review revealed that the need for DRI is increasing, both for non-traditional sectors, and especially for data-intensive areas such as AI. For example, a 2023 OECD report notes^{Footnote 1} that the computational capabilities required to train modern machine learning systems has grown by hundreds of thousands of times since 2012. According to DRAC's 2021 Current State of Advanced Research Computing in Canada, registered users, CPU resource usage, and GPU resource usage respectively increased by around 12%, 16%, and 56% per year in the previous decade (Figure 1). The demand for these resources is also accelerating rapidly, particularly for GPU resources, which grew over twenty-fold between 2012 and 2019.^{Footnote 2}

The growth at which data is generated is also increasing (e.g., data traffic on the CANARIE network grew by 131% from 2015 to 2020). According to documents and interviewees, the rate at which technology is changing, the high costs of DRI, and the lack of capacity among institutions to support highly qualified personnel (HQP), makes it challenging for them to manage DRI on their own. High Performance Computing (HPC) systems typically have a usable lifetime between three and five years before they are no longer competitive or cost effective. Close to 80% of the national HPC and community Cloud infrastructure are in immediate need of renewal to avoid major service disruption.^{Footnote 3} As such, having centralized access to DRI resources contributes to a more equitable, effective, and efficient ecosystem.

The speed of technology [advancement is] impacting the way DRI is architected, deployed, [and] operated. There is an exponential growth in the data that's being created and how you derive value from data [has] changed significantly.

Figure 1: Growth in demand for DRI

DRI ecosystem is large and complex

There is a very large scope and scale of stakeholders in the R&E ecosystem, including service providers, research and post-secondary institutions (researchers, HQP, students, administration, etc.), NREN partners, host sites, regional partners, and other funding providers (e.g., Tri-Agencies and provinces and territories) (see Appendix G). Initially, the DRI's four pillars were divided across five organizations, which led to gaps and overlapping responsibilities between pillars, a lack of ability to coordinate and address researcher needs across sectors, and did not address the continuum between data, software, networking, and computing. Interviewees said the DRI ecosystem is more difficult to govern and there are more interoperability issues if there are multiple entities delivering services and infrastructure.

The DRI Strategy consolidated responsibilities which had been fragmented and overlapped

To address DRI needs in a coordinated and efficient manner, the Government of Canada created the DRI Strategy in 2018, which provided $572.5M, with $52M per year ongoing, to deliver more open and equitable access to advanced computing and big data resources to researchers across Canada. The consolidation of key elements under DRAC (see Figures 2 and 3), which was completed by March 2022, aimed to provide researchers with better-integrated services and a single-point of entry for DRI needs. It was said that a coordinated, national strategy that guides investments in DRI is needed to avoid inefficient, uncoordinated investments, and provide sufficient computing resources to keep pace with other countries' investments. Interviewees highlighted that a national approach leads to economies of scale, reduces siloes and duplication, enhances interoperability among institutions, and increases sharing of resources and collaboration, domestically and internationally.

This aligns with other jurisdictions, including long-standing strategies such as the European Strategy Forum on Research Infrastructures, established in 2002, and Australia's National Collaborative Research Infrastructure Strategy, established in 2004, as well as more recent strategies, such as the U.S. government's 2021 National Strategic Overview for Research and Development Infrastructure and the UK government's 2021 DRI Programme.

There is collaboration between DRAC and CANARIE in addressing the needs of the ecosystem

According to interviewees, DRAC and CANARIE are unique and complementary, communicate and work effectively together, and have clarity in their respective roles and responsibilities. It was said that DRAC and CANARIE's technologies are also distinct, so even when there is some overlap, a large functional separation exists between the services. The DRI Strategy aimed to improve coordination in the ecosystem. To this end, CANARIE and DRAC produce a Joint Ecosystem Plan and Report each year, laying out how the two organizations work together on ecosystem governance, ARC, RDM, RS, networking, and cybersecurity. CANARIE and DRAC also developed a Joint EDI Strategy, as well as other joint DRI initiatives. However, some interviewees identified opportunities to work more collaboratively together, such as collaborating on joint funding projects or initiatives.

Figure 2: DRI ecosystem pre-consolidation

Figure 3: DRI ecosystem post-consolidation

CANARIE has been effective in meeting needs

CANARIE services are supported by the 13 provincial and territorial partners that comprise Canada's National Research and Education Network (NREN) (Figure 4). Each NREN partner connects directly to the institutions (universities, colleges and research facilities) in their region to ensure researcher needs are addressed.

Interviews indicated that CANARIE has been effective in addressing its core mandate and the needs of the DRI ecosystem. This included international network connectivity, via working with organizations to understand network demands, developing HQP to handle emerging needs, and examining network traffic to forecast needs. Users indicated that CANARIE has provided increased access and met their needs by providing consistent support and funding, as well as flexibility in its services.

CANARIE is viewed as essential in our community and a well-managed element of the Canadian DRI ecosystem.

They've always done a tremendous job in what they've provided. The networks have grown in capacity to serve more players and users [and] they've added different services to make sure that everyone is enabled with the tools that they need.

Figure 4: Canada's National Research & Education Network (NREN)

DRAC consolidated services, but further work is needed

According to interviewees, DRAC provides a more cohesive, integrated, national perspective on DRI-related needs. Similar to CANARIE's NREN, DRAC coordinates with five regional organizations as well as five host sites (Appendix E) to provide access and support to the computer resources (ARC) required by researchers across Canada (Figure 5).

Interviewees said DRAC has been meeting researcher needs in the maintenance of existing services (e.g., access to ARC resources), but has not provided many enhancements (e.g., ARC infrastructure and RS investments), due to implementation challenges as well as its short duration of existence (established in 2019) as compared to CANARIE (established in 1993). Most activities have been focused on implementation and transitioning of services. While the services have all been consolidated under DRAC, integration of these consolidated services (e.g., DRAC's three DRI pillars) is an ongoing process.

DRAC is still in the formative stages of development...We're all struggling to make up for the very significant setbacks and delays [and as a result] Canada continues to fall behind in capacity and capability.

Figure 5: DRAC's Advanced Research Computing (ARC) Platform

Cybersecurity is a significant risk area in the R&E sector

Cybersecurity is a high priority in the R&E sector, given the highly collaborative, decentralized, and interconnected nature of the work, with many users. The pace of research and technological evolution is leading to increased cybersecurity risks, with cyber threats and attacks becoming more common. As Canada's research institutions continue to increase the amount of data (e.g., IP) they collect and share through the DRI ecosystem, the ability to keep this data safe from theft becomes more important. Disruptions can result in catastrophic loss of research data, reputational damages and impact operations. R&E institutions also hold a large amount of data valued by cybercriminals. The approach to cybersecurity is also fragmented, there is a lack of overall accountability for cybersecurity, and the maturity level of institutions' cybersecurity programs vary.

CANARIE established the Cybersecurity Initiatives Program (CIP) to increase alignment of the R&E community on cybersecurity, address priorities and gaps, and to advance the cybersecurity capabilities of the ecosystem. In 2022, Canada's Shared Security Operations Centre was integrated into CANARIE's cybersecurity programming to strengthen the coordinated national approach to cybersecurity and CANARIE began planning/consultations on the development of a federated security operations centre. To increase consistency and standardization and reduce exposure to cyberattacks across the R&E sector, DRAC developed a cybersecurity framework, with layered security controls for technology, processes, and people to detect, prevent, and respond to cybersecurity events.

DRAC and CANARIE's cybersecurity activities are aimed at reducing cybersecurity risks in the R&E sector

Figure 6

Increased collaboration and coordination on cybersecurity is needed

There is collaboration between CANARIE and DRAC via executive membership in each other's cybersecurity committees, they have regular touch points, and share plans. The Contribution Agreement required DRAC to collaborate with CANARIE on the Cybersecurity Framework and so DRAC sought CANARIE's feedback and approval on the Framework. While DRAC and CANARIE meet regularly, share documents, and provide status updates, interviewees believed that awareness of CANARIE's cybersecurity activities could be improved. It was felt that it would facilitate alignment and avoid duplication between the two organizations.

There were moderate concerns among some interviewees that for certain cybersecurity activities, it had not been delineated whether CANARIE or DRAC is the lead, although the CA specifies responsibility on a few elements (CANARIE for the Joint Security Project and DRAC for the Cybersecurity Framework). It was also said that as CANARIE and DRAC grow and expand their cybersecurity mandates, it's possible for overlap, which has yet to occur because the scope of cybersecurity is so large. However, once a foundation has been built, it was said that the cybersecurity infrastructure for the two organizations needs to be more integrated because it would be detrimental to have two isolated systems in the long-run.

Implementation (DRAC only)

The first four years focused on establishing the organization, planning, and transitioning responsibilities

• 2019-20: year 1 was focused on standing-up the organization, consulting stakeholders (of which researchers are the key stakeholders), creating the governance and membership model, and establishing the Board of Directors, Researcher Council, and management and administration.
• 2020-21: year 2 focused on the development of strategic planning and transition documents and on the assessment of researcher needs and the DRI landscape. The CANARIE Transition Plan was completed in January 2021, the Compute Canada Transition Plan was delayed from January 2021 to April 2021, and the Strategic Plan was delayed from December 2020 to September 2021.
• 2021-22: year 3 was focused on transitioning services to DRAC and the development of the National Service Delivery Model (NSDM), Funding Model (FM), and Multi-Year Funding Proposal (MYFP). DRAC completed the transition of RDM from the Canadian Association of Research Libraries, finalized the transition of RDM and RS from CANARIE, and ARC responsibilities from CFI and Compute Canada for March 31, 2022.
• 2022-23: year 4 was focused on the transition of the Federation to a new service delivery and funding model. The NSDM, FM, and MYFP were completed three months behind schedule, in September 2022. The NSDM and FM Implementation Plan was delayed by six months to the end of March 2023. As a result of delays for NSDM, FM and MYFP, several milestones reliant on their approval were at risk of not being completed on time. The development of the DRI service classification model was also pushed to the 2023-24 fiscal year.

Implementation took longer than planned

As of 2022-23 (Q3), DRAC reported that it had completed over one-quarter of all its eligible activities (its NSDM, FM, and MYFP were late). There were 9 (27%) deliverables that were "at risk" (NSDM and FM Implementation Plan, DRI service catalog, Training Requirements List, user data Management Plan, and five cybersecurity deliverables) and 4 (12%) deliverables that had not started.

DRAC officials said there was an underestimation of the complexity, time, and budget required to implement and transition services. Most stakeholder groups interviewed said that it was very difficult to transition the decentralized delivery model to a more centralized, national model and upgrade services at the same time, while also being dependent on matching funding. Some DRAC officials, recipients/users, and DRI players said DRAC struggled with change management and there was a lot of effort in the first few years trying to work within the legacy systems/processes that were already in place under the previous organizations, which hampered its ability to effectively provide funding and work with its service delivery partners. Other stakeholders interviewed noted that underestimation of the time required to implement third-party organizations is a recurring challenge that needs to be better addressed.

These challenges were exacerbated by COVID-19, which slowed down consultations and implementation. Specifically, in July 2020, in consultation with ISED, DRAC updated some critical dates in response to the effects of COVID-19, and to allow for more time to engage and consult DRI communities.

DRAC consulted stakeholders to inform its design and identify needs

To inform membership and governance structure, DRAC held six in-person and five virtual meetings (November 2019), where around 220 members of the DRI community participated.

For the Current State Assessments, DRAC had four working groups and consulted (throughout 2020 and 2021) with 30 participants, who represented 21 organizations.

For the Needs Assessment (Figure 7), 112 organizations submitted position papers on DRI issues (October 2020), 1,380 researchers were surveyed (February 2021), and 8 town halls were held (May 2021).

For the Strategic Plan (Figure 8), DRAC had 16 consultations with 120 participants (2021-22), and the results of the needs assessment and current state assessments also fed into the strategic plan.

Multiple groups identified concerns with engagement and collaboration

Despite the consultations noted above, multiple stakeholder groups interviewed felt that a complex, heavy organizational structure was implemented in a top-down manner and without adequate consideration of stakeholder needs in the design and delivery of the programming. They perceived that there was insufficient engagement, communication, and/or collaboration with researchers/partners.

Some also said that DRAC has been less collaborative and engaging in their day-to-day interactions with researchers than the previous DRI organizations they dealt with prior to consolidation. Some interviewees also noted that there were challenges with collaboration at the leadership level, noting that there was a deterioration in engagement levels.

As a result, interviewees said there was a need to improve the clarity of roles and responsibilities between DRAC and funding recipients and partners, develop more effective mechanisms to support effective coordination and collaboration with institutions and regional partners, and provide more direct and meaningful involvement and consultation opportunities for the research community. Some DRAC officials interviewed acknowledged that there's still more work to do in terms consulting stakeholders and making adjustments to better meet stakeholder needs.

Figure 7: Needs Assessment consultations

Figure 8: Strategic planning process

Further clarity of governance, roles, and responsibilities is needed

The governance of DRAC is made up of the following core components: the Board of Directors, the Members, and the Researcher Council.

• The Board of Directors oversees the governance and management of DRAC.
• DRAC members support and guide the work and governance of DRAC.
• The Researcher Council advises DRAC on researcher needs, strategic vision, and priorities.

DRAC officials noted that there was effective governance, clear strategies, engagement mechanisms, and relationships, connected via a federated model with committees and working groups. However, it was said that DRAC's national leadership role is not yet well understood by stakeholders because of the historically decentralized nature of the pre-existing Federated model.

Recipients/users and DRI players said that while DRAC is now operating more effectively, and improvements were noted as a result of changes in organizational leadership, initially there was a lack of clarity in DRAC's structure and purpose, implicated stakeholders, strategic direction, and implementation plan. Further, it was felt that there is still not adequate clarity on the structure of DRAC, its oversight, or its governance.

Some funding recipients found that certain roles and responsibilities between DRAC and funding sources are still unclear. Specifically, interviewees said that further clarity was needed in the aspects of DRI funded by DRAC versus other DRI players (e.g., CFI, granting councils, P/Ts, and institutions). Although funding agreements were developed in collaboration with all recipients of DRAC funding between December 2022 and April 2023, some interviewees still had questions about funding terms and conditions. They wanted clarity on how DRAC's distribution of funding to institutions is determined, the role of regional organizations, and the rules/requirements for in-kind contributions. As such, there is a need for DRAC to enhance funding recipients' level of understanding by further clarifying roles and responsibilities and improving communication. A few interviewees suggested that there should have been more direct consultations with institutions on the funding terms and conditions to ensure they were clearly defined, understood, and met institutions' needs.

Improved tracking of program performance is needed

While DRAC submitted a Performance, Evaluation, Risk and Audit Framework in 2021 to ISED, only some of the outcomes and indicators identified in the Framework were included in the 2022-23 progress report and annual report. As a result, it was difficult to measure the achievement of some objectives and track performance over time.

DRAC Cybersecurity Framework case study

Governance

According to interviewees, engagement, collaboration and consultation is baked into cybersecurity governance, and there has been participation from many stakeholders.

Interviewees found that the governance bodies were effective in moving the framework forward. The National Security Council oversees and coordinates implementation, including developing and adopting operational components, while the Cybersecurity Steering and Advisory Council is responsible for providing executive-level direction and approvals.

Security Operations

The Security Operations Team is distributed throughout the Federation. Interviewees noted that personnel management is challenging because staff are decentralized and work on both local and national services and may have competing priorities.

Some institutions also did not adequately maintain their cybersecurity staffing levels, which impacted delivery of DRAC's cybersecurity activities, so now the agreements more explicitly specify staffing requirements.

Funding is enough to maintain capacity, but may not be enough to increase capacity

Stakeholder groups interviewed generally agreed that the overall funding level is enough to maintain existing services and keep the ARC platform running but is not enough to significantly enhance capacity. DRAC officials noted that Canada is under-invested compared to other countries. For example, among its G7 peers, Canada is last in terms of total compute power. Interviewees said that given that computing is becoming more integral to the R&E sector, there will be a need to invest more over time and DRAC will also need to work towards a funding model that allows them to fund and co-fund more effectively, given the challenges in obtaining the required matching funds.

Match funding presented challenges for timely and predictable funding of activities

Under the previous funding model, the ability to obtain the required matching funding often had a significant influence on which projects were funded. The new funding model introduced when DRAC was established in 2019-20, where federal funding would cover more costs for national activities (increasing federal share from 40% to 60%), while institutions would cover more costs for local activities, was intended to provide flexibility to make more strategic investments.

Multiple stakeholder groups interviewed found that the new match requirements were still presenting challenges in funding activities in a timely and predictable manner. DRI players and DRAC officials said it is difficult when there are a multitude of institutions and provincial partners with differing funding priorities, terms and conditions, and requirements that may not be aligned with the national mandate, and match funding must be negotiated with each of them. Multiple stakeholder groups interviewed said it has been challenging moving initiatives forward, as it is difficult to align funding periods and get buy-in from provincial and institutional partners.

There is resistance from some funding partners in providing funding for national services, as there is a perception that they are funding the delivery of services to other institutions/provinces.

• Only a small portion of the universities and colleges (35 of 133 members, or 26%) contribute to fund ARC costs.
• The institutions that host the main ARC infrastructure (i.e., the host sites) bear a much larger share of costs.
• Some regions also contribute less to support ARC, which has been exacerbated by a deterioration in the funding capacity of some provincial and territorial partners.

To address these challenges, in 2022-23 DRAC developed a service classification model to identify responsibility and funding requirements at the national, regional and local levels, which included a service classification and cost eligibility framework, a funding model to inform investment co-funding, and Service Level Agreements.

DRAC Cybersecurity Framework case study

The match funding is challenging because cybersecurity tools are applied nationally, and it can be difficult to get the match when an institution or provincial partner is asked to pay 40% for a service that benefits the entire ecosystem.

For several projects, the requirement to obtain match funding influenced the types of solutions they could use for their initiatives (e.g., using a less expensive software solution with fewer functionalities to avoid match fund requirements).

There have been delays in ARC investments and the ramp-up in investment has been slow due to slower than planned implementation and challenges obtaining matching funds

ARC is a suite of digital technologies and services that enables researchers to solve research issues that are too large or complex for them to undertake on their own. The technology and tools are rapidly changing, and so the DRI requires an agile and responsive ecosystem with HQPs and sustained and predictable funding to keep pace. According to some interviewees, the pace of ARC renewal has not met the expectations of the DRI community. It was said that timely investment is critical, as the research community requires a significant increase in compute resources, and demand for capacity continues to grow. Interviewees said that renewal of ARC infrastructure is taking longer than planned, with some stakeholders reporting that the ARC refresh was significantly behind schedule, in part due to challenges in obtaining matching funds from provincial partners and institutions. The completion of the initial ARC infrastructure refresh for the National Host Sites, undertaken via the $50M ARC Expansion Program Contribution Agreement, was extended from April 1, 2021, to March 31, 2022. Interviewees noted it was largely complete, with some delays in Ontario.

After March 31, 2022, funding responsibility was transferred to DRAC. For DRAC's funding of ARC, the initial 2022-23 Multi-Year Funding Proposal included $18.5M for Regional Support and $33M for Capital and Operating spending. However, the Annual Report indicated that only $10M was provided by DRAC in 2022-23, with most of it for ARC Regional Support and a very small portion ($0.6M) for capital/infrastructure. In 2023-24, DRAC planned on making significant investments in ARC infrastructure ($73M in total ARC spending identified in its Business Plan) to replace node-for-node compute resources and a capacity replacement for storage on the four national ARC systems nearing end-of-life. However, a few interviewees said it would be difficult to complete DRAC's large ARC capital investments over such a short period (2023-24 and 2024-25). Interviewees identified a potential for delays as the refresh is dependent on obtaining the matching funds from provinces.

Figure 9: ARC infrastructure funding, 2019-20 to 2024-25

Recipients experienced issues with the timeliness and accessibility of funding

Some interviewees reported experiencing significant delays in their funding agreements following the transition to DRAC. They noted that having funding agreements move from multi-year to annual terms increased administrative burden and reduced funding stability, and the delays required time-consuming workarounds to bridge funding gaps. This challenge partly stemmed from the one-year funding agreements between ISED and DRAC for 2021-22 and 2022-23 fiscal years.

Performance

Capacity building

CANARIE enhanced network access and capacity

The CANARIE Network is the high-speed backbone of Canada's National Research and Education Network, which connects provincial and territorial networks to each other and to global research and education networks. Users/recipients reported that CANARIE's network has been available, reliable, and has sufficient capacity to meet current and future needs. From 2018-19 to 2022-23, CANARIE enhanced the speed of connections (72% increase) and increased the number of institutions (7.6% increase) with access to the network.

Figure 10: Increase in network speed x km (target)

Figure 11: Number of institutions with access to the network (target)

Over the evaluation period, CANARIE deployed additional connectivity and nodes to multiple sites to enhance domestic and international connectivity and reliability. CANARIE also enhanced connections in northern Canada, by adding Nunavut Arctic College as an NREN partner and enhancing Yukon network connectivity.

Accessing the NREN is going to allow our researchers to be able to better collaborate across the country with other researchers, [manage data, and access research tools] in a more secure way…[and] more tools and applications will function adequately.

CANARIE also provided $14M in funding to NREN partners via directed funding, to help ensure the NREN evolves consistently and strategically, and non-directed funding (competitive and interconnection and co-delivery funding), to support upgrades to network infrastructure.

CANARIE DAIR Cloud Program case study

CANARIE's DAIR program provides SMEs with free access to cloud services. DAIR helps SMEs develop and test their products, which supports a quicker path to market.

Initially, DAIR provided access to public cloud via a 3rd party cloud management platform, but it had limited capacity and so companies had to transition their products to another platform when development entered the production phase, which led to increased development time and cost.

To better support SMEs, DAIR introduced the Enhanced Services model which includes a DAIR-developed cost control platform for SMEs to use, as well as access to product development/testing services powered by commercial cloud (e.g., AWS).

CANARIE provided funding to support development of RS, and RDM

RS is fundamental to research, given its role in helping researchers make new discoveries and share their outputs with others. Both technological and cultural trends in DRI are driving the continual development of RS, making it even more critical in advancing research.

RDM supports the effective and responsible conduct of research and increases the ability to store, find and reuse research data, as well as facilitating national and international collaboration and sharing of data.

From 2018-19 until the programs transitioned to DRAC, CANARIE also launched seven funding calls for RS and RDM, which ultimately funded 56 projects – approximately $14M in funding for RS projects and $2.8M in funding for RDM projects, compared to forecasted spending of $16M and $4M, respectively.

Most recipients found that funding was provided in a timely manner, although one stakeholder found it difficult to identify the amount of funding available in advance of implementing network services. For RS projects, Figure 12 shows that the average turnaround time between funding applications and funding notifications decreased by 16%. Similarly, the turnaround time for RDM projects decreased by 47%, and for NREN projects it decreased by 29%.

Figure 12: Number of days between CANARIE's funding applications and notifications, 2015-2020 versus 2020-2023

CANARIE funding contributed to the creation and enhancement of research software

For the RS projects funded by CANARIE, the performance targets were met in most years (Figure 13), and there was a 100% increase in the number of CANARIE supported software services/platforms/tools created/enhanced per year over the evaluation period (2018-19 to 2022-23).

Figure 13: CANARIE supported software services/platforms/tools created/enhanced (target)

While there have been needed investments in ARC capacity, further work is needed

ARC, which refers to the elements required to perform computationally and data-intensive research and data management, including high-performance computing and storage, is critical to modern research and demand is growing rapidly (e.g., in big data, artificial intelligence, etc.). Interviewees said there have been some improvements to computing capacity and in 2023 DRAC began work on the refresh of aging infrastructure across national host sites. As part of the DRI Strategy funding, ISED provided $50 million to expand capacity and upgrade the ARC systems for the five main host sites (see Appendix E) by Winter 2021 and in 2023-24 DRAC began to roll out its planned investments in ARC infrastructure. Figure 14 shows that central processing unit (CPU) and graphics processing unit (GPU) computing and disc and tape storage capacities increased, and performance targets were met. The target of 3 ARC machines in the top 250 globally was also maintained for fiscal years 2019-20 to 2022-23.

Figure 14: Total CPU and GPU computing capacity and storage capacity (petabytes) of ARC Host Sites

However, interviewees said there have been significant delays in the refresh of ARC infrastructure and capacity is not sufficient to meet current and projected user needs. The current state reports for ARC and the 2021 Researcher Needs assessment found there was an insufficient supply of CPU and GPU resources to fulfill the growing demand. Interviewees said challenges with matching funding, the complexity of roles and responsibilities, and disruptions caused by the pandemic contributed to the delayed ARC capacity investments, due to increased computing hardware costs stemming from supply chain disruptions in the post-COVID 19 era and increased global demand.

Progress has been made on implementing RDM activities, but there have been delays in research software funding activities

While 22.1% of DRAC's spending was on RDM (investments in RDM systems and three project funding calls), RS funding was delayed. Responsibility for RS was transferred to DRAC on March 31, 2022. CANARIE's last funding calls for RS were in 2020-21 and that funding was fully disbursed by the end of 2022-23. Following transition of responsibility, DRAC's initial proposal, which included transition funding for RS teams and platforms beginning in 2023-24, was not approved by ISED in its 2023-2025 Multi-Year Funding Proposal, due to a lack of an RS strategy (a strategy has since been developed but has not been finalized). Consequently, interviewees noted that funding had not yet been provided and was not expected until at least 2025. As a result, they were experiencing difficulties enhancing their products and retaining staff.

Recognition and adoption

CANARIE and DRAC have taken a strategic approach to engagement with researchers

DRAC and CANARIE engage with researchers across all disciplines, with the biggest users in STEM fields, but increasingly in other disciplines (i.e., social sciences and humanities). Engagement with researchers is largely indirect. The large and diverse researcher base makes direct outreach more difficult (e.g., there were 21,783 active users of the National ARC platform in 2022-23), as there is a diversity of needs, interests, DRI capabilities, and researchers may not be aware they are using DRI services and do not have time for engagement. As such, what researchers are interested in engaging on and how to structure an engagement is a challenge. Consequently, engagement with researchers is primarily indirect, via NRENs, host sites, partner, and institutional representatives. Direct interactions are most frequent with researchers who are super users (i.e., researchers that regularly work with large datasets). DRAC's engagement approach prioritizes researchers with a high influence and dependence on DRAC and uses a Researcher Council to engage key researcher representatives.

Direct engagement was done to increase awareness and assess user needs

DRAC and CANARIE undertook direct engagement of stakeholders to facilitate increased awareness of their services and understanding of users' needs. To do this, DRAC and CANARIE participate in events, hold conferences, use mailing lists and newsletters, conducts user needs surveys, host forums, engage with delivery partners, participate in roadshows, create marketing materials, deliver webinars, and use their websites and social media. However, recipients said more work is needed to raise awareness of services (e.g., reaching stakeholders who are not a part of the existing DRI community, particularly researchers in non-STEM fields who may not even be aware they are using DRAC infrastructure).

Through direct and indirect engagement processes, DRAC has informed various areas of its activities, such as its governance structure, ARC, RDM, and RS Current State assessments, Needs Assessment, and its Strategic Plan. Similarly, CANARIE's engagement and outreach has helped inform the refinement of its programming, and increased its uptake (e.g., DAIR and the Cybersecurity Initiatives Program).

CANARIE DAIR Cloud Program case study

Recipients noted there is low awareness of CANARIE services among start-ups. Outreach is more challenging, since there are no immediate connections or networks.

However, DAIR has increased outreach with incubators and accelerators. With CANARIE's AWS partnership, Amazon now promotes DAIR to its mailing list and CANARIE hosts presentations with AWS guest speakers.

DRAC Cybersecurity Framework case study

DRAC officials interviewed noted there was 2-3 months of engagement on the framework. Stakeholders suggested additions and modifications and DRAC took all the feedback and incorporated it.

For the framework, DRAC conducted 628 consultations (see Table 3), distributed 11 surveys, and organized three town hall meetings within the ecosystem.

CANARIE had increased uptake of its services, tools and network

The number of users of CANARIE-supported software services, platforms, and tools under the CANARIE Research Software program increased by 184% (Figure 15). The reuse of these tools similarly increased by 170%, with project data indicating a significant number of "person-days" saved from reusing software (i.e., the days of work for a person to create new software instead of reusing it).

Figure 15: Users of CANARIE-supported software services, platforms, and tools (target)

Traffic to and from major Canadian research infrastructure also increased, although the targets were not met in 2020-21 and 2022-23, with the pandemic impacting traffic in 2020-21.

Figure 16: Traffic to and from major Canadian and international research infrastructure (Number of terabytes)

There were significant increases in the number of disciplines using funded software (115%), while use across multiple research teams was smaller (9%). Project data showed most of the funded RS projects were multidisciplinary (87%) and around half of the RS and RDM projects had collaborators.

CANARIE DAIR Cloud Program case study

Program data showed there were large increases in the number of SMEs using DAIR services (57%) and in the use of BoosterPacks (392%) over the evaluation period, as a result of enhancements to the services and increased marketing and outreach activities.

Figure 17: Number of SMEs using DAIR services

Interviewees said that DAIR led to cost savings and allowed them to develop products quicker/more effectively. Program data indicated that 18 participants developed at least one application/service in 2021-22 and 2022-23. For example, ISED's 2019 case study of DAIR's support to the Canadian automotive software company Rebel Technologies found that it provided resources and opportunities that helped them commercialize faster.

CANARIE Cybersecurity Initiatives Program case study

In 2022-23, there were 213 Canadian institutions participating (target of 205). Interviewees explained that the CIP saves the time and effort that would have been required to procure tools on their own. It was said that most initiatives are very easy to implement and use, as they are largely automated and free of charge, and CANARIE provided support when needed. The Firewall and Threat Feed require a few hours to set up and little maintenance, the Intrusion Detection System requires 8-12 hours, and the NCA requires a few days.

However, small institutions do not always have the personnel to adopt initiatives and thus had slower uptake. Interviewees explained that CANARIE is exploring ways to make them more accessible, by providing more support from CANARIE, making the initiatives more turnkey, or adding more vendor support.

There was increased acknowledgement and use of CANARIE research software, both domestically and internationally

Figure 18 shows that acknowledgment of CANARIE-supported technologies increased. Most CANARIE project participants cited or acknowledged their use of CANARIE-supported software. The number of presentations demonstrating CANARIE-supported technologies and RDM nearly doubled over the evaluation period. Project data similarly shows that 68% of funded projects for RS and RDM had demos/ presentations/showcases/papers/ posters presented at Canadian and international events.

Figure 18: Number of presentations, demos and papers to demonstrate CANARIE-supported technologies

Figure 19 shows that the use of CANARIE-funded research software by international groups increased by 119%, indicating that there was increased recognition and usefulness of the software internationally. The number of international collaborators also increased (by 80%). Project data also showed that around half of the funded projects had international collaborators and the majority (69%) of RS projects were used by international research teams.

Figure 19: International groups making use of CANARIE-supported software

There were challenges in the use of Research Data Management plans and policies

RDM plans/strategies are meant to improve the quality and efficiency of the research and facilitate collaboration among researchers through improved accessibility and usability of data for other researchers.

The number of RDM plans created per year was about 1,300, but there were challenges in meeting its performance objectives (e.g., its target in 2021-22 was for 2,000 plans). Similarly, the number of RDM policies implemented was well below its targeted level each year, with only 3 to 6 policies implemented annually.

For 2021-22, RDM Plans were expected to increase due to a new Tri-Agency RDM Policy, but the policy was delayed to March 2021 and to date RDM plans have been required for only a handful of Tri-Agency programs, beginning in Spring 2022.

Figure 20: Number of RDM plans created (target)

DRAC membership levels were slightly below target

DRAC is a membership-based organization. Members participate in governance based on their interest in doing so: for example, members are entitled to vote at the Annual General Meeting to approve slates of candidates for Board Members. The number of members is dynamic and can fluctuate over time. However, membership is not a pre-requisite to access DRAC services: they are available to members of the research community regardless of their institution's membership status. Program data shows overall DRAC membership was relatively stable between 2019-20 and 2022-23, varying between 133 and 146 members, DRAC was slightly below its target of 149 member institutions by 2023.

Figure 21: DRAC membership by membership type

There was an increased number of ARC users and a high demand for ARC capacity

Since 2019-20, the number of active ARC users increased by 36%, which exceeded DRAC's target (19,192). As a result, demand for resources has grown each year, and ARC systems are running at maximum capacity. For projects that need additional capacity, researchers must apply to Resource Allocation Competitions (RAC). For the years 2019-20 to 2022-23, DRAC provided almost half of the requested CPU capacity, most of the requested storage capacity, and a small portion of requested GPU capacity (see Figure 22). Evidence from the RAC capacity requests demonstrate the high level of demand for resources. For 2022-23, users requested 460,346 CPU core years, while the total combined capacity of all the ARC systems was only 322,720 CPU core years. Similarly, users requested significantly more GPU core years than available.

Figure 22: Average CPU, GPU, and storage allocations over the evaluation period

Further, the Researcher Needs Assessment indicated that the RAC process for computer resources was viewed by researchers as cumbersome and slow, took large amounts of human resources to complete, needed to be frequently renewed, and the periods for which they were allocated did not always match the needs of researchers (thereby contributing to underutilization of resources, if they are unable to use them in the period allocated). Further, different types of GPUs offer different orders of magnitude in performance, which makes it hard for researchers to estimate needs in terms of the number of GPUs/GPU years needed.

The number of users of DRAC's data management systems/services increased from 2019-20 to 2022-23

DRAC assumed the responsibilities of Research Data Canada and Portage and to this end, DRAC has furthered these organizations' work in establishing and developing RDM services, tools, and repositories. While DRAC has not established formal baselines or targets, as many of these services/systems were new or only recently transferred to DRAC, data showed significant and sustained year-over-year increases in the use of these services/systems.

There was a 1,469% increase in registered users of the Federated Research Data Repository (FRDR), which is a national, bilingual platform for sharing and preserving Canadian research data, managed by DRAC.

Figure 23: Number of FRDR users

There was a 49% increase in DataCite Canada members, which is a non-profit managed by DRAC and the Canadian Research Knowledge Network, that provides Digital Object Identifiers and metadata registration services for institutions.

Figure 24: Increase in the number of DataCite Canada Members

Figure 25: Number of Borealis Research Data Repository participants

The number of datasets available through these data management repositories also increased, thereby increasing access to data

The number of datasets available through FRDR increased by 218% (from 118 to 375) and the number of datasets available through Borealis increased by 116% (from 6,340 to 13,720).

The use of these dataset repositories also increased, with FRDR file downloads and dataset views increasing by 189% and 1,948%, respectively.

Figure 26: FRDR dataset views

Figure 27: FRDR file downloads

Cybersecurity

DRAC and CANARIE established a higher security baseline

Interviewees explained that DRAC and CANARIE activities have encouraged stakeholders to prioritize cybersecurity risk. They provided expertise and capacity that institutions may not otherwise have had access to and established a higher security baseline among institutions. While cybersecurity efforts were effective, it was felt by some interviewees that they need to move faster, as spending is not keeping pace with the growth in threats and is lower than in other sectors (e.g., banking).

CANARIE cybersecurity activities

CANARIE's Cybersecurity Initiatives Program (CIP), launched in 2020, includes a threat feed, intrusion detection, firewall, cybersecurity assessments, and benchmarking. In addition to the CIP activities, CANARIE deployed tools to improve overall network security, such as a platform to combat distributed denial-of-service (DDoS) attacks and a proof of concept for Multi-Factor Authentication.

Interviewees said that the high use of CIP tools demonstrates that they are of value and having an impact. Users noted improvements to security capabilities and maturity as a result of having more effective tools for protection against cyber threats and having tools to identify their organizations' cybersecurity gaps and needs. Recipients reported that the ability to detect and respond to threats improved, with program data showing that in 2022-23, there were:

• 8.1M threat vectors blocked
• 286K threats identified
• 34K times action taken

DRAC Cybersecurity Framework

Interviewees agreed that DRAC's activities are contributing to securing the research environment, the approach is working well, and DRAC is meeting its objectives. Currently the Alliance-Federation has a low cybersecurity maturity level. The overarching goal of the Cybersecurity Framework is to mature cybersecurity capabilities from 0.75 to 1.18 (a rating based on the National Institute of Standards and Technology 4 tiers of cybersecurity maturity) by 2024-25, which would move DRAC closer towards becoming a Tier 2 "Risk Informed" organization (i.e., aware of cybersecurity risks and share information on an informal basis). To accomplish this, there are 57 identified deliverables in the Framework, categorized under areas such as: Policy and Governance; Risk Management; and Asset Inventory. Progress on these deliverables can be seen in Figure 25. Among the deliverables that were still in-progress, DRAC reported that about half were delayed/behind schedule.

Figure 28: Cybersecurity Framework deliverables progress

Skills development

DRAC is addressing identified gaps in DRI training

DRAC users said that there needs to be resources and support for education and training, as DRI requires a certain skill set to effectively use, both for basic users such as researchers as well as for the HQPs that operate and maintain the digital research infrastructure. There are hundreds of courses provided by institutions and regional organizations, with around 30K training hours provided to 10,000 participants annually. Interviews and documents indicated it is challenging for researchers to sift through these offerings to find the training they require and recommended having a centralization training platform.

There are also large differences in the availability of training across academic departments, institutions, and regions. DRAC's needs assessment found the gap was particularly pronounced for social sciences, humanities and health researchers, where there was a notable lack of training opportunities for HQP, poor documentation, few workshops/courses, and inadequate support for DRI at the institutional level.

The needs assessment found that funding for research support staff and training of HQPs were the top priorities for researchers, and noted areas such as data management, software development and use of HPC systems are key areas for training (Figure 29 and 30). DRAC has acknowledged that a comprehensive and strategic training approach is required to match the complexity and diversity of training needs.

Figure 29: Top 5 areas of researchers' DRI needs identified in the needs assessment

Figure 30: Top 5 areas identified in the needs assessment where DRI training is most required

DRAC is developing supports and platforms to address training gaps

DRAC identified the development of a data-driven, ecosystem-wide national training strategy as a high priority. While a multitude of DRI training resources already exist, there is currently no national development program or platform to upskill researchers or research support personnel. The initiative aims to address gaps in training provision, particularly for RDM, RS, and Cybersecurity, which currently lack technological infrastructure and content management support at the national level. This initiative will also integrate RDM and RS content into existing ARC-focused training programs.

In 2022-23, DRAC worked with national and regional training providers to compile a list of training requirements for researchers and submitted its approach in its Multi-Year Funding Proposal, which includes a:

• National Training Support Team to support researchers, including three FTEs to develop and implement the strategy.
• Discovery Platform to help researchers more easily find training and resources across the ecosystem.
• Training Portal to support national and regional training programs.

CANARIE guidance and support was effective and accessible

Users/recipients were satisfied with the effectiveness and accessibility of guidance and supports available for CANARIE's services, initiatives, and tools. A lot of the learning/training provided to stakeholders is via "learning by doing", through technical support, and troubleshooting of issues. CANARIE also relies on its NREN partners to provide training and support.

CANARIE also provides technical documentation and guides, configuration tools, videos, and webinars for services such as eduroam, the Cybersecurity Initiatives Program, and DAIR. Interviewees also noted collaborative learning and training via initiatives such as Security Information and Event Management, which is a collaboration among 13 NREN partners.

DRAC Cybersecurity Framework case study

DRAC is developing a cybersecurity training program to fill a gap for institutes that don't have resources to provide basic cybersecurity training. Among researchers, there is low awareness of training opportunities. Some institutions have mature programs, while others have limited offerings, and so training is not accessible for all stakeholders. DRAC is looking to increase awareness of existing training offerings, align and augment them, and make training available to all institutions. To accomplish this, DRAC will use the platform it is developing for its broader Training and Awareness Program.

CANARIE Cybersecurity Initiatives Program case study

Interviewees said CANARIE provides the appropriate level of support to institutions to facilitate the use of the CIP offerings (IDS, Cybersecurity Assessment, Benchmarking, Threat Assessment, and Firewall), which includes technical documentation and support, webinars, training sessions, and communities of practice (e.g., a Slack channel and regular meetings for IDS). Interviewees also said there is indirect learning/training where institutions consult with each other on cybersecurity and implementation issues.

DAIR is the only spot in Canada that has a concentration of entrepreneurs using the cloud that can swap expertise and advice in a single and easily accessible location.

Efficiency

CANARIE spending and its allocation among activities was less than planned

Expenditures were below CANARIE's budget estimates in most years, in part due to supply chain delays (e.g., the refresh of the Central fibre system). As of 2022-23, the share of expenditures allocated to network operations (66%) was below the share indicated in the Contribution Agreement (72%). However, the 2023-24 budget indicated that network expenditures were expected to increase.

Figure 31: CANARIE budgeted vs actual expenditures

For the evaluation period, matching and in-kind contributions (e.g., provincial government, partner funds, etc.) was about $1.85M per year (about 6% of total revenues). For CANARIE funded projects, (NREN, RDM, and RS), the amount of funding leveraged from provincial and local sources ranged from 18-42%, depending on the project and showed a slight decline (2-7%) between the 2015-2020 and the 2020-23 funding periods. For NREN projects, CANARIE met its target of 65/35 split between CANARIE and other funding sources.

Figure 32: Leveraged Funding for CANARIE's project funding calls

CANARIE's administrative expenditures were projected to be within the 15% ratio required under the Contribution Agreement

While administrative costs for CANARIE were higher than 15% in some years, the large increase in program expenditures planned for 2023-24 was expected to bring the overall administrative cost ratio to within the allowable limit for the current Contribution Agreement period.

Figure 33: CANARIE administration costs as % of expenditures

DRAC was unable to use the allocated funding within approved timeframes

In total, ISED allocated $375 million to DRAC for fiscal years 2019-20 to 2023-24. As a result of delays, this funding was extended an additional year to 2024-25. As of 2022-23, DRAC's cumulative annual expenditures were $50.5M. DRAC had an additional $228.4 million in approved spending for fiscal years 2023-24 to 2024-25, leaving at least $96.2 million of the original allocation expected to be unspent within the term of the current funding agreement (Figure 34).

Funding was not leveraged by DRAC from partners until the 2022-23 fiscal year

For the first three years of implementation (2019-20 to 2021-22), 100% of DRAC's revenue was from ISED. As per the Contribution Agreement, DRAC's goal, to be achieved beginning in 2023, is to fund approximately 60% of infrastructure and services with ISED funding and 40% with match funding leveraged from partners. In 2022-23, ISED funding ($31.5 million) leveraged $15 million in funding from partners.

Figure 34: DRAC funding allocations, 2019-20 to 2024-25

DRAC's relative expenditures on ARC were less than initially targeted

Targeted funding ratios for DRAC pillars were 65% for ARC, 11% for HQP, 3% for RS and 5% for RDM. From 2019-20 to 2022-23, expenditures were allocated as follows: 22.7% ARC, 22.1% RDM, 13.7% HQP, 3% cybersecurity, and 0% RS (Figure 35). ARC and RS were significantly below targeted amounts, while RDM and HQP were significantly above, however, DRAC planned on significantly increasing its ARC expenditures ($73 million) in 2023-24. This was in part due to COVID-19 impacts, supply chain disruptions, and inability to get matching funds from partners (requiring a robust framework between levels of government).

Figure 35: DRAC expenditure ratios for DRAC pillars

DRAC's administration costs as % of expenditures were high in the first three years

As per the Contribution Agreement, DRAC can claim up to ten (10) percent of the total contribution (i.e., $375 million) for general administration. These expenditures were 19.4% ($10M) for the years 2019-20 to 2022-23, but significantly increased DRI pillar expenditures in 2023-24 and 2024-25 are expected to bring the ratio down to under 8%, if implemented as planned.

Equity, Diversity, and Inclusion (EDI) governance and delivery measures are in place for both CANARIE and DRAC

CANARIE and DRAC formed a joint EDI working group and developed a Joint EDI Strategy to embed EDI measures within processes, communications, and recruitment.

CANARIE

CANARIE's 15-member Board of Directors is composed of industry and academic institution representation. To support EDI, the Board strives to maintain a gender balance of between 40% and 60%, as well as a regional and both official languages representation.

To advance EDI, CANARIE joined the Government of Canada's "50-30 Challenge" and has made an organization-wide commitment to individual (e.g., EDI training) and corporate actions (e.g., reviewing HR policies) that contribute to EDI. Further, CANARIE developed a Multi-Year Accessibility Plan to integrate accessibility practices across all facets of the organization.

DRAC

The outcomes of Year 1 (2019-2020) of DRAC's inception were to establish DRAC's governance through DRI community engagements. Equity, Diversity, Inclusion and Accessibility (EDIA) principles were central to the Board of Directors and Researcher Council outreach and recruitment processes. As a result:

• DRAC committed to achieving the "50-30 Challenge"
• Half of the inaugural Board of Director members identified as female and included Indigenous representation as well as linguistic and geographic diversity
• Half of the inaugural Researcher Council identified as female and 25% as visible minorities. The council is composed of multi-disciplinary researchers from across the regions.

DRAC created an EDI framework to facilitate integration of EDI into its initiatives (e.g., the Researcher Needs Assessment and the New Service Delivery Model). DRAC is also engaging Indigenous stakeholders to better understand their research and data sovereignty needs. Documents indicated that further work is needed to reach the target of 30% representation of staff identifying as members of underrepresented groups.

In 2021-22, DRAC formed the EDIA Committee and four working groups to advance EDIA:

• Cultural Benchmarking Survey Working Group
• EDIA Corporate Events Working Group
• First Nations, Inuit and Métis Engagement Working Group
• Recruitment and Retention Working Group

Summary

Conclusion

• There is a continued need to support DRI with a coordinated strategy, and DRAC and CANARIE are largely complementary in this role, however DRI needs are not yet being sufficiently addressed. CANARIE continues to make progress in achieving its expected outcomes related to capacity enhancements, recognition and adoption, and skills development, while further progress is needed by DRAC, particularly for enhancements to capacity. Additionally, DRAC implementation has been challenging and key activities have taken longer than expected, in part due to challenges with match funding. Both organizations have seen delays and deviations in their expenditures to date.

Lessons learned

• The establishment of a centralized, national program for DRI infrastructure within a decentralized environment was a highly complex undertaking, which required a significant amount of time and resources to implement. These challenges were further exacerbated by the disruptions stemming from the COVID-19 pandemic, as well as supply chain issues. Within such a complex and transitional environment, implementation and making timely investments to enhance services/capacity has been challenging. For future initiatives involving similarly complex environments, further planning and assessment of the ecosystem and organizational landscape could help to better understand and effectively mitigate potential implementation challenges and risks before they arise.

Finding 1: The need for DRI is increasing, but many institutions do not have the resources to address it on their own. To help address this need in a more efficient and coordinated manner, the DRI Strategy was created. However, further work is needed by DRAC in terms of addressing the DRI needs of the ecosystem.

Finding 2: There is a need to increase the cybersecurity capabilities and alignment of institutions in the DRI ecosystem. CANARIE and DRAC established activities to address this need, but increased collaboration is needed to minimize overlap, ensure continued alignment over time, and address new cybersecurity risks in the research and education sector as they emerge.

Finding 3: DRAC implementation took longer than planned. Challenges were experienced in implementing the organization, transitioning DRI responsibilities to DRAC, navigating the delivery model and tracking performance to effectively measure success. These challenges were exacerbated by the COVID-19 pandemic.

Finding 4: While DRAC has consulted stakeholders and improved the effectiveness of its governance, engagement, and relationships over time, more adjustments and meaningful consultations are needed to better respond to stakeholder needs and increase awareness of roles, responsibilities, relationships, and funding mechanisms.

Finding 5: DRAC's new funding model was meant to address some challenges with match funding. However, challenges continue to be experienced in funding activities in a timely and predictable manner.

Finding 6: Renewal of ARC infrastructure has been slow and challenging, partly due to difficulty obtaining match funds and the pace of renewal does not meet the growing needs of the research community.

Finding 7: CANARIE enhanced network capacity and access, while DRAC's ARC capacity was insufficient to meet overall demand despite meeting its performance targets. There were also delays in DRAC's RS investments.

Finding 8: DRAC and CANARIE's engagement with researchers has been strategic, largely through representatives, as well as more targeted, which has facilitated increased awareness and the identification of user needs.

Finding 9: There was increased uptake of CANARIE's services and tools domestically and internationally, but there were challenges meeting targets for RDM plans and RDM policies.

Finding 10: DRAC membership was relatively stable over time. The number of ARC and RDM users has grown and the number of datasets available and used in the RDM systems also increased. The requested CPU and GPU capacities were significantly higher than available capacities and challenges were experienced with the effectiveness and efficiency of the allocation process.

Finding 11: DRAC and CANARIE activities supported increased prioritization of cybersecurity among institutions, enabling progress towards increasing the security capabilities of the DRI ecosystem.

Finding 12: There is a large variety of DRI training dispersed unevenly across institutions. As a result, there are visibility and accessibility challenges. DRAC is in the process of developing a training team, portal, and platform to address these gaps. For CANARIE, users were satisfied with the guidance and support provided.

Finding 13: CANARIE experienced some deviations in expenditures to date. For DRAC, delays in implementation ultimately contributed to lower than planned expenditures on DRI pillars and projects, along with higher than planned expenditures on governance and administration.

Finding 14: DRAC and CANARIE have implemented measures to increase representation in their organizations.

Recommendations

Recommendations: The evaluation led to six recommendations.

SRS will work with the third-party organization(s) to monitor the implementation of the following recommendations:

Recommendation 1: CANARIE and DRAC should explore opportunities to enhance CANARIE's information sharing and collaboration with DRAC on cybersecurity-related activities and further clarify roles to minimize overlap in the development of cybersecurity initiatives.

Recommendation 2: There is a need for DRAC to enhance funding recipients' level of understanding by further clarifying roles and responsibilities and improving communication. DRAC should explore approaches to facilitate further meaningful engagement of funding recipients and other stakeholders (e.g., via a comprehensive stakeholder engagement strategy).

Recommendation 3: DRAC should develop a consistent and comprehensive approach to reporting on the outcomes and indicators of the Performance, Evaluation, Risk and Audit Framework.

Recommendation 4: DRAC should explore how to increase the timeliness and predictability of match funding, including examining lessons learned and approaches used by other organizations.

Recommendation 5: DRAC should explore additional tools or measures to minimize negative impacts on the research software community stemming from the funding transition period for research software funding and support.

Recommendation 6: DRAC should develop a strategy or approach to improve the ARC resource allocation process to better meet the needs of researchers.

Appendices

• Appendix A: CANARIE logic model
• Appendix B: DRAC logic model
• Appendix C: Evaluation methodology
• Appendix D: Evaluation challenges
• Appendix E: ARC National Host Sites
• Appendix F: CANARIE NREN partners
• Appendix G: End notes

Appendix A: CANARIE logic model

 

Appendix B: DRAC logic model

 

Appendix C: Evaluation methodology

To address the evaluation questions, information was triangulated from multiple lines of evidence, including the following qualitative and quantitative sources.

Document review

The document review included: key government priority-setting documents (i.e. 2018 Federal Budget); CANARIE and DRAC foundation documents (i.e. Performance Measurement Strategies); and CANARIE and DRAC reporting documents (i.e. Corporate/Business Plans and Annual Reports), including joint reporting documents (i.e. Cybersecurity Framework).

Performance, administrative and financial data review

AEB reviewed CANARIE and DRAC administrative, financial and performance data to assess the extent to which progress has been made towards achieving expected outcomes (short- to long-term) and to assess the efficiency of its delivery model. Where possible and relevant, analysis of performance data was disaggregated by indigenous status, racialized person status and disability status to incorporate Gender-Based Analysis (GBA+) into the evaluation.

Key informant interviews

AEB interviewed 33 stakeholders among the following groups to gather diverse perspectives on the relevance, performance and efficiency of CANARIE and DRAC:

• Funded entity senior management and staff (e.g., CANARIE/DRAC Board of Directors and Executives);
• ISED officials;
• Innovation partners and DRI players (including international);
• Ultimate recipients of CANARIE/DRAC funding; and
• Users of the services offered by CANARIE and/or DRAC.

Case studies

AEB studied four projects supported by CANARIE (2) and DRAC (2) to provide a more detailed perspective on the progress of both organizations towards achieving their expected outcomes, along with gathering the views of project participants. Case studies were selected in consultation with the DRI program area.

Appendix D: Evaluation challenges

Respondent bias

There was potential for respondent bias given that some interview participants were involved in program delivery or are funding recipients. As such, some of the findings may be positively biased. There was equal potential for negative bias due to strong negative community perception of DRAC due to the leadership style of the previous DRAC CEO.

To mitigate this, the purpose of the interview (an analysis of the DRI Strategy as a whole and not simply DRAC's performance) and its strict confidentiality was clearly communicated to participants. Further, responses were cross-validated across stakeholder groups as well as other lines of evidence.

Data availability

The previous CANARIE evaluation found that data on funded projects was limited for CANARIE's Research Data Management program given that the CANARIE Board of Directors only approved this program in October 2017 and proposals to begin funding projects only began in 2018-19.

For the current evaluation, project data was available, and this data was also validated with additional information obtained from CANARIE's performance measurement data, program documents, and interview findings.

Appendix E: ARC National Host Sites

Source

Technical glossary

Advanced Research Computing (ARC) refers to the elements required to perform computationally and data-intensive research and data management, including high-performance computing and storage. ARC relies on high-speed networks, software, standards and data-management services. Canada's National ARC host sites include:

• McGill University (Quebec): The Béluga and Narval systems are heterogeneous multi-purpose supercomputers, each of them with four principal components: a CPU subsection, a GPU subsection, data-storage devices and a high-speed interconnect linking together the other three subsystems. As of March 2023, Narval had 75,584 CPU and 648 GPU core years of computing capacity and Béluga had 32,080 CPU core years of computing capacity.
• University of Victoria (British Columbia): The Arbutus system is an OpenStack cloud, with an emphasis on hosting virtual machines and other cloud workloads. Its powerful storage and compute capabilities are designed to support researchers processing, sharing and storing massive data sets. As of March 2023, Arbutus had 33,724 vCPU and 416 vGPU core years of computing capacity.
• Simon Frasier University (British Columbia): The Cedar system is a heterogeneous cluster, suitable for a variety of workloads. With over 3.6 petaFLOPS of computing power. As of March 2023, Cedar had 92,528 CPU, 1,352 GPU, and 1,920 vCPU core years of computing capacity.
• University of Waterloo (Ontario): the Graham system is a heterogeneous cluster, suitable for a variety of workloads. The parallel file system and external persistent storage are similar to the Cedar System's, but the interconnect is different and there is a slightly different mix of compute nodes. As of March 2023, Graham had 36,432 CPU, 540 GPU, and 9,984 vCPU core years of computing capacity.
• University of Toronto (Ontario): The Niagara system is a homogenous cluster owned by the University of Toronto and run by SciNet and is intended to enable large parallel jobs of 1,040 cores and more. It was designed to optimize throughput of a range of scientific codes running at scale in an energy efficient way. As of March 2023, Niagara had 80,960 CPU core years of computing capacity.

Appendix F: CANARIE NREN partners

13 provincial and territorial partners form Canada's National Research an Education Networks (NREN). The NREN is a collective of infrastructure, tools, and people, that connects researchers, innovators and educators to more than 100 NREN's around the world.

NRENs provide ultra high-speed information networks and services dedicated to the unique needs of universities, colleges, and research institutes. In Canada, the NREN also connects teaching hospitals, libraries, museums, innovation centres, federal research labs, and in some provinces, K-12 schools. Canada's NREN partners include:

• ACORN-NL: is dedicated to the implementation and formalization of a regional advanced network in Newfoundland and Labrador that will link post-secondary institutions, their major research centres, and the K-12 system.
• ACORN-NS: objectives are to refine, implement and maintain a resilient, secure, advanced research and education network architecture; reliably operate Nova Scotia's advanced research and education network including the CANARIE Point of Presence; promote the value and innovation created by the advanced research and education network; and maintain a sustainable model for ACORN-NS.
• Aurora College: Aurora College provides advanced-network services for the Northwest Territories.
• BCNET: is a not-for-profit, shared IT services organization that represents the interests of our members—colleges, universities, and research institutes in British Columbia.
• Cybera: is Alberta's not-for-profit organization responsible for driving economic growth through the use of digital technology. Our core role is to oversee the development and operations of Alberta's cyberinfrastructure – the advanced system of networks and computers that keep government, educators, not-for-profits, and entrepreneurs at the forefront of technological change.
• ECN-New Brunswick/PEI: provides a sustainable, economical, and reliable high-speed network platform to educational and research organizations throughout the region. It connects them to research and innovation networks across Canada and around the world.
• MRnet: is Manitoba's Regional Advanced Research and Education Network. MRnet provides a very-high-speed, fibre optic network linking its research and education members to the world's R&E networks.
• Nunavut Arctic College: will represent Nunavut on the NREN and will work with partners to create a research and education network in Nunavut. NAC is a public agency of the Government of Nunavut and services the largest post-secondary service region in Canada.
• ORION: is a not-for-profit organization dedicated to empowering Ontario researchers, educators, and innovators. The organization's community includes more than two million users at over one hundred universities, colleges, hospitals, and research institutions, as well as the majority of Ontario's school boards.
• RISQ: As a not-for-profit organization, the Quebec Scientific Information Network (RISQ) is a private telecommunications network that has served the education sector in Quebec since 1989.
• SRNET (Saskatchewan Research Network Inc.): is a not-for-profit research and education network, that provides dedicated high-speed network (10 Gbps, 1 Gbps, 100 Mbps) access to institutions and companies across Saskatchewan.
• Yukon University: Yukon University provides a vital link between students and researchers in the Yukon, and national and international education and research networks.

Appendix G: Key stakeholders in Canada's DRI landscape

 

Appendix F: End notes