Language selection


Evaluation of Innovation Programming

September 2017

View the print-friendly version: PDF

Table of Contents

Western Economic Diversification Canada (WD) would like to thank all of the key informants and survey participants who generously gave of their time and knowledge to take part in the Evaluation of Innovation. Without their participation and their insights, this report would not have been possible. The evaluators acknowledge the work done by Goss Gilroy Inc. (the consultants) in collecting key informant, focus group and survey data.


Section 1: Evaluation Recommendations

WD's Innovation programming makes strategic investments with western Canadian businesses, industry organizations and post-secondary institutions to enhance business innovation through research and development, and technology commercialization. The evaluation concludes that WD's innovation programming is relevant and achieving results. The evaluation identified possible improvements to enhance the design and delivery of the programming and to strengthen achievement of longer-term outcomes.

Recommendation 1: The department should develop a system to track longer-term outcomes on projects after project completion.

Recommendation 2: The department should review its project intake process to improve upon its frequency and predictability, and its interaction with applicants during the project assessment stage.

Recommendation 3: The department should carefully assess how it will provide meaningful feedback to unsuccessful applicants on their applications

Return to the top of this pagetop of page

Section 2: Introduction

2.2 Background

The Department of Western Economic Diversification Canada (WD) was established in 1987 to lessen western Canada's (British Columbia, Alberta, Saskatchewan, and Manitoba) strong economic dependence on its natural resources. Under the Western Economic Diversification Act, 1988, the department is mandated to "promote the development and diversification of the economy of Western Canada and to advance the interests of Western Canada in national economic policy, program and project development and implementation."

Innovation programming within the department contribute to WD's (the department's) strategic outcome of growing and diversifying the western Canadian economy. The department uses two grant and contribution (G&C) programs to provide funding for projects under its innovation programming activities. The initiatives are:

2.3 Evaluation Context and Scope

This evaluation study was included in the department's evaluation plan (2015–20). This evaluation addresses the core issues of relevance and performance identified in the Treasury Board Policy on Results. The scope of the evaluation included all WD innovation projects (WDP and WINN) from 2010–11 to 2014–15. The scope was expanded to include completed projects implemented prior to 2010–11 specifically to gather data on longer-term results.

Characteristics of the Projects Selected for the Evaluation

2.4 Evaluation Approach, Design and Methodology

The evaluation issues addressed in this study related to relevance and performance. The methodology for the evaluation include:

2.5 Limitations

Return to the top of this pagetop of page


Section 3: Relevance

3.1 Continued Need for Innovation Programming

The report, Innovation Canada: A Call to Action[1] (commonly known as the Jenkins Report) concluded that productivity growth drives increases in average per capita incomes and business competitiveness, and that productivity growth is primarily the result of innovation. The report also concluded that innovation gains require an environment or ecosystem conducive to innovative activity, which is supported by both the public and private sectors (e.g., innovation requires sufficient investment in R&D, the presence of high quality scientific research institutions, extensive collaboration between universities and industry, the protection of intellectual property, high levels of competition, and access to various forms of capital and financing).

Overall, Canada tends to lag other OECD nations in terms of innovation performance. For example, according to the most recent international comparison conducted by the Conference Board of Canada in 2015[2], Canada ranked 9th among the 16 Organizations for Economic Co-operation and Development (OECD) countries in terms of innovation. The report noted that performance on a few innovation indicators has improved, but there are signs of emerging and persistent weaknesses on a number of others. If the four western provinces were included within 16 OECD countries, British Columbia would rank 8th amongst the 16 countries (after Sweden, Denmark, Finland, the US, Switzerland, the Netherlands, and Austria). Alberta would rank 11th, and Saskatchewan and Manitoba would each rank 15th.

Findings from research are consistent with findings from key informant interviews and focus groups conducted for the evaluation. When asked what they see as the primary needs and issues for innovation programs in Western Canada, key informants and focus group representatives highlighted the following:

3.1.1 Alignment of WD Innovation Programming to Western Canada Innovation Needs

Expected results for WD delivery of innovation programming include advanced technology development proven through successful deployment in an operational setting, enhanced financial capacity of businesses to pursue technology development and commercialization, and sales of knowledge-based products/processes/services by businesses. Other immediate outcomes are applied R&D technology development involving validation in a simulated environment, technology commercialization capacity by ensuring prototypes are ready for demonstration in an operational setting, and highly qualified personnel to support commercialization of knowledge-based products, processes, and services. These results aligned with innovation needs and support WD's mandate to grow and diversify the western Canadian economy.

The key informants noted that most of WD's investments in technology commercialization are well-aligned with the needs of businesses given that the investments are targeted directly or indirectly (i.e. with the support delivered through non-profit organizations) at businesses.

Applicants of WDP and WINN funded projects were asked to rate the extent to which there continues to be a need for WD to provide funding for projects that focus on innovation on a scale of 1 to 5 where 1 is not at all and 5 is to a great extent. The perceived need for innovation programming was high and funding recipients gave an average rating of 4.8. Applicants who did not receive funding for their projects provided a rating of 4.6 on the need for WD funding. Some of the reasons given for the need were shortage of capital, the need to diversify the western Canada economy and create strategic long term benefits, build on business clusters and fill gaps in innovation infrastructure.

3.1.2 Gaps in Innovation Programming

The gap most commonly mentioned by key-informants was the limited availability of funding relative to the high demand. As a result, many good projects are rejected. Some WD respondents noted that it was difficult for them to comment on whether the programs are aligned with the needs of businesses, given that staff have little access to first-hand information about the specific needs or major issues faced by businesses.

Other gaps noted by key informants include the need to:

3.1.3 Other Innovation Programs

Several federal, provincial and industry programs provide funding to innovation initiatives in Western Canada. A few of the programs identified through document reviews and key-informant interviews are highlighted below:

  1. The Industrial Research Assistance Program (IRAP) – A National Research Council of Canada program that provides financial support to small and medium-sized enterprises in Canada to help them undertake technology innovation
  2. Centres of Excellence for Commercialization and Research Program – A federally funded program that bridges the gap between innovation and commercialization by matching clusters of research expertise with the business community.
  3. Natural Sciences and Engineering Research Council of Canada (NSERC) – A federal government agency that provides grants to postsecondary students and postdoctoral fellows to research in the natural sciences and in engineering.
  4. Canadian Foundation for Innovation (CFI) – Created by the Government of Canada, the CFI operates as an independent, non-governmental body and funds innovation research by universities, colleges, research hospitals and non-profit research organizations.
  5. Sustainable Development Technology Canada (STDC) – A Government of Canada funded initiative to support the development and demonstration of innovative clean technology projects.

Mechanisms have been implemented by WD to ensure complementarity and coordination of funding for projects to the extent possible. For example, a Memorandum of Understanding was signed between WD and STDC to coordinate funding efforts. IRAP is a traditional grant program focused on early stages of commercialization and is perceived to be complementary to WD. IRAP officers are also engaged in assessment of the technical merit of the WINN applications.

Key informants perceived WD's innovation programming to be more flexible than most other identified programs and noted that WINN is the only government program that is providing repayable contributions specifically focused on commercialization of technology. It was noted that the WINN program is structured in a way to contribute to a variety of costs (capital, salaries, materials and equipment) while other funding programs may be more restricted to specific types of eligible costs.

Funding partners of projects viewed WD programs to be complementary and help augment other programs. Focus group participants indicated that WD funding played a critical role in leveraging funds from other partners. About half of federal and provincial government representatives noted that the presence of WD funding for a specific project may influence their decision to participate in projects by providing funding or other assistance to organizations. Most key informants noted that other similar funding programs overlap with WD to the extent that they are intentionally designed to be stacking programs and facilitate some risk sharing. Equipment grants for research institutions were identified by key informants as one area of potential overlap between WD innovation programming with other funders such as NSERC and CFI.

3.2 Alignment with Government Priorities

The federal government budget 2017 introduced an innovation and skills plan to build Canada as a world-leading innovation economy to create jobs and grow the middle class. The budget supports innovation in key growth industries such as clean technology, digital and agri-food. New measures in the budget are designed to improve access to financing, encourage investment, support the demonstration of technologies and create jobs. This aligns with key expected results identified for WD's innovation programming. Expected results include clean technology, access to financing for businesses, technology development and demonstrations, and the creation of highly qualified personnel to support technology commercialization.

Nearly all key informants perceive WD activities to be aligned with emerging government priorities and expectations. Currently much of program funding for WINN is focused on clean technology which is a major priority of the government.

3.3 Alignment with Federal Roles and Responsibilities

The roles and responsibility of the federal government (and WD) in innovation programming is set out in the federal government's innovation and skills plan. The plan states that to help Canada realize its potential as a global leader in innovation, the Government must ensure that its services best meet the needs of Canada's innovators and job creators. Stakeholders interviewed for the evaluation agreed that WD's innovation programming is aligned with federal roles and responsibilities.

The government (and WD) role in fostering an innovation ecosystem is supported by the OECD innovation strategy presented in 2015[3] at a meeting of the OECD Council in Paris in June 2015. The strategy was well received at the meeting and affected policy developments in many countries. The strategy concludes that governments play a key role in fostering a sound environment for innovation, investing in the foundations for innovation, helping overcome barriers to innovation, and ensuring that innovation contributes to key goals of public policy.


Innovation Canada: A Call to Action (Expert Panel report on Review of Federal Support to Research and Development), 2011
How Canada Performs: A Report Card on Canada 2015 by the Conference Board of Canada
OECD Innovation Strategy 2015 – An Agenda for Policy Action. Presented at the OECD Council at Ministerial Level in Paris in June 2015

Return to the top of this pagetop of page


Section 4: Performance: Achievement of Expected Outcomes

4.1 Performance Measurement Data

The performance measurement system used by WD requires funding recipients to collect and report data on a number of agreed upon performance indicators (usually up to a maximum of five indicators per project). When the projects are approved, targets are established for each indicator. Recipients provide periodic updates to WD on performance indicators and targets during the project and at its conclusion.

The purpose of most of WD innovation projects is to facilitate the development and/or commercialization of new technology. If successful, over the longer-term, most of these projects will contribute to the business revenues, attract new investment, and the create employment. The exercise of analyzing the survey data with data from WD database identified a variety of limitations with respect to reporting on results. These include:

4.2 Innovation Projects Impact

In the survey, funded recipients were asked to rate the extent (on a scale of 1 to 5, where 1 is not at all, 3 is to some extent, and 5 is to a great extent) to which the project resulted in a variety of different types of impacts. The highest rated impact was the development of networks, partnerships, and alliances (4.4). Other highly rated impacts for recipients funded included on-going investment in R&D (4.0), development of new/improved technologies (4.0), adoption of new technologies (4.0), attraction/retention of HQP (3.9), development of technical or business skills (3.9), as well as development of prototypes or new technology demonstrations (3.9).

Where recipients reported that their project generated specific types of impacts, they were asked to quantify the impact (e.g. in terms of the number of new partnerships and alliances developed). Recipients expressed that impacts:

In terms of financial impacts, recipients reported that their projects contributed (or are expected to contribute in the short-term) to further investment of over $590 million, increased investment in R&D totaling $311 million, increased revenues of $177 million by new companies created, and costs savings of $58 million.

4.3  Longer-Term Results

Most significant results from projects are achieved after the end of WD funding. The 12 case studies conducted for the evaluation illustrated significant long term results after project completion. Most targets established for projects examined for the case studies were achieved or exceeded. In most cases, the success of the projects led to additional projects between WD and the organizations. Examples from the case studies include:

The on-going commercial success of projects such as the NeuroArm with the University of Calgary is one example of longer-term outcomes of funded projects. A review of the website for the project shows that the NeuroArm continues to be profiled internationally through conferences, workshops and publications. The NeuroArm have been successfully commercialised and is currently used in neurosurgical procedures in hospitals.

4.4 Incrementality

WD funding accounted for 32% of the total funding for WDP projects and 44% of the total funding for WINN projects included in the evaluation. Funded recipients indicated that most of their projects were dependent upon the availability of funding from WD. Projects were likely to have been delayed, cancelled or reduced in scope in the absence of WD funding. On average, the recipients estimated that there was a 40% likelihood that their project would have proceeded in some form (e.g. reduced or delayed) even in the absence of support from WD.

Over half of WINN applicants who did not receive funding from WD (55%) reported that their projects were implemented but with a reduced scope, a delayed start, or a longer implementation period. Some of the applicants were able to access funding from other sources including IRAP, Growing Forward (Invest Agriculture BC), private capital or private equity, foreign partners and NSERC. At the time of the survey, 7% of projects had been completed, with 52% of projects ongoing. Seventeen percent of respondents said their projects were suspended while another 7% reported that their project had not yet started.

4.5 Client Satisfaction

About 90% of funded recipients expressed satisfaction in their dealings with WD staff. They indicated that WD provided clear direction to them regarding what was expected in a proposal. Most unfunded applicants tended to be less satisfied with the application requirements and criteria as well as in their dealings with WD staff. Reasons for dissatisfaction include:

When asked to rate the extent to which the assistance provided by WD met the needs of their organization on a scale of 1 to 5 where 1 is not at all and 5 is to a great extent, WDP recipients provided an average of 4.9 while WINN recipients provided an average rating of 4.6. In addition to the funding received from WD, the recipients attributed the success of their projects to a strong team, collaborations with partners external to the project, a strong vision and leadership, and organizational support. Some of the factors that were identified as constraining the success of projects included delays in project implementation, changes to the operating environment in which the project was designed (e.g., severe declines in oil prices and other commodity prices), and technical issues.

Return to the top of this pagetop of page


Section 5: Performance: Design and Delivery

In May 2014, WD moved to a call for proposals (CFP) process where applicants submitted project proposals during set time periods based on WD's Strategic Priorities. The CFP system is used for the majority of WD programming areas not just for innovation projects.

The first CFP for WDP closed in May 2014 while the second closed in February 2015. Under the two intakes, WD received 665 WDP applications and approved 111 projects seeking $127 million in total WD funding. Three CFPs have been held to date for WINN: closing in the fall of 2013, the fall of 2014 and spring 2016. Under the 3 intakes, WD received 797 applications and approved 85 projects seeking $96.9 million in total WD funding.

Stakeholders commented that the CFP process is open, transparent and fair process that allows for comparative analysis of projects and better alignment with Government of Canada plans and priorities. After the 2nd intake, improvement were made in areas such as the applications review and approval processes, scoring matrices, and communications

The majority of stakeholders across multiple lines of evidence for the evaluation (key informant interviews, focus groups, surveys and case studies) expressed concern in the following areas of the CFP process:

Other concerns raised on the CFP process by some stakeholders across multiple lines of evidence include:

Suggestions for Improvements to Design & Delivery

Suggestions for Improvements offered by stakeholders include:

Leverage Rate

WD approved funding for the 311 innovation projects totaling nearly $585 million. The WD funding was leveraged with significant funding from other sources including industry, non-profit organizations, and other government departments. The overall project costs totalled $1.79 billion, which means that $2.07 in funding was contributed from other sources for every dollar approved by WD. This compares well with findings from the 2012 Evaluation of Innovation[4] (each department dollar was matched by $2.50 from other contributors in 2012).

Evaluation of Innovation Activity 2012

Return to the top of this pagetop of page

Date modified: