Evaluation of the Innovation Activity

Executive Summary

Introduction

Innovation is one of five program activities supporting Western Economic Diversification Canada's strategic outcome of a developed and diversified western Canadian economy. The goal of the department's innovation activities is to strengthen the knowledge-based economy in western Canada by: 1) fostering both existing resource or manufacturing industries and new/emerging knowledge-driven industries within western Canada; 2) creating linkages among participants in the innovation system; and 3) investing in equipment and infrastructure to support industry and build capacity.

To achieve its innovation objectives, the department invests in six sub-activity areas:

1) Technology adoption and commercialization: promoting technology adoption and commercialization in the marketplace;
2) Technology linkages: increasing connections among innovation system members;
3) Technology research and development: developing new technologies with commercial potential;
4) Community innovation: increasing technological capacity in communities;
5) Technology skills development: increasing the number of highly qualified workers through training; and
6) Knowledge infrastructure: increasing the physical assets and capacity of a cluster.

This report reveals the findings, conclusions and recommendations of the evaluation of the Innovation Activity, comprising of departmental innovation programming and funded projects therein. The evaluation assessed the relevance and performance of the department's innovation programming and examined all projects approved for funding between fiscal years 2007-08 to 2011-12. The methodology included document/literature review, file and database review, analysis of comparable programs, 76 key informant interviews, 50 project proponent interviews, eight case studies, an outcome assessment of comparator organizations and four focus groups.

Eligible recipients for departmental innovation funding are not-for-profit organizations. A total of 202 innovation projects were approved for funding during the evaluation time period. 33 of the 202 projects were under the Western Economic Partnership Agreements, which was undergoing evaluation concurrently with the innovation programming. To avoid interviewing the proponents of the 33 projects twice, they were excluded from the evaluation of the innovation programming.

Relevance – Continued Need for Programming

The sub-activities and program authority terms and conditions allow the department to support a wide range of innovation projects from basic research to market entry. In fact, the department's programming flexibility allows it to fill funding gaps not supported by other programming. The department's funding filled two gaps of particular importance to recipients: the gap between pure science and commercialization and the gap in capital infrastructure support. Interviewees claim there is still a major gap in transferring research findings and associated intellectual property to the private sector in order to facilitate commercialization. Alternative funding appears to be available for some projects, or parts of projects, but not others. About half of the projects would not have proceeded without department funding. However, the evaluation interviewed 10 organizations that had not received department funding and discovered nine had successfully undertaken innovation projects similar to those funded by the department but without department funding, suggesting there are alternative funding sources for some projects.

Relevance – Alignment with Departmental and Federal Government Priorities

The Innovation Activity aligns with the department's strategic outcome of developing and diversifying the western Canadian economy. The department's innovation approach also aligns with the federal priorities as outlined in the 2007 Science & Technology Strategy, the Digital Economy Strategy, and the Review of Federal Support to Research and Development. In addition, the 2011 Speech from the Throne and Budget 2012 confirmed the government's continued support for innovation.

Relevance – Consistency with Federal Roles and Responsibilities

The Innovation Activity is consistent with the federal government's responsibilities of ensuring prosperity and providing funding for societal benefit. The government plays a distinct role in the system of research and development by performing and supporting research for the public good.

Performance - Achievement of Expected Outcomes

For the innovation activity as a whole, most projects appeared to successfully realize at least two, and often more, of the immediate outcomes corresponding to the six subactivities. Although it is too early to assess longer term outcomes, projects are expected to contribute to at least two, and often more, of the intermediate outcomes. Given innovation is, by nature, risky, unpredictable and long term, the high success rate raises questions as to how innovative the projects are. The evaluation found evidence to conclude that the department is following a safe funding approach and that the department's approach is short term in ensuring projects successfully meet predetermined short term objectives.

Success factors included the funding that enabled recipients to undertake new activities and provide new services to support industry, departmental staff support in assisting recipients to craft proposals and establish collaborations and the flexibility in application timelines. On the negative side, there were impediments to success. First, insufficient capital or operating funds made some projects unsustainable over the long term. Second, recipients encountered cash flow problems while waiting several months for reimbursement of eligible expenses. Finally, the department requires that recipients spend their annual budget by fiscal year end, which sometimes caused extra cost and risk as recipients were pressured to obtain leading-edge equipment and other resources within short timelines.

The evaluation found that outcomes of projects did not align perfectly to one sub-activity and there were too many unique indicators. The impact of these issues lies in the inability to assess outcomes by sub-activity and in obscuring the impacts of project outcomes through the use of unique indicators.

Performance - Demonstration of Efficiency and Economy

For the most part, the programming appeared to be economical, efficient and cost-effective. It cost the department one dollar to award and manage $12.80 in innovation transfer payments and each department dollar spent was matched by $2.50 from other contributors. Suggested improvements focused on reducing paperwork and effort created by the project development/application, project review/approval and the accounting aspects of the reporting process. Recipients would like clarification regarding department priorities and the purpose of the department's consultation component of the project development phase. Practices that worked well for the department included the open application and early dialogue processes. The advantage of partnerships with co-funders was that the department could benefit from the sector-specific knowledge of those partners and therefore get a good sense of how the project will benefit the intended industrial sector; partnering also reduces the risk involved in funding innovation. Partnerships bring industry stakeholders together in a non-competitive way, moving sectors forward and permitting greater coverage of industry than direct support to companies. On the negative side, it may be unreasonable to expect co-funders to support truly leading edge innovative projects and it is difficult for recipients to find two aligning funding sources at the same time that have similar priorities and are willing to fund the same type of project.

Possible Improvements

Interviewees suggested the department could streamline its application, approval and accounting/reporting processes. The department could provide proponents with clarification on departmental priorities and the purpose of the department's requirement for applicants to consult with others during the project development phase. Finally, the department could review its requirements for funding advances to alleviate some of the cash flow problems faced by proponents.

The department can focus programming on priority areas while at the same time maintaining some of the flexibility that has allowed the department to fill funding gaps and accommodate regional variations and needs. Interviewees suggested the department consider whether to fund not-for-profit organizations and/or companies, and whether to focus on research and/or commercialization outcomes.

Recommendation

The evidence gathered in the evaluation and the analysis thereof supported the following recommendation:

Recommendation 1: The department should review its program delivery processes to streamline programming administration and clarify departmental innovation priorities.

Introduction

Acknowledgement

Western Economic Diversification Canada (the department) would like to thank all of the key informants, case study participants and survey participants who generously gave of their time and knowledge to take part in the Evaluation of the Innovation Activity. Without their participation and their insights, this report would not have been possible. The evaluators would also like to thank Dr. Stephen Murgatroyd for his wise advice and important contribution as an external innovation expert on the steering committee. Finally, the evaluators acknowledge the work done by Strategic Review Group Inc (the consultants)in collecting key informant interview and survey data.

Background

Innovation is one of five activities supporting Western Economic Diversification Canada's strategic outcome of a developed and diversified western Canadian economy. The department's website states that "innovation is the process of transforming knowledge into new products, processes and services which, in turn, generate new economic benefits. For this process to succeed, a complete system must be available that supports the movement of a new idea from initial concept, through research and development, to a ready-for-market product. A highly developed innovation system, or "cluster", is focused on a specific area of strength and includes:

• Universities, research facilities, industry, government labs, and other "knowledge infrastructure" that develop new technologies and a skilled workforce,
• Early stage venture capital financing to help bring technologies to markets,
• Industry associations and other organizations that link the players in the innovation system,
• Firms capable of developing and adopting new technologies, and that are connected to local and global markets, and
• A business environment that fosters innovation.¹

The department's innovation activities strengthen the knowledge-based economy by: 1) fostering both existing resource or manufacturing industries and new/emerging knowledge-driven industries within western Canada; 2) creating linkages among participants in the innovation system; and 3) investing in equipment and infrastructure to support industry and build capacity.

To achieve its innovation objectives, the department invests in six sub-activity areas:

1) Technology adoption and commercialization: promoting technology adoption and commercialization in the marketplace;
2) Technology linkages: increasing connections among innovation system members;
3) Technology research and development: developing new technologies with commercial potential;
4) Community innovation: increasing technological capacity in communities;
5) Technology skills development: increasing the number of highly qualified workers through training; and
6) Knowledge infrastructure: increasing the physical assets and capacity of a cluster.

Innovation Programming

The department's innovation activities were funded under the Western Diversification Program (WDP) or under one of the sub-components of the WDP. Eligible recipients for departmental innovation funding are not-for-profit organizations. The department's set of innovation funding vehicles are described below.

Western Diversification Program² is the main program through which the department invests in initiatives that enhance and strengthen the economy of western Canada. The program supports innovation by funding the department's activities to strengthen the western Canadian innovation system. As an umbrella program authority, it contains a number of sub-components that fund innovation projects.

Urban Aboriginal Strategy, a sub-component under the WDP authority, promotes independence and opportunities for Aboriginal people living in urban areas. The strategy supported innovation by funding the department's activities to promote entrepreneurship and innovation among Aboriginal people living in urban centres.

Intervac, a sub-component under the WDP authority, funded the construction of a facility (InterVac) located at the University of Saskatchewan. InterVac will be the first facility in North America to focus on pathogens affecting both animals and humans. This initiative supports innovation by funding the department's research and technology development activities and advances the federal interest in vaccine research.

Winnipeg Partnership Agreement, a sub-component under the WDP authority, was a five-year urban development agreement between the governments of Canada, Manitoba, and the City of Winnipeg. The agreement supported innovation by funding the department's activities aimed at building Winnipeg's knowledge-based economy and strengthening its innovation capacity.

Western Economic Partnership Agreements, a sub-component under the WDP authority, are designed to address joint federal-provincial priorities and respond to regional needs that are consistent with national economic priorities. The agreements support innovation by funding the department's activities focused on introducing new products, technologies and services to consumers.

Rick Hansen Foundation, a sub-component under the WDP authority, advances knowledge and research aimed at improving the lives of people with spinal cord injuries. This investment supports innovation by funding research in health related life sciences and technologies.

Throughout the remainder of this document, the set of innovation funding programs and sub-components will be referred as innovation programming. The department spent $70.1 million on innovation programming in 2007-08. The department's most recent Departmental Performance Report (2010-11) indicated that: 1) spending on innovation decreased from $86.9 million (2009-10) to $77.1 million (2010-11); and 2) the department provided $86.8 million to 43 innovation projects that then leveraged $209.8 million from other sources ($2.46 for each departmental dollar spent).

Western Diversification Program Terms and Conditions

Funding, in the form of a contribution or a grant, may be allocated on a payable or non-payable basis. Grants may be used when the assessment demonstrates a low risk project where monitoring is not required through the life of the project, due to the strength of the recipient, and the confidence in the use and value of the required funding. Projects requiring ongoing monitoring of progress and use of funds will be funded as contributions. Contributions can be made to several organizations, including non-profit organizations, educational institutions, and other government departments (federal, provincial, and municipal). A wide range of costs can be covered by the funding, including operational costs, equipment acquisition, and personnel costs.

top of page

Logic Model

Evaluators consulted with innovation programming managers to develop the following logic model.

Evaluation Context

Using the existing WDP authority, the department's innovation programming will be evaluated based on the Performance Measurement Framework of the WDP. The evaluation focused on the degree to which projects achieved their intended objectives and contributed to the development and diversification of the western Canadian economy. This evaluation addressed the relevance and performance of the department's innovation programming. The evaluation covered the fiscal years 2007-08 to 2011-12. The evaluation results will contribute to program renewal and continuous improvement.

Although this is the first evaluation at the innovation activity level, an evaluation of the Western Diversification Program was completed in 2008. As an umbrella program authority, The Western Diversification Program funds all four activities outlined in the department's Program Activity Architecture. It became apparent during the 2008 evaluation that the broad focus of the Western Diversification Program made it unwieldly to evaluate as a whole and that a better approach for this evaluation would be to narrow the focus to the innovation activity and follow up with evaluations of the other activities in subsequent years.

Evaluation Scope and Objectives

The evaluation of the Innovation activity was scheduled in the department's evaluation plan (2010-15) and satisfies requirements of the 2009 Treasury Board Policy on Evaluation. This evaluation of the Innovation Activity included various data collection methods outlined below. The objectives of the evaluation and the core evaluation issues are presented in Table 1.1.

Evaluation Approach and Methodology

This is a theory-based evaluation, which used program theory to guide the evaluation. The evaluation was planned as a quasi-experimental design involving a non-equivalent control group. To maximize the objectivity and relevance of the conclusions, the evaluation was guided by a steering committee containing senior managers and an external innovation expert. Evaluators also sought feedback from program staff throughout the evaluation process.

Evaluation Study Activities

Preliminary Consultations

Preliminary consultations were conducted with regional departmental staff to develop a comprehensive list of projects, the list of key informant interviewees and the case studies. The evaluation framework was reviewed by the evaluation steering committee and senior department management. Through these consultations, some preliminary evaluation information was also obtained.

Documents and Literature Review

Four groups of documents were reviewed as part of the evaluation:

• General Background documentation (e.g. Treasury Board Submissions, documents that describe innovation programming, rationale, theory, etc.);
   
• Departmental reports;³
   
• Program & Policy Documentation (e.g., Departmental Performance Reports, departmental database, project files); and
   
• Literature on innovation programming and best practices.

File Review

Using a customized data extraction template, the evaluation team analysed all information contained in the department's databases (Project Gateway and the GX financial system) as delivered through the department's reporting tool (WD Reporting Centre). The study included all projects⁴ approved for funding between April 1, 2007 and the time the database review was completed in June 2011. A sample of nine projects from the Alberta region was selected for paper file review to: 1) validate the accuracy of the database information; and 2) determine if there was additional outcome information in the paper files that would justify reviewing all paper files in all four regions. The sample contained accurate and complete information, suggesting further file review was unnecessary. This decision assumed similar file completion and accuracy standards across all four regions. 202 innovation projects, totalling $306 million in departmental funding, were approved between April 1, 2007 and June 2011. As of June 2011, 98 (49%) projects were complete or had last claim approved. The majority of projects were approved under the Western Diversification Program (154 projects totalling $176 million in departmental funding) and its sub-component, the Western Economic Partnership Agreements (36 projects totalling $59 million in departmental funding). Approximately two thirds of projects identified either Knowledge Infrastructure (57 projects) or Technology Adoption and Commercialization (70 projects) as the sub-activity (Table 2.1).

In British Columbia, Alberta and Saskatchewan, innovation spending levels were higher in 2007-08 and 2010-11 than in the middle years. Manitoba's innovation spending decreased every year. In the regional business plans, each region highlighted their key sectors, or clusters, of funding focus; however, the data show a similar distribution of projects across sectors with the exception of life sciences (54 projects) which claimed approximately 44% of all funding. Other sectors receiving more than 5% of funding included: other technologies (21% of funding ), multi-sector (10% of funding ), enviromental technology (7% of funding) and information technologies (6% of funding).

Mapping Analysis

The steering committee chose to forego a comparative analysis of similar programming because western Canada is a unique ecosystem requiring unique innovation programming. As such, there is no meaningfully comparisons to the innovation programming in western Canada. Instead, the analysis consisted of a "mapping" of innovation-related programs to compare the department's programming to the spectrum of programs available to western Canadian organizations pursuing innovation projects. The mapping analysis also included a summary of five reports, written within the last seven years, that looked at barriers to innovation and commercialization in Canada and compared Canada's ecosystem with international innovation ecosystems.

The programs selected for the analysis were identified by proponents when asked "what other funding organizations do you receive funds from?" This produced a list of:

1. Regional (Provincial) Programs: a summary of existing information and studies on available Innovation programming in each Region.
2. Federal Programs: a summary of existing information and studies focusing on the four largest federal organizations and programs⁵ funding research and development, as per Innovation Canada: A Call to Action.
3. Report Summaries: review of five reports⁶ of studies commissioned by the Federal Government within the past seven years to look at the state of science and technology, innovation, and commercialization in Canada (and internationally).

Interviews with Key Informants

The consultants developed and pre-tested the questionnaires and then conducted individual or group key informant interviews by telephone. The consultants completed 73 key informant interviews with 76 key informants including:

• 24 proponents and 10 representatives of non-recipient organizations. These participants formed the two comparison groups for the Outcome Assessment and their selection is detailed in the Outcome Assessment section below;
   
• 5 declined applicants: two from Saskatchewan and one from each of the other three regions. To select the sample, the list of 20 declined/unfunded applicants was stratified by region and two applicants were chosen at random from each region. Five declined/unfunded applicants agreed to participate;
   
• 14 interviews with 17 departmental staff and management. The interviewees were those department management considered to be key staff and management involved in innovation programming;
   
• 11 representatives of other funding agencies: 2 agencies in British Columbia, 4 in Alberta, 1 in Saskatchewan, 2 in Manitoba, and 2 that are federal agencies. This group included co-funders as well as other organizations that fund innovation projects. Interviewees were identified by departmental program managers as being representatives of co-funding agencies that the department most frequently works with or other Regional Development Agencies with a similar mandate. In total, 12 representatives from other funding agencies were contacted for an interview but one was non-responsive; and
   
• 9 interviews with experts in Canadian innovation. These individuals were identified during interviews and meetings as people who are knowledgeable about innovation in Canada and should be consulted as part of the Evaluation. In total, 12 Innovation experts were contacted and 9 agreed to participate.

Proponent Survey

There were 140 proponents associated with the 202 innovation projects within the scope of this evaluation. 33 of the 202 projects were funded under the Western Economic Partnership Agreements, an initiative undergoing evaluation concurrently with this evaluation. Because these 33 proponents would be contacted for the other evaluation they were excluded from the survey universe. The survey universe also excluded the 24 proponents selected for a key informant interview and the proponents associated with the 8 case studies, leaving 108 potential survey participants. The survey successfully contacted 50 (46%) proponents of 62 projects. The survey participants were from all four regions and all six innovation subactivities. One project was funded under the Rick Hansen Foundation and the remainder were funded under the Western Diversification Program. Although proponents from the other innovation funding vehicles did not participate, their performance measurement information was collected through the file review and only three of the eleven projects funded under those components are complete. All three complete projects were funded under the Winnipeg Partnership Agreement.

The consultants developed and pre-tested the questionnaire. Each proponent in the survey universe was sent the survey by email. Proponents were provided the opportunity to complete the survey by filling out the Word document attachment or over the phone. Each proponent in the survey universe was contacted up to four times requesting that they complete the survey.

Case Studies

Each of the four regions was asked to suggest two of their funded innovation projects for case study. The eight projects were to be complete or near completion and represent a range of outcomes. In total, evaluators interviewed 16 stakeholders of eight case studies. Specifically, the eight case studies included:

• Two projects from each region;
• Projects that were indicated in the departmental database as being complete or near completion: three had first claim approved, one had final claim approved and four were complete;
• A mixture of project types: three were Technology Adoption and Commercialization, two were Knowledge Infrastructure, two were Technology Research and Development and one was Technology Skills Development;
• A mixture of proponent types: five were university-based, two were not-for-profit corporations, one was a not-for-profit society, one was a not-for-profit applied research, development and testing organization;
• A range of sectors: two were life sciences, four were multi-sector, one was other technology and one was environmental technology; and
• A range of sizes in terms of departmental funding: three were approved for $200,000 to $350,000, three were approved for between one million and two million dollars, and two were approved for approximately three million dollars.

In conducting the case studies, the evaluation team collected all background information and documentation summarizing project implementation, impact and outcomes to date. Site visits were completed for seven of the eight case studies to observe the project, interview representatives involved with the projects and review any other relevant documentation. In one case, a telephone interview was completed instead of a site visit because the proponents had closed their office in western Canada.

Outcome Assessment

The Outcome Assessment was intended to provide additional evidence on attribution and program impacts by selecting two groups for comparison: one group of proponents and one group of organizations who had not received department funding (non-recipients).

1. Selection of 24 proponents: 81 proponents satisfied the initial selection criteria: 1) project had received at least $100,000 in funding; 2) the project began between April 2007 and December 2009; and 3) the final report was available to provide complete performance data. The 24 participants were selected from the 81 proponents to represent all regions, subactivities and types of organizations (academic, incubator, association). When possible, proponents with significant experience with the department (i.e. more than three projects) were chosen over less experienced proponents to maximize chances of detecting important differences when compared to non-recipients.
2. Selection of 10 non-recipients: the consultants identified an initial list of 55 non-recipients by reviewing departmental business plans and directories of associations. The list was reviewed by regional departmental officers and 33 were confirmed as non-recipients of any departmental funding. These organizations were contacted for an interview and several were then removed from the list because they were in the process of developing a proposal, awaiting a decision on a submitted proposal, recently declined for departmental funding or a for-profit organization. In the end, the group consisted of 10 organizations from three regions: British Columbia (4 organizations), Alberta (3 organizations) and Saskatchewan (3 organizations). None of the three organizations contacted in Manitoba agreed to an interview. The 10 selected organizations satisfied the following criteria: 1) not-for profit organization; 2) mandate focused on economic development; 3) aligned with regional sectoral priorities (i.e. life sciences, information and communication technologies); 3) located, or have significant presence, in the western provinces; 4) never received funding from the department's innovation programming.

Focus Groups

The consultants conducted one focus group in each of the four regions in February 2012. All proponents and co-funders who were interviewed were invited to attend the focus group in their region. Departmental participants were selected by the Director General of Operations in each region. There were 48 focus group participants in total: 33 proponents, 4 co-funders and 11 departmental staff. Each focus group contained between 10 and 15 participants.

A consultant presented the field research findings at the focus groups and then facilitated group discussions. The objectives of the focus groups were to review and validate the field research findings and explore ways to enhance the effectiveness and efficiency of similar programming in the future.

Limitations of the Methodology

Project File Review: We extracted the data for the file review using the department's reporting tool, the Reporting Centre. We found the Reporting Centre to be inflexible and not user-friendly, spending weeks verifying data, re-running reports and manually adding/removing data. The performance measurement data, in particular, needed significant manipulation before it was suitable for analysis. In some instances, it was easier to extract information directly from the department's databases (Project Gateway and GX) rather than the Reporting Centre.

Case Studies: Case studies are nonprobability samples of projects chosen for a specific purpose. In this evaluation, the case studies were chosen to represent a range of outcomes. As with any interview process, the case studies may be biased according to respondent experience and recall. Some respondents, for example, joined a project after it had started and could not comment on what had occurred in the project's early stages. Finally, there was the potential for measurement error related to using questionnaires that were not rigorously tested for validity or reliability. For example, the meaning of terms such as "technology clusters" and "technological capacity" were open to respondent interpretation. In acknowledgement of the potential for bias, the case studies in this evaluation serve as one of several lines of evidence.

Key Informant Interviews: Twenty of the 73 interviews were conducted with individuals not directly involved with the department such as experts, representatives from non-recipient organizations, and representatives from other Regional Development Agencies. The level of knowledge and understanding of the department's innovation programming varied among these interviewees and some could not address every question.

Proponent Interviews and Surveys: It was difficult locating some proponents because they had left the organization that they were with when they received departmental funding. In these instances, the organization was asked for forwarding contact information or for the contact information of someone currently within the organization who had taken over the project and could speak about it. Also, it was difficult locating other proponents because their contact information was incorrect. In these instances, up-to-date contact information was obtained from the organization or its website. Finally, some proponents felt it was too early in their project to rate the achievement of outcomes (particularly immediate outcomes). Proponents were then asked to predict longer-term outcomes for their project.

Focus Groups: The focus group participants were asked to validate and interpret evaluation findings. The nature of focus groups implies that the main comments do not necessarily reflect the opinions of all participants on an issue.

Outcome Assessment: The objective of the outcome assessment was to compare the success of projects funded by the department to projects undertaken by non-recipient organizations. Therefore, the two groups should be as similar as possible on, for example, criteria such as their eligibility for departmental funding. Eligibility for departmental funding was based on the department's risk assessment. Because the risk assessment was not applied to non-recipients, the capacity and sophistication of non-recipients may not match that of funded proponents. Comparable performance data was often not available for non-recipient and funded projects, impeding comparison of performance between the two groups. A further impediment to the analysis was the small size of the non-recipient sample and the lack of participation from Manitoba.

Attribution: Determining the value added by the department's innovation programming is challenging over the long term because outcomes such as a stronger knowledge-based economy are the result of many factors working together. This evaluation uses contribution-focused analysis to infer WD's role in achieving strategic outcomes leading to developing and diversifying the western Canadian economy.

Relevance

Continued Need for Programming

Economic data supports the continued need for innovation programming. Innovation drives productivity which, in turn, determines the wages and standard of living of Canadians. "We have a prosperity gap in Canada – a gap between our potential and actual economic results. This prosperity gap is a productivity gap, and the productivity gap is an innovation gap."⁷ The prosperity gap accounts for estimated annual losses of: 1) $112 billion in tax revenues for all three orders of government; and 2) $12,900 in after-tax disposable income for each Canadian household.

Furthermore, temporal trends reveal Canada's productivity performance is decreasing: "Since 1984, relative labour productivity in Canada's business sector has fallen from more than 90% of the U.S. level to about 76% in 2007. Over the 1985-2006 period, Canada's average labour productivity growth ranked 15th out of 18 comparator countries in the OECD."⁸

The department funds research and development, an essential input into the innovation system and one that Canadians see as key to future prosperity. A random telephone survey of 1203 Canadians asked the question: "how important do you think research and development is to Canada's future prosperity (on a scale from 1=not important at all to 10=very important)." Seventy-five percent of Canadians in every region of the country felt research and development was important to future prosperity (rating of 8,9 or 10).⁹

The focus groups indicated that the department's programming flexibility allows it to fill funding gaps, such as funding of equipment for testing and demonstration, not supported by other programming. In fact, the sub-activities and program terms and conditions allow the department to support a wide range of innovation projects from basic research to market entry. Key informants stated that the department's innovation programming was filling two distinct funding gaps: 1) bridging the gap between pure science and commercialization: most other agencies focus on research; and 2) supporting capital infrastructure: the department is the only funding agency providing this type of support in British Columbia and Saskatchewan. While there were alternative funding sources for certain projects or elements of projects, there was no substitute for departmental funding for approximately half of the projects. They would not have proceeded without departmental support. Specifically, these projects involved the development of information sharing networks, support for incubators and accelerators, the establishment of centres for manufacturing and prototype development, small equipment purchases, and the installation of testing and demonstration equipment. The case studies support the finding that innovation funding was moderately to highly incremental. Of the eight case study projects, five were found to be totally incremental (that is, the project would not have proceeded in the absence of departmental funding) and three of the projects would likely have proceeded, but only after a delay or with a reduced scope (partially incremental). The incrementality of the funding was confirmed in the focus groups. The focus group participants also felt that, despite the department's focus on commercialization, there is still a major gap in transferring research findings and associated intellectual property to the private sector in order to facilitate commercialization.

In contrast to the funded proponents, nine of the ten non-funded organizations had successfully undertaken innovation projects similar to those funded by the department but without departmental support. When asked why they did not approach the department for support, they indicated: 1) they already had funding (3 organizations); 2) they did not know about the department and the support it provides (3 organizations); and 3) they did not believe their project aligned with departmental support (4 organizations).

In summary, most key informants (all proponents, most co-funders and experts) agreed that the department's innovation programming is fulfilling a need and there is a continued need for the funding. However, there were mixed messages related to the usefulness of the funding, particularly going forward. Alternative funding was available for elements of some projects but not others. Focus group participants indicated the department was the "first stop" for funding for some proponents while it augmented funding for other proponents. Some key informants (including proponents, co-funders, experts and staff) felt the department could move toward a long-term funding strategy for priority sectors. Departmental managers indicated the department is currently attempting to implement a long-term strategy for sectors; however; it may not yet be fully achieved in some regions. The focus groups were divided: many would like to see a broader distribution of the types of projects the department funds and reduced focus on infrastructure while others believed the department should develop a more focused strategy for its innovation support.

Alignment with Departmental and Federal government priorities

The Innovation Activity supports the department's strategic outcome of developing and diversifying the western Canadian economy. The 2010-11 Departmental Performance Report states that the department's "approach to innovation aligns with the overall federal approach, which is outlined in the 2007 Science & Technology Strategy¹⁰ and in recent initiatives, such as the development of the Digital Economy Strategy¹¹ and the Review of Federal Support to Research and Development¹². "The 2011 Speech from the Throne confirmed the Government's continued support for innovation over the next five years as part of its plan to "create the right conditions for growth and job creation".¹³ Most recently, the government committed to new supports for innovation in its Budget 2012.

Key informants indicated that annual regional and corporate business plans identify priorities that guide project selection. Although regions tend to interpret priorities differently, the Minister is responsible for final project approval thereby bringing some consistency to the process.

Consistency with Federal Roles and Responsibilities

Innovation and productivity determine the living standards of Canadians. The federal government is responsible for: 1) ensuring prosperity across all regions of Canada; and 2) filling funding gaps in research and development that can benefit society. "Business, government, higher education sector and non-profit organizations all play a part in the system of research and development. Their research may be driven by different motivations, but they all contribute to the advancement of knowledge and well-being of Canadians. While their activities overlap, their roles are distinct...government and private non-profit organizations perform and support research for the public good."¹⁴

Key informants had mixed opinions on the department's suitability to fund innovation. Some key informants felt the department is more politically neutral than the provinces and therefore able to focus on growing the economy and targeting funding towards meeting unique regional needs. Other key informants disagreed, stating that innovation requires a level of risk and a tolerance for failure that contradicts the government's seemingly risk-averse approach to funding.

Performance: Achievement of Expected Outcomes

General Findings

Overall, most of the interviewed proponents believed their projects successfully contributed to at least two, and often more, of the immediate outcomes corresponding to the six subactivities¹⁵. Although it is too early to assess longer term outcomes, projects are expected to contribute to at least two, and often more, of the intermediate outcomes.¹⁶ The case studies generated a wide range of community impacts including development of skilled trades or researchers, spin-off projects, improved agricultural operations, environmentally responsible industrial processes, economic diversification, improved mining productivity, penetration of new technologies into the economy and improving quality of life of patients.

Focus group participants and key informants identified departmental funding as a critical factor to the success of the projects. The support from department enabled proponents to undertake new activities and provide new services to support industry. The informants also noted the critical role of departmental staff support in assisting proponents craft proposals and establishing collaborations. Another factor mentioned in the focus groups was flexibility in application timelines that allowed proponents to approach the department at any time with an idea that could then lead to a proposal and a funded project.

The two success factors most commonly identified by proponents were: 1) access to capital; and 2) access to equipment for technology development and testing. Additional success factors mentioned by the case studies included: partnerships with the department, communities, industry, government; and good project management, teamwork and vision.

Proponents faced cash flow problems and some projects were not sustainable over the long term.

Conversely, the greatest impediment to success for proponents across all regions and subactivities was insufficient capital or operating funds (40%). One common factor determining success among the case studies was ongoing operational funding: the more successful case studies had managed to secure ongoing partnerships and operational funding whereas the others had not. Economic factors such as the fluctuating dollar and rising construction/labour costs also hindered the progress of some of the case studies. One of the two other impediments discussed in the focus groups was the requirement that a proponent spend their annual budget by fiscal year end (March 31), often leading to extra work, cost and risk because the window of time between approval and March 31 was insufficient to obtain leading-edge equipment and other resources quickly. In one instance, for example, a proponent received approval for funding in December and had to spend the annual budget in three months. In another case, a proponent was pressured to order and receive equipment by year end, then the equipment sat in storage until the facility was built and by that time there were technical difficulties with the equipment. The other impediment was related to cash flow: the department reimburses eligible expenses and it can take several months to receive funds after submitting a claim. This reimbursement process differs from most other funding organizations that provide up-front funding with year-end reconciliation.

Performance Measurement

Innovation Activity

The department tracks innovation activity-level outcomes through the following three performance indicators.

1. Total western Canadian university income from the Commercialization of Intellectual Property. This indicator has been on a decreasing trend since 2006, dropping from $27-$28 million (2003-2006) to $22.9 million (2007) and $16.5 million (2008). The target of $28.1 million was not met. The 2010-11 Departmental Performance Report indicated that the decline could reflect inconsistent reporting practices at universities.
    
2. Business Expenditure on Research and Development as a percentage of Gross Domestic Product. This indicator hovered around 0.5%, unchanged from 2004. The target of 0.55% was "mostly met".
    
3. Employment in Natural and Applied Science and Related Occupations as a percentage of Total Employment. The target for this indicator was 7% and the performance status could not be determined due to unavailability of data.

The usefulness and quality of these three indicators is questionable because there is a three-year lag in availability of the data for the first two indicators and the third indicator is based on the census which is done every five years. The lack of timely data on these three indicators could impede the department's ability to track its progress. The first indicator may be unreliably reported. The apparent stability of the second indicator suggests it is unresponsive to the impact of the department's activities over time. Furthermore, a target of 0.55% for "Business Expenditure on Research and Development" seems somewhat low considering the 1% target for Canada and Canada's international competitors.

Several department staff observed that a significant number of innovation projects could also be classified as business development projects under the Business Development Activity. We examined the 242 unique indicators of the 98 complete/nearly complete innovation projects to determine how many were similar to business development performance indicators. For example, one unique indicator of an innovation project was "# of businesses launched", which was comparable to the business development indicator of "# businesses created or maintained or expanded". In total, we found 15 (6%) unique innovation indicators that could be measuring business development outputs/outcomes, which implies that any significant overlap between the two activities is not captured by the performance measurement system.

Innovation Sub-Activities

There are 24 Performance Activity Architecture performance indicators (standard indicators) corresponding to the six innovation sub-activities (Table 4.1). Some of the standard indicators were rarely used suggesting they may be irrelevant.

In addition to the 24 standard indicators, there were 442 unique indicators for the 202 projects. A 2009 study of two innovation sub-activities¹⁷ suggested that the department: 1) reduce overlap between sub-activities by deciding what type of projects belong in which sub-activity; and 2) reduce the number of unique indicators. These issues prevent accurate outcome assessment by sub-activity and lead to measurement errors because unique indicators cannot be aggregated for assessing and reporting outcomes.

The importance of these issues led us to examine the 242 unique indicators of the 98 complete/nearly complete projects and found:

• Overlap between subactivities: assessed by estimating the number of unique indicators that could be measuring an outcome from a different sub-activity. Findings: 136 (56%) of the 242 unique indicators could be measuring an outcome from a different sub-activity. For example, 53 (65%) of the 82 unique indicators of "Knowledge Infrastructure" projects measured outcomes corresponding to other sub-activities. Furthermore, when rating project success, every proponent claimed their project was very successful in achieving outcomes related to at least two subactivities. Key informants also commented that projects did not allocate perfectly under one sub-activity, but rather overlapped with multiple sub-activities, leaving staff to decide which sub-activity to use and resulting in inconsistent allocation across regions.
   
• Too many unique indicators: assessed by estimating the number of unique indicators that could be replaced by standard indicators. Findings: standard indicators could have replaced 31 (13%) of the 242 unique indicators. For example, 11 unique indicators under "Technology Skills Development" measured training and therefore overlapped the standard indicators. Across all sub-activities, 54 unique indicators measured training and could have been replaced by standard indicators.

It therefore seems the performance issues identified in the previous study continue to challenge performance measurement. The impact of the issues lies in the inability to assess outcomes by sub-activity and in obscuring the impacts of project outcomes through the use of unique indicators. Key informants agreed that the success of projects was not fully captured and claimed performance indicators were primarily being used as milestones to track progress.

According to the final reports on the 98 completed projects, 87 (89%) met or exceeded their performance expectations. Of the three complete projects funded under the Winnipeg Partnership Agreement, two met or exceeded expectations.

The high success rate raises questions as to how innovative the projects actually are. Innovation can be defined as "new or better ways of doing valued things"¹⁸ and is, by nature, risky, unpredictable and long term. An estimated 80% of innovative attempts will fail and a high success rate implies a safe funding approach that lacks ambition.¹⁹ The evidence implies the following two conclusions:

1. The department appears to follow a safe approach. Focus group participants felt the department tends to fund projects focused on infrastructure rather than people/resources because infrastructure is less risky. The department's approach to funding is safe in the sense that every project undergoes a risk assessment using a risk review tool that produces a categorical risk score of low, medium or high for each project. Our inspection of the risk scores for the innovation projects revealed only two with high risk scores.
    
2. The department's approach focuses on the short term. The department's approach appears to ensure projects successfully meet predetermined short term measures. Almost all complete projects met or exceeded their performance measurement expectations. The majority of the eleven projects and four case studies that did not meet their performance measurement expectations could not yet be classified as failures because they encountered legitimate delays and were expected to meet their objectives eventually. Focus group participants observed that the department seems to be more comfortable in being involved earlier rather than later in the innovation cycle, even though the real challenges lie in the later phases and in trying to get technologies out into the market.

Expected Result - Innovation: a stronger knowledge-based economy

Description: facilitate the West's transition to a knowledge-based economy.

Literature on the role of innovation in strengthening the knowledge-based economy and achieving a diversified and developed regional economy

Literature supports the positive impact of innovation on the knowledge-based economy: "Through innovation, new knowledge is created and diffused, expanding the economy's potential to develop new products and more productive methods of operation."²⁰ The link between innovation and economic development is also well-established, although "economic benefits accrue only when a technology or technique is brought to the marketplace where it can be sold to generate income or applied to increase productivity." ²¹

Evidence that innovation programming is strengthening the knowledge-based economy and achieving a diversified and developed western Canadian economy

Projects funded by the department's Innovation programming each align to one of six sub-activities under the department's Innovation Activity. Each subactivity contributes to the Innovation Activity's ultimate outcomes of 1) a stronger knowledge based economy and 2) the development and diversification of the western Canadian economy. The expected results and distribution of the 202 innovation projects is summarized below:

• Technology adoption and commercialization: 35% of projects
  • Expected Result: An increase in the number of technologies developed in research institutions that have commercialization potential and an increase in technologies adopted by existing firms.
  • Description: support small- and medium-sized enterprises to increase the commercialization and adoption of technologies, products, processes and services in the marketplace.
• Technology linkages: 12% of projects
  • Expected Result: Increased connections and synergies among innovation system members.
  • Description: invest in networks, industry associations and other initiatives or events designed to build connections and synergies among the players of the innovation system.
• Technology research and development: 10% of projects
  • Expected Result: Applied research and development leading to technologies with commercialization potential.
  • Description: support applied research and development that has the potential to lead to a commercial product or service.
• Technology skills development: 7% of projects
  • Expected Result: Increase in training, education and skills building of highly qualified people (HQP)
  • Description: support development of specialized skills related to technology and commercialization through targeted training and through internships, which enhance the skill level of scientists and their ability to respond to industry needs.
• Community innovation: 8% of projects
  • Expected Result: Increased technological capacity in a community.
  • Description: support the innovation capacity of communities across western Canada through support for planning studies, skill development in remote and northern areas, and enhancing knowledge infrastructure.
• Knowledge infrastructure: 28% of projects
  • Expected Result: Increase in physical assets for research and development or training.
  • Description: support knowledge infrastructure such as research equipment or buildings, which are key for research and development or specialized training.

The expected results listed above are the short-term immediate outcomes. The intermediate outcomes included: a strengthened innovation system in Western Canada; increased technology development, adoption and commercialization; further development of technology clusters; people complete and incorporate training; research is shared and used; and western firms innovate to create wealth. The long term outcomes included: a stronger knowledge-based economy; and development and diversification of the western Canadian economy.

The programming realized a wide range of short term impacts

For the innovation activity as a whole, projects appear to be strengthening the knowledge-based economy by creating new products, services, information and knowledge available to industry. Proponents believed their projects had successfully achieved short term outcomes, the greatest impact being on training and skills development. The least impact (rating of 3.6 out of 5), was on the adoption of new technologies, which is the outcome associated with 35% of projects and a focus of the department's programming. The relatively lower rating for this outcome could be because it was too early in the project lifecycle to achieve commercialization outcomes. In fact, over the intermediate term, proponents expect projects to significantly contribute to several of the outcomes including increased technology development, adoption and commercialization.

Interviewees also largely believed that projects were diversifying the western Canadian economy. Some stated that their projects would benefit several sectors: for example, proponents working with technologies in the information and communication technologies and genomics sectors stated that advances in these sectors can enable new product offerings in other sectors (i.e. health). Other projects focused on diversifying a specific sector: for example, moving the forest products industry beyond primary production into more value-added products. A few proponents stated that they did not believe that their project would have a significant impact on diversifying the economy because they were undertaking projects within traditionally mature or strong western Canadian sectors such as the oil and gas, and agricultural and food supply industries.

Projects increased the education, training, and skills of highly qualified personnel either through formal training or through an informal transfer of knowledge. However, the focus groups indicated: 1) the training of highly qualified people seems to be more of a priority in some regions than others; and 2) there is some confusion as to whether the gaps are in technical training/competencies or business skill or some combination of both.

Unintended Impacts

Two distinct unintended positive impacts were identified. However, no unintended negative impacts were identified consistently.

The first unintended impact was an increased coordination among stakeholders. Again, most credited the department's position as a neutral party and departmental staff efforts as a liaison or partnering broker in identifying and creating more collaborative arrangements early in the process.

Secondly, departmental support increased the credibility of the organization, thereby raising their national and international profile. This resulted in other spin-off benefits such as additional investment from other funders and increased ability to attract highly qualified people to the organization.

No unintended negative impacts were consistently identified by interviewees. However, a small number of interviewees did state that some organizations might develop a reliance on departmental support, and it would be a hardship if departmental priorities changed.

Performance: Demonstration of Efficiency and Economy

Economy

For the most part, the programming was well-structured but process changes could have improved effectiveness.

For the most part, proponents believed the department was achieving its intended outcomes in an economical manner. Proponents, key informants and focus group participants commonly suggested the department could operate more economically by reducing paperwork and effort required for project development/application, project review/approval and for the accounting aspects of the reporting process. Focus group participants also questioned the purpose of the department's consultation component of the project development phase: to some proponents, it appeared to be an artificial exercise with an unclear goal.

Design of the programming

Most proponents approved of the design of the innovation programming, particularly its flexibility in comparison to other programs that have very specific and limited criteria. Some proponents felt the programming could be improved by decreasing approval times and allowing for the alteration of projects during the life-cycle. Approval times were somewhat long: 24% were within three months and 38% were more than six months (five of these were more than two years). Some practices that were working well for the department and some possible improvements included:

• The open application process and early dialogue had some advantages. An open application process uses flexible application timelines which allow proponents to apply for funding at any time. The department currently follows an open application process which proponents consider a key programming success factor that allows for early dialogue with the department to shape the project. The department could improve the effectiveness of the early dialogue by clarifying departmental priorities and focus for proponents so they can align projects. One disadvantage of the open application process is that it can lead to a short window of time between project approval and March 31, the date at which the proponent is required to have spent their annual budget. Although proponents can request a reprofiling of funds from one fiscal year to the next, it is often not possible because the budget is fully committed for the upcoming year.
   
• Partnerships with co-funders had advantages and disadvantages. The department currently partners with other funders. Key informants and focus group participants saw several advantages to partnering. Partnering allows the department to benefit from the sector-specific knowledge of those partners. This provides the department with a strong indication that the project will benefit the intended industrial sector and reduces the risk involved in funding innovation. Funding not-for-profit organizations in collaboration with co-funders brings industry stakeholders together in a non-competitive way. This creates synergies within and across sectors, moves sectors forward and permits greater coverage of industry than direct support to companies. Co-funders provide a good proxy for the relevance of a project. There are also some disadvantages. It may be unreasonable to expect co-funders to support truly leading edge innovative projects and that these proponents spend time pursuing additional funds instead of undertaking their projects. In order to match funding, proponents must find two aligning funding sources at the same time that have similar priorities and are willing to fund the same type of project.
   
• Coordination of programming could enhance regional strengths. Focus groups participants felt that regional organizations that promote innovation could be coordinated with one another and that the department was well positioned to bring organizations together to coordinate priorities and build upon regional strengths.
   
• Funding advances could have helped with cash flow problems. Many funding organizations provide up-front funding with year-end reconciliation. Because departmental funding is in the form of contributions rather than grants, eligible expenses are reimbursed, which can take several months and cause significant cash flow problems and hardship for proponents. The department can advance money, however the proponent must demonstrate need, which is difficult under the department's current set of requirements for advances. Only 31 (15%) of the 202 projects received any advance funding and the proportion appears to decrease over time with only 10% receiving advances since 2009. This raises questions as to the accessibility of advance funding.
   
• Funding for ongoing operational costs. Thirteen key informants (proponents, staff, co-funders and non-recipients) indicated the department could provide ongoing funding for operational costs of program-based organizations. Focus group participants were divided on the benefit of ongoing funding: some felt it would create a dependence on departmental funding while others felt programs need ongoing funding for sustainabililty and there is no other source of funding.
   
• Support for not-for-profit organizations versus companies. Eight key informants (proponents, staff, experts and co-funders) believed the department should support companies directly and three of the innovation experts emphasized that the farther away the intervention is from industry, the less the impact on industry. Some focus group participants agreed that greater outcomes would be achieved if the department supported companies directly. However, direct funding to the private sector is risky and a better option would be to provide funding through, for example, not-for-profit commercialization groups because these organizations have worked with the private sector and therefore: 1) are in a position to assess the risks and "weed out" potentially unsuccessful companies; and 2) understand the significant time and effort required in coaching, mentoring and training. Some focus group participants disagreed with direct department support to companies because it would overlap with other programs and department officers would require specific training to support companies effectively.

Efficiency

Administrative Efficiency

As a measure of administrative efficiency, the department's total dollars spent on transfer payments for innovation ($72.8 million) was compared to total operating expenses for innovation ($5.7 million)²². The comparison showed that it cost the department one dollar to award and manage $12.80 in innovation transfer payments. The department's efficiency compares favourably with other departments' grants and contributions programs reporting $3.63²³, $6.10²⁴ and $11.1²⁵ in transfer payments per one dollar cost to the department; however, these measures reflect a wide variety of factors and may not be directly comparable to the innovation programming.

The department's operating resources included 28 full time equivalent (FTE) staff in 2010-11, down from 33 FTE in 2009-10.

For the most part, proponents believed that the department was undertaking activities in an efficient manner. The two most common suggestions for improvements were: 1) improve the approval process in terms of the number of proposal iterations required, the unpredictability of approval, and timeliness to receive a decision; and 2) increased flexibility in departmental accounting and reporting processes.

On average, key informants also felt activities were undertaken in an efficient manner. Some proponents were operating under the incorrect perception that the department does not have programs. The proponents stated that, since the department does not have specific programs but funds specific projects instead, the department can be efficient, focused, directed, and clear in what they are asking for. Those key informants who provided a high efficiency rating credited departmental staff for facilitating an efficient process through the development of a strong working relationship early on in the process. Other interviewees provided two reasons for providing a lower efficiency rating: 1) the slow approval process that lacked transparency; and 2) onerous financial or performance reporting requirements.

Cost-Effectiveness

Although most projects required co-funders, there were three exceptions - two projects funded under the Western Economic Partnership Agreements and another funded under the Western Diversification Program. The Western Economic Partnership Agreements allow the department to fund entire projects. The third project, funded under the Western Diversification Program, funded equipment costs that could not be covered because of cost overruns during the construction of two facilities. The facilities were nearing completion but could not function as intended without the equipment. On average, the department funded 29% of project costs. Department funding was instrumental in attracting some funding from other sources (Table 5.1). Table 5.2 summarizes the department's contribution to the total project costs and the dollars contributed by other funders for each departmental dollar invested (dollars leveraged). Although leveraging varied by program, overall, each department dollar was matched by $2.50 from other contributors. A leveraging ratio of $2.50 compares favourably with the $1.44 leveraged by other innovation programming²⁶.

Mapping Analysis Results

It is difficult to place the department's programming within the spectrum of all innovation programming in Canada, partly because there are approximately 500 federal and provincial programs that fund research, technology or firm development in Canada²⁷ and partly because there is limited evaluation information on the programs. However, the mapping analysis found significant regional variation in both the number and focus of other programming open to proponents funded by the department. In British Columbia there were four notable innovation funding organizations focusing primarily on funding specific geographic areas of the province or specific sectors of technology. In Alberta, there were six funding organizations and the focus was on commercialization. In Saskatchewan there was three funding agencies and a strong sectoral focus on agricultural bioproducts. Manitoba has recently undergone a change towards consolidating its provincial support for innovation.

Best Practices and Possible Improvements

Bringing together the results from all lines of evidence collected under this evaluation yields the following list of best practices and opportunities for improvement.

• Best Practices. Practices that seem to be working well for the department and its proponents include the open application process, the department's early dialogues with proponents and the department's partnerships with co-funders.
• Possible Improvement: Program Delivery. Focus group participants and proponents suggested opportunities to simplify program delivery to save departmental resources and reduce client stress. The department can: review its requirements for funding advances to improve accessibility to proponents; reduce the time and paperwork involved in the application, approval and accounting/reporting processes; and provide proponents with clarification on departmental programming, priorities/focus and the purpose of the consultative process.
• Possible Improvement: Programming Focus. The department can improve its innovation programming by focusing on priority areas while at the same time maintaining some of the flexibility that has allowed the department to fill funding gaps and accommodate regional variations and needs. Interviewees suggested possible areas of focus, such as whether to fund not-for-profit organizations and/or companies and whether to focus on research and/or commercialization. Commercialization likely requires a longer term, riskier approach than the current approach. In fact, in its own research, the department identified a funding gap in access to risk capital and a potential need for government involvement in technology commercialization for firms in Western Canada.²⁸ Other literature cautions, however, that the issue is more complicated than just increasing access to risk capital. The innovation expert member of our steering committee highlighted research showing that commercialization is most likely to occur when there is a 3:1 ratio of private to public investment in research and development; the corresponding ratios of private:public investment in western Canada have changed little since 2002-2003 when they were: 1.13:1 (British Columbia), 0.73:1 (Alberta), 0.30:1 (Saskatchewan) and 0.45:1 (Manitoba).²⁹
• Possible Improvement: Performance measurement: the evaluation was hindered in assessing the success of projects by performance measurement problems. The most important departmental problems related to: 1) the overlap in sub-activities which prevented analysis by sub-activity; and 2) the widespread use of unique indicators, which concealed many of the important impacts of projects from departmental decision-makers. Arguably the best projects would be those that spanned more than one subactivity or activity; however, much of their impact is captured by unique indicators that cannot be aggregated for reporting purposes. The department is currently revamping its Program Activity Architecture and Performance Measurement Framework and the result will, ideally, produce performance information that is fully available to decision makers and reportable in performance reports.

Recommendation

Although the innovation programming was well-structured and realized short term objectives, the evaluation identified possible improvements to program delivery. The following recommendation is based on evidence gathered and conclusions discussed in this study:

Recommendation #1: The department should review its program delivery processes to streamline programming administration and clarify departmental innovation priorities.

• Evidence from the evaluation indicated potential areas that could improve the effectiveness and efficiency of program delivery processes such as wider access to funding advances, streamlining the application, approval and accounting/reporting processes, and clarifying the departmental priorities and the purpose of the consultative process.