Western Economic Diversification Canada
Symbol of the Government of Canada

Common menu bar links

Performance: Achievement of Expected Outcomes

General Findings

Overall, most of the interviewed proponents believed their projects successfully contributed to at least two, and often more, of the immediate outcomes corresponding to the six subactivities15. Although it is too early to assess longer term outcomes, projects are expected to contribute to at least two, and often more, of the intermediate outcomes.16 The case studies generated a wide range of community impacts including development of skilled trades or researchers, spin-off projects, improved agricultural operations, environmentally responsible industrial processes, economic diversification, improved mining productivity, penetration of new technologies into the economy and improving quality of life of patients.

Focus group participants and key informants identified departmental funding as a critical factor to the success of the projects. The support from department enabled proponents to undertake new activities and provide new services to support industry. The informants also noted the critical role of departmental staff support in assisting proponents craft proposals and establishing collaborations. Another factor mentioned in the focus groups was flexibility in application timelines that allowed proponents to approach the department at any time with an idea that could then lead to a proposal and a funded project.

The two success factors most commonly identified by proponents were: 1) access to capital; and 2) access to equipment for technology development and testing. Additional success factors mentioned by the case studies included: partnerships with the department, communities, industry, government; and good project management, teamwork and vision.

Proponents faced cash flow problems and some projects were not sustainable over the long term.

Conversely, the greatest impediment to success for proponents across all regions and subactivities was insufficient capital or operating funds (40%). One common factor determining success among the case studies was ongoing operational funding: the more successful case studies had managed to secure ongoing partnerships and operational funding whereas the others had not. Economic factors such as the fluctuating dollar and rising construction/labour costs also hindered the progress of some of the case studies. One of the two other impediments discussed in the focus groups was the requirement that a proponent spend their annual budget by fiscal year end (March 31), often leading to extra work, cost and risk because the window of time between approval and March 31 was insufficient to obtain leading-edge equipment and other resources quickly. In one instance, for example, a proponent received approval for funding in December and had to spend the annual budget in three months. In another case, a proponent was pressured to order and receive equipment by year end, then the equipment sat in storage until the facility was built and by that time there were technical difficulties with the equipment. The other impediment was related to cash flow: the department reimburses eligible expenses and it can take several months to receive funds after submitting a claim. This reimbursement process differs from most other funding organizations that provide up-front funding with year-end reconciliation.

Performance Measurement

Innovation Activity

The department tracks innovation activity-level outcomes through the following three performance indicators.

  1. Total western Canadian university income from the Commercialization of Intellectual Property. This indicator has been on a decreasing trend since 2006, dropping from $27-$28 million (2003-2006) to $22.9 million (2007) and $16.5 million (2008). The target of $28.1 million was not met. The 2010-11 Departmental Performance Report indicated that the decline could reflect inconsistent reporting practices at universities.
     
  2. Business Expenditure on Research and Development as a percentage of Gross Domestic Product. This indicator hovered around 0.5%, unchanged from 2004. The target of 0.55% was "mostly met".
     
  3. Employment in Natural and Applied Science and Related Occupations as a percentage of Total Employment. The target for this indicator was 7% and the performance status could not be determined due to unavailability of data.

The usefulness and quality of these three indicators is questionable because there is a three-year lag in availability of the data for the first two indicators and the third indicator is based on the census which is done every five years. The lack of timely data on these three indicators could impede the department's ability to track its progress. The first indicator may be unreliably reported. The apparent stability of the second indicator suggests it is unresponsive to the impact of the department's activities over time. Furthermore, a target of 0.55% for "Business Expenditure on Research and Development" seems somewhat low considering the 1% target for Canada and Canada's international competitors.

Several department staff observed that a significant number of innovation projects could also be classified as business development projects under the Business Development Activity. We examined the 242 unique indicators of the 98 complete/nearly complete innovation projects to determine how many were similar to business development performance indicators. For example, one unique indicator of an innovation project was "# of businesses launched", which was comparable to the business development indicator of "# businesses created or maintained or expanded". In total, we found 15 (6%) unique innovation indicators that could be measuring business development outputs/outcomes, which implies that any significant overlap between the two activities is not captured by the performance measurement system.

Innovation Sub-Activities

There are 24 Performance Activity Architecture performance indicators (standard indicators) corresponding to the six innovation sub-activities (Table 4.1). Some of the standard indicators were rarely used suggesting they may be irrelevant.

In addition to the 24 standard indicators, there were 442 unique indicators for the 202 projects. A 2009 study of two innovation sub-activities17 suggested that the department: 1) reduce overlap between sub-activities by deciding what type of projects belong in which sub-activity; and 2) reduce the number of unique indicators. These issues prevent accurate outcome assessment by sub-activity and lead to measurement errors because unique indicators cannot be aggregated for assessing and reporting outcomes.

The importance of these issues led us to examine the 242 unique indicators of the 98 complete/nearly complete projects and found:

  • Overlap between subactivities: assessed by estimating the number of unique indicators that could be measuring an outcome from a different sub-activity. Findings: 136 (56%) of the 242 unique indicators could be measuring an outcome from a different sub-activity. For example, 53 (65%) of the 82 unique indicators of "Knowledge Infrastructure" projects measured outcomes corresponding to other sub-activities. Furthermore, when rating project success, every proponent claimed their project was very successful in achieving outcomes related to at least two subactivities. Key informants also commented that projects did not allocate perfectly under one sub-activity, but rather overlapped with multiple sub-activities, leaving staff to decide which sub-activity to use and resulting in inconsistent allocation across regions.
     
  • Too many unique indicators: assessed by estimating the number of unique indicators that could be replaced by standard indicators. Findings: standard indicators could have replaced 31 (13%) of the 242 unique indicators. For example, 11 unique indicators under "Technology Skills Development" measured training and therefore overlapped the standard indicators. Across all sub-activities, 54 unique indicators measured training and could have been replaced by standard indicators.

It therefore seems the performance issues identified in the previous study continue to challenge performance measurement. The impact of the issues lies in the inability to assess outcomes by sub-activity and in obscuring the impacts of project outcomes through the use of unique indicators. Key informants agreed that the success of projects was not fully captured and claimed performance indicators were primarily being used as milestones to track progress.

Table 4.1 Number of Approved Innovation Projects by Performance Activity Architecture Performance Indicator and Sub-Activity, April 2007 to June 2011
Sub-Activity Number
Projects*
Technology Adoption & Commercialization Total projects=70
# licenses executed 7
# patents filed/issued 9
# prototypes developed 31
# spin-off companies formed 7
# technologies adopted 14
# technologies to market 21
# technology demonstrations 29
Venture Capital $ Invested 5
Technology Linkages Total projects=24
# attendees at funded conferences 15
# partnerships/networks formed 10
# members 7
Technology Research & Development Total projects=20
# demonstrations of viability of technology 8
# Products or processes that are identified for further R&D 8
# publications in recognized science journals 7
# skilled personnel 6
# patents filed/issued 4
Technology Skills Development Total projects=14
# people trained 10
# training courses 5
Community Innovation Total projects=17
# individuals with enhanced skills 7
# studies 8
$ invested in knowledge infrastructure or tech com facilities 3
Knowledge infrastructure Total projects=57
# of square meters dedicated to R&D and skills training 15
# physical assets 45
Value of R&D undertaken in the new facility or using new equipment supported under this project 10
*Totals sum to more than 202 because projects had multiple indicators

Return to the top of this pagetop of page

According to the final reports on the 98 completed projects, 87 (89%) met or exceeded their performance expectations. Of the three complete projects funded under the Winnipeg Partnership Agreement, two met or exceeded expectations.

The high success rate raises questions as to how innovative the projects actually are. Innovation can be defined as "new or better ways of doing valued things"18 and is, by nature, risky, unpredictable and long term. An estimated 80% of innovative attempts will fail and a high success rate implies a safe funding approach that lacks ambition.19 The evidence implies the following two conclusions:

  1. The department appears to follow a safe approach. Focus group participants felt the department tends to fund projects focused on infrastructure rather than people/resources because infrastructure is less risky. The department's approach to funding is safe in the sense that every project undergoes a risk assessment using a risk review tool that produces a categorical risk score of low, medium or high for each project. Our inspection of the risk scores for the innovation projects revealed only two with high risk scores.
     
  2. The department's approach focuses on the short term. The department's approach appears to ensure projects successfully meet predetermined short term measures. Almost all complete projects met or exceeded their performance measurement expectations. The majority of the eleven projects and four case studies that did not meet their performance measurement expectations could not yet be classified as failures because they encountered legitimate delays and were expected to meet their objectives eventually. Focus group participants observed that the department seems to be more comfortable in being involved earlier rather than later in the innovation cycle, even though the real challenges lie in the later phases and in trying to get technologies out into the market.

Expected Result - Innovation: a stronger knowledge-based economy

Description: facilitate the West's transition to a knowledge-based economy.

Literature on the role of innovation in strengthening the knowledge-based economy and achieving a diversified and developed regional economy

Literature supports the positive impact of innovation on the knowledge-based economy: "Through innovation, new knowledge is created and diffused, expanding the economy's potential to develop new products and more productive methods of operation."20 The link between innovation and economic development is also well-established, although "economic benefits accrue only when a technology or technique is brought to the marketplace where it can be sold to generate income or applied to increase productivity." 21

Evidence that innovation programming is strengthening the knowledge-based economy and achieving a diversified and developed western Canadian economy

Projects funded by the department's Innovation programming each align to one of six sub-activities under the department's Innovation Activity. Each subactivity contributes to the Innovation Activity's ultimate outcomes of 1) a stronger knowledge based economy and 2) the development and diversification of the western Canadian economy. The expected results and distribution of the 202 innovation projects is summarized below:

  • Technology adoption and commercialization: 35% of projects
     
    • Expected Result: An increase in the number of technologies developed in research institutions that have commercialization potential and an increase in technologies adopted by existing firms.
       
    • Description: support small- and medium-sized enterprises to increase the commercialization and adoption of technologies, products, processes and services in the marketplace.
  • Technology linkages: 12% of projects
     
    • Expected Result: Increased connections and synergies among innovation system members.
       
    • Description: invest in networks, industry associations and other initiatives or events designed to build connections and synergies among the players of the innovation system.
  • Technology research and development: 10% of projects
     
    • Expected Result: Applied research and development leading to technologies with commercialization potential.
       
    • Description: support applied research and development that has the potential to lead to a commercial product or service.
  • Technology skills development: 7% of projects
     
    • Expected Result: Increase in training, education and skills building of highly qualified people (HQP)
       
    • Description: support development of specialized skills related to technology and commercialization through targeted training and through internships, which enhance the skill level of scientists and their ability to respond to industry needs.
  • Community innovation: 8% of projects
     
    • Expected Result: Increased technological capacity in a community.
       
    • Description: support the innovation capacity of communities across western Canada through support for planning studies, skill development in remote and northern areas, and enhancing knowledge infrastructure.
  • Knowledge infrastructure: 28% of projects
     
    • Expected Result: Increase in physical assets for research and development or training.
       
    • Description: support knowledge infrastructure such as research equipment or buildings, which are key for research and development or specialized training.

The expected results listed above are the short-term immediate outcomes. The intermediate outcomes included: a strengthened innovation system in Western Canada; increased technology development, adoption and commercialization; further development of technology clusters; people complete and incorporate training; research is shared and used; and western firms innovate to create wealth. The long term outcomes included: a stronger knowledge-based economy; and development and diversification of the western Canadian economy.

The programming realized a wide range of short term impacts

For the innovation activity as a whole, projects appear to be strengthening the knowledge-based economy by creating new products, services, information and knowledge available to industry. Proponents believed their projects had successfully achieved short term outcomes, the greatest impact being on training and skills development. The least impact (rating of 3.6 out of 5), was on the adoption of new technologies, which is the outcome associated with 35% of projects and a focus of the department's programming. The relatively lower rating for this outcome could be because it was too early in the project lifecycle to achieve commercialization outcomes. In fact, over the intermediate term, proponents expect projects to significantly contribute to several of the outcomes including increased technology development, adoption and commercialization.

Interviewees also largely believed that projects were diversifying the western Canadian economy. Some stated that their projects would benefit several sectors: for example, proponents working with technologies in the information and communication technologies and genomics sectors stated that advances in these sectors can enable new product offerings in other sectors (i.e. health). Other projects focused on diversifying a specific sector: for example, moving the forest products industry beyond primary production into more value-added products. A few proponents stated that they did not believe that their project would have a significant impact on diversifying the economy because they were undertaking projects within traditionally mature or strong western Canadian sectors such as the oil and gas, and agricultural and food supply industries.

Projects increased the education, training, and skills of highly qualified personnel either through formal training or through an informal transfer of knowledge. However, the focus groups indicated: 1) the training of highly qualified people seems to be more of a priority in some regions than others; and 2) there is some confusion as to whether the gaps are in technical training/competencies or business skill or some combination of both.

Unintended Impacts

Two distinct unintended positive impacts were identified. However, no unintended negative impacts were identified consistently.

The first unintended impact was an increased coordination among stakeholders. Again, most credited the department's position as a neutral party and departmental staff efforts as a liaison or partnering broker in identifying and creating more collaborative arrangements early in the process.

Secondly, departmental support increased the credibility of the organization, thereby raising their national and international profile. This resulted in other spin-off benefits such as additional investment from other funders and increased ability to attract highly qualified people to the organization.

No unintended negative impacts were consistently identified by interviewees. However, a small number of interviewees did state that some organizations might develop a reliance on departmental support, and it would be a hardship if departmental priorities changed.

 


[15] The immediate outcomes included: an increase in the number of technologies developed in research institutions that have commercialization potential and an increase in technologies adopted by existing firms; increased connections and synergies among innovation system members; applied research and development leading to technologies with commercialization potential; increase in training, education and skills building of highly qualified people; increased technological capacity in a community; increase in physical assets for research and development or training. Their correspondence with the sub-activities is listed in Section 4.1: Innovation Sub-Activities.

[16] The intermediate outcomes included: a strengthened innovation system in Western Canada; increased technology development, adoption and commercialization; further development of technology clusters; people complete and incorporate training; research is shared and used; western firms innovate to create wealth.

[17] Ference Weicker & Company, 2009, "Impact Assessment of the Technology Adoption and Commercialization and Knowledge Infrastructure Sub-Activities of the Innovation Component of the Western Diversification Program".

[18] Council of Canadian Academies, 2009. "Innovation and Business Strategy. Why Canada Falls Short." ISBN 978-1-926558-14-1.

[19] Burt Perrin, 2001. "How to – and How not to – Evaluate Innovation."

[20] OECD, 2005, "The Measurement of Scientific and Technological Activities: Guidelines for Collecting and Interpreting Innovation Data: Oslo Manual, Third Edition".

[21] Wendy H. Schacht, "Industrial Competitiveness and Technological Advancement: debate Over Government Policy", December 2010.