Western Economic Diversification Canada
Symbol of the Government of Canada

Common menu bar links

Evaluation Methodology

This section describes the design, methodology and limitations of the evaluation.

Evaluation Logic Model and Framework

A formal logic model was not developed for the department's Trade and Investment activities. The following logic model, which was designed to help conduct the evaluation reflects the activities, reach, outputs and outcomes in the Market and Trade Development and Foreign Direct Investment sub-activities within the Business Development program activity of the Program Activity Architecture (PAA).

Program Logic Model for Trade and Investment
Activities
  • Implementation of G & C projects
  • Delivery of the North American Platform Program (NAPP)
  • Implementation of G & C projects related to the IRB policy
  • Support of Gateway & Corridors initiatives
  • Trade and Investment Research and Advocacy
Outputs Market and Trade
  • Research
  • Capacity building and outreach (training/mentoring and seminars)
  • Market development
  • Marketing
  • Missions and trade infrastructure
Foreign Direct Investment
  • Research
  • Investment training
  • Investment marketing and promotion events/activities
  • Investment missions
Reach Market and Trade: Federal departments, provincial/municipal governments; provincial/municipal/regional economic agencies; industry associations and small and medium-sized enterprises (indirectly) Foreign Direct Investment: Provincial/municipal/regional economic agencies; potential and existing investors
Immediate Impacts

Market and Trade
Increased awareness

  • Increased awareness of Western Canada in priority markets
  • Increased awareness of trade opportunities for firms in Western Canada

Increased capacity

  • Increased ability for firms in Western Canada to develop/expand into new export markets
  • Increased capacity of western small and medium-sized enterprises to access global value chains, including procurement opportunities

Increased partnerships/networks

  • Increase in international partnerships, including science and technology partnerships
  • Increase in new and existing products for international markets
  • Increase international penetration of western goods and services into international markets

Improved trade infrastructure

  • Improvements in ports and related infrastructure (to improve shipping)
  • Improvements in sector-related infrastructure (i.e. tourism) to improve exports
Foreign Direct Investment

Increased awareness
  • Increased awareness in select markets of Western Canada as a competitive investment destination
  • Increased awareness of Western Canada's technology capabilities
  • Increased awareness of Western Canada's gateway and corridors

Increased capacity

  • Increased awareness of investment trends/issues and best practices
  • Increased capacity to attract foreign investment

Increased foreign investor interest in Western Canada

  • Increase the number of foreign investors seeking to invest in Western Canada
Intermediate Impacts Market and Trade – Increased participation in international markets through increased export capacity and sales for SMEs in Western Canada
Foreign Direct Investment – Increase in foreign investment in Western Canada
Long-term Strong small and medium-sized enterprises in Western Canada with an improved capacity to remain competitive in the global marketplace

This logic model summarises the inter-relationships between the activities, outputs and intended impacts of the department's support for Trade and Investment. The framework for the evaluation was developed based on this logic model.

Return to the top of this pagetop of page

Major Lines of Evidence

This project was undertaken in two phases. The first phase consisted of the development of the evaluation terms of reference and a detailed evaluation work plan. The plan outlined the strategies and methodologies which were then implemented in the second phase of the project. The field research undertaken in the second phase of the project included document and literature reviews, key informant interviews, case studies and focus groups.

Preliminary Consultations

Preliminary consultations were conducted with WD headquarters and regional staff to develop comprehensive lists of projects, key informant interviewees, survey participants and case studies. This led to the development of the terms of reference for the evaluation. The terms of reference, including the evaluation framework, was reviewed by the department's International Competiveness Team and senior management. Through these consultations, some preliminary evaluation information was also obtained.

Document and Literature Review and Project Database Analysis

A detailed review was conducted of the department's documents as well as literature relevant to WD's Trade and Investment activities. Project and performance information related to WD's Trade and Investment projects was generated from the department's project database and analysed. The document and literature review included:

  • A detailed review of relevant policy documents developed by the department, including strategic plans, annual reports for the department's Trade and Investment activities, briefing notes, corporate plans and priority papers.
  • Literature research related to Trade and Investment developed by experts.
  • A review of past Western Diversification Program evaluation reports.
  • A review of literature on Atlantic Canada Opportunities Agency's evaluation of its Trade and Investment activities.

Key Informant Interviews

In total, 42 key informants were interviewed. The following table provides the target number and number of completed interviews for each key informant group as well as a short description of each group's sample.

Key Informants Interviewed by Group
Key Informants Completed Target Number Description of Sample
WD Management and Staff 18 20
  • WD senior management, including Assistant Deputy Ministers and senior managers, as well as WD staff responsible for Trade and Investment activities.
Other Federal, Provincial and Regional Government Representatives 16 10 – 15
  • 6 representatives from other federal government departments, including the DFATD, Agriculture and Agri-Food Canada, and Transport Canada.
  • 6 representatives from provincial government departments engaged in Trade and Investment.
  • 4 representatives from regional organizations engaged in Trade and Investment.
Other Industry Stakeholders and Experts 8 10 – 15
  • 5 representatives from industry associations
  • 2 experts in policy research on Trade and Investment
  • 1 regional development organization
Total 42 40 – 50  

The table below depicts the distribution of respondents across regions.

Key Informants Interviewed by Region
Region WD Representatives Other Government Industry and Experts Total
Manitoba 3 6 2 11
Saskatchewan 4 4 1 9
Alberta 3 4 2 9
British Columbia 2 2 3 7
WD Policy and Strategic Direction Edmonton/Ottawa 6 0 0 6
Total 18 16 8 42

All interviews with key informants were done by telephone. Interviewees were contacted at first by email and provided with a letter from the department that explained the purpose of the evaluation and invited them to participate.

Return to the top of this pagetop of page

Client Survey

The following table provides the target number and number of completed interviews for each client and beneficiary (i.e., project participants) group, as well as a short description of each group's sample.

Proponents Surveyed by Group
Target Group Completed Target Number Description of Sample
Project Recipient Survey 41 40 – 50
  • Project recipients were from 36 Western Diversification Program projects, 2 Conference Support Payments projects, and 3 Western Economic Partnership Agreement projects.
Beneficiary (Project Participants) Survey 25 20 – 25
  • Beneficiaries of and participants in Trade and Investment projects (e.g., SMEs or partners that participated in missions or attended a trade show) were surveyed. Contacts were gathered during the project recipient interviews.
Proponents that applied but did not received funding 8 5 – 10
  • Types of projects included projects to attract foreign investment, to host a networking and information event, to attend a trade show or mission, and to develop tools and programming to assist new SME exporters.
Organizations that did not apply for assistance from WD 5 5
  • This list was developed based on an Internet search of similar Trade and Investment organizations that did not receive funding (e.g., non-profit industry associations engaged in Trade and Investment activities in Western Canada). The search included a review of Industry Canada's Directory of Business and Trade Associations.
Total 79 70 – 90  

The table below depicts the distribution of respondents across provinces.

Proponents Surveyed by Region
Province Project Recipients Beneficiaries/ Participants Proponents That Were Not Approved Organizations That Did Not Apply Total
Manitoba 7 13 0 1 21
Saskatchewan 11 4 2 1 18
Alberta 7 6 3 1 17
British Columbia 16 2 3 2 23
Total 41 25 8 5 79

Case Studies

Case study reviews were one of the critical lines of evidence for this evaluation. Case study reviews were conducted of 11 projects funded by the department delivered by eight recipients. The case studies were selected to include a cross-section of projects by province, type of project, size of project and type of recipient organization. The selection of specific projects for the case studies also took into consideration the stage of project development and input provided by departmental representatives regarding the projects. The case studies included at least two projects from each province. The mixture of projects focused on capacity building (international certification and global readiness services), international marketing and trade shows, investment attraction and trade infrastructure.

In conducting the case studies, the evaluation team collected and reviewed background information, including project proposals, progress reports, completion reports and other project outputs. Interviews with proponents of these projects were included as part of the key informant interviews. The list of case study recipients with a description of their projects is as follows:

  • Alberta Women Entrepreneurs: One project to create a diversity certification and global supply chain access program for women business owners across Western Canada.
     
  • Business Link Business Service Center in Alberta and Canada/British Columbia Business Services Society: Two projects to establish global business and export advisory services in Alberta and British Columbia.
     
  • Canadian Hydrogen and Fuel Cell Association: One project to raise the profile of the hydrogen and fuel cell industry during the 2010 Olympics.
     
  • Motion Picture Production Industry Association: One project to market and attract investment in British Columbia's film, television and digital production industry.
     
  • Regina Exhibition Association: One project to attract new incoming buyers to the Western Canadian Farm Progress Show.
     
  • Saskatchewan Trade and Export Partnership (STEP) and Manitoba Trade and Investment Corporation (MTIC): Two projects to host pavilions for western Canadian agricultural equipment manufacturers at the Agritechnica 2011 international tradeshow.
     
  • Centreport Canada Incorporated: One project to start up and operate Winnipeg's inland port.
  • Agence nationale et internationale du Manitoba: One project to support Centrallia 2012, an international business-to-business forum.

Focus Groups

In total, 39 representatives participated in four focus groups. The breakdown of participants is shown below.

Key Informants Vancouver Winnipeg Saskatoon Edmonton Total
WD Representatives 1 2 2 53 10
Other Provincial and Federal Government Representatives 2 3 3 3 11
Project Recipients and
Industry Stakeholders
4 6 4 4 18
Total 7 11 9 12 39

The participants were selected based on their knowledge and involvement in WD's Trade and Investment activities. Some were also interviewed as part of the key informant interviews and project recipient and proponent surveys. All participants were invited by email and telephone. The sessions were two hours in length.

Evaluation Challenges and Limitations

  • Potential for respondent bias. The evaluation findings are based, in part, on the views of those with a vested interest in the program and potentially biased in their responses regarding outcomes. Several measures were taken to reduce the effect of respondent biases and validate interview results, including the following: (i) interviewers communicated to participants the purpose of this evaluation, its design and methodology, as well as the strict confidentiality of responses; (ii) interviews were conducted by telephone by skilled interviewers; and (iii) the respondents were asked to provide a rationale for their ratings, including a description of specific activities which contributed to the reported outcomes.
     
  • Limited knowledge and familiarity of stakeholders with the department's Trade and Investment activities. Some respondents had difficulty recalling specific aspects of the department's Trade and Investment activities and were unable to comment on outcomes. In these cases, the evaluators obtained opinions on the needs and gaps to support Trade and Investment in their region, as well as any recommendations they had for the department's involvement in Trade and Investment.
     
  • Low number of industry respondents in key informant interviews. There were a lower number of industry representatives and experts interviewed than were targeted. This was due in part to interviewing some industry representatives as project funding recipients instead of key informants. To address this limitation, a snowball sampling technique was applied in which some key informants were asked to recommend other representatives engaged in Trade and Investment in government or industry that were then interviewed.
     
  • Project beneficiaries reporting for only 12 out of 42 projects. Most projects had no clear beneficiaries due to the nature of their activities. Project beneficiaries were selected from small and medium-sized enterprises or partners that participated in projects involving trade missions or trade shows. The responses of beneficiaries were used to complement responses from project recipients and to add examples and details of areas where some projects have achieved particular outcomes or follow-on investments.
     
  • In reviewing the results of the focus groups, it is important to note that the discussion reflects the opinion of those who were in the room. The participants in the focus groups are not necessarily representative of all those who have been involved in WD's Trade and Investment activities. Given the nature of focus groups, the main conclusions do not necessarily mean that all participants spoke to all relevant issues or, if they had, would necessarily agree with the opinions which were expressed by others.
     
  • Attribution: Determining the value added by the department's Trade and Investment activities is challenging over the long term because outcomes at the intermediate and long-term level are the result of many factors working together. This evaluation uses contribution-focused analysis to infer the department's role in achieving strategic outcomes that lead to developing and diversifying the western Canadian economy.

 


[3] Note: There were three representatives from WD Policy and Strategic Direction and two representatives from WD-Alberta Region at the Edmonton focus group.