Western Economic Diversification Canada
Symbol of the Government of Canada

Common menu bar links

Evaluation Methodology

This section describes design and the methodology of the evaluation.

Major Lines of Evidence

This project was undertaken in two phases. The first phase consisted of initial interviews as well as a file and document review leading to the development of a detailed Evaluation Work Plan. The Plan outlined the strategies and methodologies which were then implemented in the second phase of the project. The field research undertaken in the second phase of the project included document and literature review, key informant interviews, case studies, and focus groups.

1.  Document and Literature Review

A detailed review was conducted of WEPA documents and files as well as literature relevant to the activities of the agreements. The document and literature review included:

  • A detailed review of relevant documentation including the WEPA Results Based Management and Accountability framework, briefing notes, project databases, project final reports, and the 2008 WEPA Evaluation Report.
     
  • Developing a profile of WEPA in terms of program objectives, activities, outputs, intended outcomes, delivery model, budget, and logic model outlining the causal linkages between the objectives, activities, outputs, and outcomes.
     
  • A review of the Evaluation Framework prepared by the department which outlines the evaluation issues, questions, indicators, and data sources to be used in conducting the evaluation.
     
  • A review of literature on other similar agreements that have been established in other regions. A particular focus was placed on agreements involving the federal government and provincial or territorial governments with objectives related to economic development or diversification.
     
  • A review of literature on other programs in western Canada that share similar objectives as WEPA. These programs were based on the results of a literature review as well as interviews with the key informants including federal and provincial government representatives.

2.  Interviews with Key Informants

As part of this evaluation, interviews and surveys were conducted with 143 key informants including WD representatives, representatives of provincial governments involved in WEPA, economic development experts, other stakeholders, representatives of other programs in Western Canada, representatives of similar agreements in other regions, project proponents, representatives of spin-off projects, and proponents of applications that did not receive funding.

The following table provides the target number and number of interviews completed for each key informant group as well as a short description of each key informant sample. As demonstrated in the table, the target was to complete 121 to 135 key informant interviews.

Table 6: Sample Key for Key Informant Interviews
Key Informants Completed Target Number Description of Sample
WD Representatives 26 25 Purposive Criterion sampling. Departmental Representatives were selected based on their knowledge and involvement in WEPA activities as well as regional distribution. 28 representatives were selected and were 26 interviews completed. Of the 26 departmental representatives, 4 were members of the management committees and 22 were WD officers or program leads working in regions. Of the 26 representatives, were Alberta, 7 British Columbia, 6 Manitoba, and 5 from Saskatchewan.
Provincial Government Representatives 10 9 Purposive sampling. The participants were selected based on their involvement in WEPA. Of the 10 representatives interviewed, 3 representatives were from Alberta, 4 from British Columbia, and 2 each from Saskatchewan and Manitoba.
Economic Development Experts 6 8 to 10 Purposive sampling. Through internet research and document review, we identified 10 major academia and research institutions, and economic think-tanks in Canada. We contacted each organization and asked about their knowledge and expertise in economic development in western Canada overall and specifically about WEPA. We interviewed 6 experts who considered themselves sufficiently knowledgeable to provide input.
Other Stakeholders 15 13 to 16 Purposive Criterion sampling. We identified 73 industry associations, economic development groups, research organizations and non-profits that are active in regions, sectors, or projects relevant to WEPA. We contacted all 73 organizations to determine the level of familiarity with the department and WEPA as well as their willingness to participate.
Representatives of Other Programs in Western Canada 4 4 to 5 Purposive/Snowball Sampling. Through internet research, document review and referrals from key informants, we identified programs in western Canada that fund similar initiatives. We then surveyed programs representatives who were involved in delivering or managing these programs.
Representatives of Similar Agreements in Other Regions 4 4 to 5 Purposive Sampling. We identified similar federal/provincial programs and selected five which shared goals most similar to those of WEPA. 4 interviews were completed with representatives involved in these programs.
Project Proponents 50 35 Census. We attempted to contact all project proponents and completed interviews with representatives of 50 of the 65 projects. Of the 50 proponents, 9 represented projects funded in Alberta, 14 represented projects in British Columbia, 12 represented projects in Manitoba, and 15 represented projects in Saskatchewan.
Spin-Off Projects 19 15 to 20 Purposive/Snowball Sampling. During the interviews with project proponents, we asked them to identify other projects and initiatives created as a result of the project activities or organizations that significantly benefited from the project implementation. Based on the interviews we were able to identify 20 such organizations and projects and completed 19 interviews.
Proponents of Projects That Did Not Receive Funding 9 8 to 10 Census. We were provided by a list of 19 representatives associated with applications to WEPA that did not received funding (i.e. were rejected or withdrew their application). We contacted each representative (excluding those who were turned down for one application but approved for another and therefore were interviewed as project proponents) and were able to complete 9 interviews.
Total 143 121 to 135  

All key informant interviews were conducted by telephone. Where possible, each individual was provided a copy of the questionnaire in advance of the interview.

Return to the top of this pagetop of page

3.  Case Studies

Case study reviews were one of the critical lines of evidence for this evaluation. Case study reviews were conducted of 8 projects funded by the department. The case studies were selected to include a cross-section of projects by province, type of project, size of project, and type of proponent organization. The selection of specific projects for case studies also took into consideration the stage of project development (i.e. completed or close to completion); the willingness of the proponent to participate, and input provided by departmental representatives regarding the projects.

More specifically, the eight case studies:

  • Include two projects from each province;
     
  • Had made substantial progress in implementation by the time of the evaluation. Of the eight projects, four were known to be completed, two had received 90% to 100% of their funding from the department, one had received 70% to 80% payout, and one had been paid 50% to 60% of the approved funding;
     
  • Include a mixture of technology, market/trade development, and business productivity projects. Of the eight projects, three were categorized as Improve Business Productivity; one was categorized as Market/Trade Development and four were categorized as Technology Adoption & Commercialization.
     
  • Include one non major-urban project (Northern Manitoba) and two large dollar projects (over $2 million in departmental funding). In terms of departmental funding, two projects were approved for $2 million to $6 million, four were approved for $1 million to $2 million, one was approved for $500,000 to $1 million, and one was approved for less than $500,000.
     
  • Include a mixture of different types of proponents including three university-based projects, one service organization, one public private partnership, one sector council, one provincial tourism agency, and one provincial research organization.

In conducting the case studies, the evaluation team collected and reviewed background information including project proposals, progress reports, completion reports and other project outputs, and conducted an interview with the departmental project officer responsible for the file. Other methods include conducting a site visit to the location of six of the eight projects to review the work undertaken as well as examining documentation regarding the impact and outcomes generated to date. The evaluation team also conducted interviews with the project proponents who have been involved in the project, and interviews with a sample of other representatives involved in or impacted by the project. As demonstrated in the following table, a total of 42 interviews were conducted for the case studies.

Table 7: Case Study Interviews by Nature of Involvement
WEPA III Case Study Interviews
Nature of Involvement Frequency
Project Proponents 8
WD Project Officer 8
Others Involved 14
Users of Project Outputs 12
Total 42

The information obtained from the case study reviews was analyzed and summarized and the key findings included in this report.

4.  Focus Groups

As part of the evaluation four focus groups were conducted. Focus group sessions were staged at departmental offices in Winnipeg, Saskatoon, Edmonton and Vancouver. Representatives were from the department, the respective provincial government, and other selected stakeholders. An outline of possible questions was forwarded to participants in advance to help them prepare for the session. The focus of the discussions varied somewhat from site to site, depending upon the composition of the groups, the interests of the participants, and relevance of particular questions to that jurisdiction. In total, 37 representatives participated in the focus groups.

Table 8: Focus Group Participants by Type of Key Informant
Type of Key Informant Winnipeg Saskatoon Edmonton Vancouver Total
Proponents/Stakeholders 5 5 3 4 17
WD Representatives 1 4 5 4 14
Provincial Representatives 1 1 2 2 6
Total 7 10 10 10 37

The participants were selected based on their knowledge and involvement in WEPA activities. Most were also interviewed as part of the key informant interviews. All participants were invited by e-mail and telephone. The sessions were two hours in length.

5.  Data Analysis and Reporting

The data from each of the evaluation methodologies was summarized to address each of the relevant evaluation issues/questions. The data analysis strategy includes the triangulation of multiple lines of evidence. This involves the extraction of the results from each line of inquiry that relate to each evaluation issue and cross validation of the findings. This step took into account the strengths and limitations of each line of inquiry.

Evaluation Challenges and Limitations

During the process of conducting this evaluation, a number of challenges were encountered; however several techniques were employed to overcome each challenge. The main challenges included:

  • Timing of the evaluation. In terms of effectiveness, the evaluation focused on immediate and intermediate outcomes. However, most of the supported projects were newly funded, which means that sufficient time has not passed to measure the progress made against intended intermediate outcomes. Many project activities were still underway or only recently completed and the impacts not yet measurable. The impact of many of the lessons learned, strategies tested, models developed and results achieved can only be measured over a longer-term.
     
  • Staff turnover in some of the proponent organizations and organizations which applied unsuccessfully for funding. Some of the lead staff responsible for developing proposals which were ultimately not approved for funding as well as some lead staff associated with funded projects were not available for an interview due to staff turnover. To overcome this challenge, the evaluation team tried to interview other representatives of the organizations who were most familiar with the project activities. Before conducting interviews, the team made sure that the representatives had adequate knowledge of the project activities.
     
  • Lack of knowledge and awareness of the department and WEPA among stakeholders and economic development experts. Many of the stakeholders and economic development experts contacted for interviews were not aware of WEPA. To overcome this challenge and reach the targeted number of interviews, the evaluation team expanded the contact list to include a large number of organizations. Of the 73 stakeholder organizations contacted, the team were able to complete 15 interviews. Of the 10 economic research organizations and think-tanks contacted, 6 interviews were completed.
     
  • Potential for respondent bias. The evaluation findings are based, in part, on the views of those with a vested interest in the program and potentially biased in their responses regarding program outcomes. Several measures were taken to reduce the effect of respondent biases and validate interview results including (i) interviewers communicated the purpose of this evaluation, its design and methodology, and strict confidentiality of responses clearly to participants; (ii) interviews were conducted by telephone by skilled interviewers; (iii) the respondents were asked to provide a rationale for their ratings including a description of specific activities which contributed to the reported outcomes; and (iv) impacts generated by particular activities were confirmed through document review, case studies, and learning circles.
     
  • Identifying representatives of similar programs and scheduling interviews. The evaluation team had to conduct extensive internet research and several telephone calls to identify representatives of similar programs who had adequate knowledge of their respective programs and are able to participate in an interview. To overcome this challenge, the team sent the representatives information about WEPA and this evaluation, explained the importance of their participation and assured confidentiality of the interviews.
     
  • Limited information available about similar programs. Finding adequate information about similar programs was a challenge. In particular, it was difficult to gather program specific data (e.g., description of program design and delivery, budget and administrative cost, performance measurement system, assessment and selection criteria, etc.). Most programs have limited data on the Internet and some programs have never conducted an evaluation (or evaluation results are not publicly available). To overcome this challenge, during interviews the evaluation team asked representatives of the similar programs to provide materials and resources which include more detailed information about the programs.
     
  • In reviewing the results of the focus groups, it is important to note that the discussion reflects the opinion of those who were in the room. The participants in the focus groups are not necessarily representative of all those who have been involved in the program. Given the nature of focus groups, the main conclusions do not necessarily mean that all participants spoke to the issue or, if they had, would necessarily agree with the opinions which were expressed by others.
     
  • Difficulty obtaining additional information from project proponents for case study reviews. The case study reviews required multiple communications with proponents to validate findings, request additional information and clarify certain aspects and conclusions. With the busy schedules of most project proponents, it was difficult to arrange times to request additional information. To overcome this challenge, the evaluation team made consistent and multiple attempts to contact, used various communication methods (e.g. phone, email), and explained the importance of their contributions in this review.