Western Economic Diversification Canada
Symbol of the Government of Canada

Common menu bar links

Evaluation Approach and Methodology

Evaluators consulted with program managers to develop the logic model underlying this theory-based evaluation.  The evaluation was planned as a quasi-experimental design involving a non-equivalent control group. To maximize the objectivity and relevance of the conclusions, evaluators sought feedback from program staff throughout the evaluation process. 

Evaluation Study Activities

Preliminary Consultations

Preliminary consultations were conducted with departmental officers in the development of the evaluation framework and the interview guides and also to discuss data availability. The officers were also consulted in developing the list of key informant interviewees and case studies. Through these consultations, some preliminary evaluation information was also obtained.

Documents and Literature Review

The evaluation included the review of documents and literature containing information relevant to the initiatives.  Three main types of documents were assessed and analyzed during the evaluation:

  • General Background documentation (e.g. TB Submissions, documents that describe CEDI and AII history, rationale, theory, etc.);
  • Program & Policy Documentation (e.g., Departmental Performance Reports, departmental database, project files); and
  • Literature on community adjustment programs, best practices in economic adjustment program design and the mountain pine beetle.

File Review

Using a customized data extraction template, the evaluation team analysed all information contained in the department’s databases (Project Gateway and the GX financial system) and other department data collection systems including paper files.  The database review was completed in July 2010 and the file review in British Columbia was completed in August 2010.  Initially, 140 of the 144 CEDI projects listed community development as the sub activity while the other four listed community planning (one project) or community adjustment (3 projects).  The sub activity for the community adjustment projects was later changed to community development and new Program Activity Architecture indicators were added.   All of the Airport projects linked to the community development sub activity.

Of the 144 CEDI projects funded, 83 (58%) were ongoing and scheduled to complete by March 31, 2011.  Eighty-six projects (60%) focused on community capacity building, 37 (26%) on economic diversification, 19 (13%) on economic infrastructure and the remaining two on value-added forestry.  There were relatively few value-added forestry projects funded under the CEDI because softwood lumber manufacturers and exporters were not eligible for funding as a consequence of the Softwood Lumber Agreement restrictions imposed on those companies and their subsidiaries.  Fifty of the CEDI funded projects were tourism related (mainly tourism development studies and feasibility studies). Other common project types included community economic development studies (19), forestry industry development projects (17), industrial land development projects (15), training projects (10), and agriculture industry development projects (9).

At the time of data collection for this evaluation, the departmental liaison officers for the airport projects had been reassigned to other duties within the regional office.  Therefore, the paper files in Vancouver were the only source of information on these projects.

Return to the top of this pagetop of page

Comparative Analysis

The evaluation included a review of comparable programs to both initiatives, including:

  1. A literature review of similar programs targeting pine beetle affected areas;
  2. Fourteen interviews with officials that are involved or have been involved in similar economic adjustment programming;
  3. Analysis of the results to develop a profile of similar programming, review the CEDI and AII against these programs, and to identify best practices and potential implications for future programming; and
  4. A review of the literature regarding best practices in economic adjustment program design.

The review concluded there is no ideal economic adjustment program for all situations; however, a review of comparable programs identified success factors for economic adjustment programming.  As indicated later in the report, CEDI and AII addressed some of these success factors but key informants claimed that others, such as monitoring, were more difficult or not addressed.

Interviews with Internal and External Key Informants

The evaluators completed 49 key informant interviews.  The 49 key informants included:

  • 13 interviews with departmental staff involved in the CEDI/AII.  These 13 staff were chosen because they were identified as the most involved and familiar with the programs;
  • 19 interviews with staff from 16 of the 19 Community Futures Organizations involved in the CEDI.  Although input from all 19 Community Futures Organizations was sought, three of the 19 declined participation;
  • 10 interviews with representatives of other government departments and stakeholders involved in/familiar with the CEDI/AII.  These 10 representatives were chosen based on literature research and snowball sampling; they were involved in the delivery of other programs and services targeting similar communities; and
  • 7 interviews with experts in the field of community adjustment/diversification.  These experts were academics and professionals, selected because they were knowledgeable about community adjustment/diversification issues in BC, particularly in the context of the mountain pine beetle infestation.

Proponent Survey

The proponent survey attempted to contact at least one proponent from each funded project.  The survey successfully contacted representatives of 105 of the 144 (79%) CEDI projects and all 3 (100%) of the airports.  The proponent survey also contacted proponents from non-funded projects including some: 1) from each zone; 2) from each of  the four categories of funding applicant:  independent, small/medium sized enterprise, aboriginal and municipal; and 3) who were rejected at the Expression of Interest stage and some who were rejected at the full proposal stage.

Consultants developed and pre-tested the questionnaire and conducted telephone interviews with 118 proponents of funded projects including:

  1. Community Economic Diversification Initiative Proponents:  a representative sample of 114 proponents of 105 CEDI-funded projects (38 led by independent organizations, 36 led by Aboriginal organizations, 31 led by municipalities and 67% from zone 2);
  2. Airport Improvement Initiative Proponents: four proponents of the three funded airports (one respondent from Kamloops, one respondent from Kelowna and two respondents from Prince George).

Consultants developed and pre-tested the questionnaire and conducted telephone interviews with 98 proponents of projects which did not receive funding including: 48 projects led by independent organizations, 22 projects led by small/medium size enterprises, 16 projects led by Aboriginal organizations and 12 projects led by municipalities.  The evaluators attempted to contact all proponents submitting Expressions of Interest; however, only 98 representatives were willing and available for interview at the time of the evaluation. 

Data collection spanned a period of four weeks between early November through early December 2010 with original contact information acquired through the departmental project database. Most proponents contacted were managers, directors, owners, or senior staff in municipal organizations, First Nations bands and other community organizations

Key informant interviews and survey results reported in Section 3 and Section 4 of this report include average respondent ratings to questions aimed at assessing the relevance and performance of the initiatives using a scale of 1 to 5, where 1 is no impact at all, 3 is somewhat of an impact and 5 is major impact.

Return to the top of this pagetop of page

Case Studies

Case studies for CEDI were selected by the program officer to represent a variety of challenges and obstacles encountered across both zones and the four program objectives.  A total of eight case studies were completed for CEDI.  In the absence of a program officer during the data collection stage for the AII, the evaluation team chose the Prince George Airport as the largest of the three airports.  A total of 17 stakeholders were interviewed for the 8 CEDI case studies and 2 stakeholders were interviewed for the one AII case study.

Outcome Assessment

To build stronger evidence of the socio-economic impact of CEDI/AII, the evaluation explored  the feasibility of two options:

  • Option 1: using pre- and post-initiative community-based socio-economic indicators for affected BC communities from sources such as Statistics Canada or the existing mountain pine beetle literature.  The indicators would measure the socio-economic impact of CEDI/AII.   
  • Option 2: Including control communities for comparison purposes. 

The report-based nature of many of the projects and the lack of timely regional data from existing data sources precluded option 1.  The following methodology was developed for option 2:

  1. Thirty-one CEDI projects were identified under the following seven sectors:  1) Tourism Development (7 communities); 2) Energy Plan Feasibility Studies (7 communities); 3) Community Economic Development Strategies (5 communities); 4) Downtown Revitalization (3 communities); 5) Forestry Related Feasibility Study (3 communities); 6) Land Development Study (3 communities); and 7) Agriculture Feasibility Study (3 communities).
  2. The community associated with each of the 31 CEDI projects was matched with a comparator community that had experienced some level of Pine Beetle devastation but had not received funding for that sector under CEDI (although they may have received funding for another sector).  The communities were matched based on region, population, Aboriginal status and level of forest dependency.
  3. Four comparator airports, not funded under AII, were matched to the 3 AII funded airports based on similar volumes of passengers enplaning and deplaning in 2008. The four airports were:  Victoria, Abbotsford, Comox and Fort St John.  During the time period of the Mountain Pine Beetle Program, these airports had undertaken various upgrades including improved taxiways or lighting and equipment purchases.
  4. Interview representatives of the 31 comparable communities and all of the comparator airports.

Return to the top of this pagetop of page

Focus Groups

All departmental staff responsible for the initiatives and all Community Futures Organization’ staff involved in the delivery of the initiatives were invited to attend one of the two focus groups conducted in Vancouver in early January 2011.  The sixteen focus group attendees  included six departmental staff and ten Community Futures Organizations’ staff. The departmental staff attended in-person while the Community Futures Organizations’ staff participated via videoconference. 

A consultant presented the field research findings at the focus groups and then facilitated group discussions. The objectives of the focus groups were to review and confirm the field research findings and explore ways to enhance the effectiveness and efficiency of similar programming in the future.

Limitations of the Methodology

Project File Review:  it was difficult to identify which of the 144 CEDI projects were incomplete as of July 2010:  the departmental database reported 83 incomplete projects while the British Columbia office reported 51 incomplete projects.  There were some financial inconsistencies with the three Airport Improvement Initiatives.  Specifically, 1) Kamloops Airport:  the most recent client report (September 2009) estimated the project cost at $24.34M, however the financial system and the departmental database still had the original value of 18,500,000; 2) Kelowna Airport: a small discrepancy between the updated project cost on the project gateway information page ($7.82M) and the original cost of $7.97M indicated in the rest of the project gateway documentation including the amended contribution agreement.  The total project costs differed in the financial system and the departmental database for 71 amended CEDI projects and one AII project, likely because the financial system had not been updated to reflect the amendments.  Therefore, the departmental database data were used to calculate leveraging.  Also, the amount disbursed on the Kelowna airport according to project gateway ($1.35M) disagreed with that shown in financial system ($1.12M) (June 30, 2010).  

Case Studies: Case studies enable in-depth analyses that are otherwise impossible. Given the expense and time involved in completing a case study, the goal was to purposefully select cases for contextual descriptive potential. While quantitative samples aim to be representative and generalizable, case study sample selection is non-random and intended to provide insight into the complex processes underlying a small number of projects.  The case study interviews relied on respondent experience, perception and recall at the time of the evaluation and results may be biased accordingly.  Some respondents, for example, became involved with the project after the completion of the application process, impeding collection of application-related information for that project. There was also overlap with the proponent surveys as the same 19 proponents interviewed for the case studies were interviewed for the survey.  In addition to recall bias, there is the potential for measurement error related to using questionnaires that were not tested for validity or reliability.  For example, the meaning of terms such as community capacity and economic infrastructure were open to respondent interpretation.  To compensate for potential bias, the case studies in this evaluation serve as one line of evidence to complement the other lines of evidence.

Key Informant Interviews:  Representatives of some funded projects had moved to different organizations, retired or did not precisely remember the project details.  Every effort was made to update contact information and give interviewees time to review documentation prior to the interview. Many of the funded projects were still ongoing and therefore the interviewees estimated the impacts whenever possible.  Of the 49 key informants interviewed, six were able to provide limited information on the Airport Improvements Initiative and their information was limited to a particular aspect of the initiative: one of the airports, one time period (such as the end of the program) or one activity (i.e. reporting, monitoring).  As a result, the key informant information reported in section 3 often relates to both CEDI/AII and, because it is already in section 3, is not repeated in section 4.   The one departmental staff member who was very knowledgeable about the Airport Improvements Initiative was unavailable for interview during the regular data collection period; this individual completed a key informant interview late in the evaluation process and the information was integrated into section 4.  

Proponent Surveys:  The main challenges included: 1) there were eight instances where information on multiple projects was obtained through a single interview, which could potentially bias and/or generalize the responses. For this reason, each respondent was counted as only one data point regardless of the number of projects in which they were involved; 2) For non-funded projects, some companies/organizations no longer existed at the time of the evaluation. This implied that some of the respondents most affected by not receiving funding were excluded from the survey. A broad sample of non-funded projects was contacted to minimize this risk; 3) many projects were either recently completed or not completed yet, leaving too little time to realize economic impacts.  Questions were therefore worded to elicit opinions of respondents in regards to the potential economic impacts for projects which were not complete (or recently completed). 

Outcome Assessment:  The size of the sample reporting a similar project was relatively small:  26 of 31 comparator communities agreed to an interview, of which 16 had undertaken a similar project.  Most of these projects are still ongoing and few reported impacts.  Nine of the projects were studies with a report being the only output.  The recommendations of these reports will need to be implemented before impacts are measurable.  Furthermore, most projects began recently with only six of the 16 projects being complete at the time of the interview.  Even in the situations where the project was completed, it is still too early for tangible results to be evident.  The projects may not be directly comparable:  the projects that were chosen in the comparable communities were from the same sector but often had a different focus in terms of, for example, the nature of the study or research or the type of infrastructure being developed.

Attribution:  Determining the net impacts of the department’s activities is challenging because comparable baseline data is lacking and, over the long term, it is difficult to disentangle the contributions of the CEDI/AII from that of its funding partners and the many other factors influencing the outcomes over a large number of years.  This evaluation uses contribution-focused analysis to ascertain whether the initiatives’ activities contributed to the achievement of objectives.  Contribution-focused analysis is not intended to establish the unique impacts of the program’s activities, but instead to indicate whether and to what extent they played a role in achieving strategic outcomes.