Evaluation of Mountain Pine Beetle Program

Executive Summary

Introduction

The mountain pine beetle epidemic decimating British Columbia’s pine timber stands began in the mid-1990’s and has since grown to one of the largest insect infestations on record in North America. In January 2007, the federal government announced the $200 million Federal Mountain Pine Beetle Program designed to address both the short-term and long-term impacts of the beetle infestation.

The Federal Mountain Pine Beetle Program comprised three key components delivered by different federal departments. Western Economic Diversification delivered the $56M communities component aimed at developing and diversifying the economies of affected communities. The $56M allotted to Western Economic Diversification was divided between the Community Economic Diversification Initiative ($36.6M) and the Airport Improvements Initiative ($19.4M).

The Community Economic Diversification Initiative (CEDI) was a two-year contribution program addressing the long-term impacts of the mountain pine beetle on approximately 180 communities in interior British Columbia.The department delivered the Community Economic Diversification Initiative in partnership with 19 Community Futures Organizations and other community stakeholders. CEDI addressed four programming objectives:

• community capacity building;
• economic diversification;
• economic infrastructure; and
• value-added forestry.

Based on recommendations of the Mountain Pine Beetle Program Advisory Board, CEDI funding was allocated to those communities deemed most at risk to biological and socio-economic impacts from the mountain pine beetle. Communities most at risk were identified by the as:

• having timber supply impacts and economic dependency on pine;
• small and remote communities, particularly First Nations communities, with economic and sustenance dependency on the affected forests;
• most immediately at risk due to current impacts of mountain pine beetle.

These metrics defined two impact zones. Zone 1, communities at risk and a sub-set Zone 2, communities at immediate risk. It was recommended that seventy percent of CEDI funding be allocated to Zone 2 communities. In terms of program delivery, a pre-marketing effort promoted the program and prepared potential proponents for the Request for Proposal process.

The mountain pine beetle affected approximately 100 First Nations communities, prompting the department’s development of an “Aboriginal Engagement Strategy” raise awareness of the program in First Nation communities and to support these communities in securing funding.  The CEDI application process involved two steps: 1) submission of an Expression of Interest; and 2) proposal submission. This application process spanned approximately nine weeks in the fall of 2007. To expedite the proposal development and approval process, proposal applications were submitted electronically and, following due diligence, batched for ministerial approval. In total, 144 projects were funded. Because many projects were incomplete as of the March 2009 deadline, the initiative was initially extended to March 31, 2010. To facilitate successful project completion, the initiative was granted a final extension to March 31, 2011.

The Airport Improvements Initiative (AII) was a two-year $19.4 million contribution program funding airport expansion projects in three communities affected by the mountain pine beetle: Prince George, Kamloops and Kelowna.

Using the existing Western Diversification Program authority, the Community Economic Diversification Initiative and the Airport Improvements Initiative were evaluated based on the Performance Measurement Strategy of the WDP. The evaluation covered the fiscal years 2007-08 to 2009-10 and assessed program relevance and performance.  Since these one-time initiatives will not be renewed, the evaluation focused on the initial, rather than ongoing, relevance of the initiatives. The evaluation methodology included document/literature review, file and database review, analysis of comparable programs, 49 key informant interviews, 118 project proponent interviews, nine case studies, an outcome assessment of comparator communities and two focus groups

Community Economic Diversification Initiative

Relevance

There was a major need for this type of programming. Many of the affected communities were small, rural, forest-dependent and lacking sufficient resources to cope with the economic, social and environmental effects of the mountain pine beetle infestation. Most projects would not have proceeded in the absence of CEDI funding; CEDI funding was the primary funding source for the CEDI projects and attracted funding from other programs. The initiative’s objectives supported one of the department’s strategic outcomes at the time, Community Economic Development, and aligned with the federal priority of Strong Economic Growth.

The initiative was consistent with federal roles and responsibilities related to supporting prosperity in all regions in Canada. A 2004 survey found that residents of areas affected by the mountain pine beetle believed the federal government should be involved although they had little trust in the federal government to properly manage for mountain pine beetle activity.

Performance - Achievement of Expected Outcomes

Although many of the CEDI projects were incomplete at the time of the evaluation, the completed projects were somewhat successful in realizing short-term objectives and generated a wide range of impacts including enhanced economic capacity and infrastructure,  improved business climate, economic stability, creation of economic opportunities, development of new businesses/industries, and diversification of local economies. However, some key informants questioned CEDI’s ability to realize its long-term goals of sustainable development and adjustment.

Success factors included staffing a dedicated Mountain Pine Beetle Unit within the WD-BC Regional Office, the two-step application process, building solid partnerships and the Aboriginal Engagement Strategy. On the negative side, the CEDI faced some implementation issues which contributed to several projects not finishing on time. The first issue was the long approval time for most projects, ranging from six months to more than one year, leaving proponents with very little time for implementation. The long approval time combined with the tight application deadlines lead to a sense of mismanaged expectations and disillusionment in the communities, adversely affecting the image of the department and the Community Futures Organizations. The second issue was the nature of follow-up monitoring as the dedicated departmental staff were reassigned after the administrative funding for CEDI ended, leaving some project proponents without the guidance and support they needed to complete their projects.

Performance - Demonstration of Efficiency and Economy

Project outputs and outcomes were moderately cost-effective and the CEDI  obtained good value with respect to the use of public funds. CEDI followed a top down program approach and the positive aspects of the initiative included its reasonable delivery costs and leveraging and its partnerships with various community-based organizations. The negative features related to the lack of advanced funding to some proponents, its appearance of being rushed into implementation and its two-year time frame which was too short to process the very large volume of project proposals and allow projects to complete on time. Designing CEDI as a time limited program overlooked one of the recommendations emerging from the evaluation of a previous time limited program, the Softwood Industry and Community Economic Adjustment Initiative. Specifically, the previous evaluation recommended to “set a longer program duration from the beginning”.

Airport Improvements Initiative

The evaluation of this initiative was hindered by the general lack of information on this initiative. All but one of the key informants had very limited knowledge of the AII.

Relevance

There was, and continues to be, a major need for funding and support from an initiative such as AII that acts  in concert with other infrastructure programs. Federal government policy precludes capital investments in airport infrastructure without explicit approval; AII was granted specific approval because of the significant impacts of the mountain pine beetle infestation and the opportunity for diversification. The initiative aligned with the federal government’s responsibility of ensuring all areas of Canada are prosperous. The AII complemented other government and private sector programs.

Performance - Achievement of Expected Outcomes

The projects funded under the AII achieved their objectives in terms of extending runways and expanding airport capacity and ability to operate year-round. One airport study found the runway extension improved operational and safety conditions for incoming air carriers. The projects are likely to be very successful in increasing airport activity levels in the long run as well.  Community and departmental support were identified as the most important success factors; Departmental involvement with First Nations bands was also critical.

Performance - Demonstration of Efficiency and Economy

AII was cost-effective and generally well-structured. Some respondents had cash-flow issues that rendered pre-payment very difficult. Respondents also said that although the departmental staff were careful to minimize delays, there were red tape issues that ended up resulting in several months of delays. Respondents said that intended results would have been achieved more effectively with additional funding during the project; instead, the community had to invest more funding.

Infrastructure projects like airports need to incorporate best practices in strategic planning, information management, asset management systems, consultation, priority setting and business case approaches. They should also include cost and benefit analysis of the project and account for social and environmental impacts. Overall, AII successfully leveraged 71% of its funding from other sources. Negative features of AII included its lack of advanced funding to some proponents and its tight deadlines. Most infrastructure programs are of longer duration.

Recommendation

The evidence gathered in the evaluation suggests the following recommendation:

1. The department should develop a template for the design and delivery of economic diversification programs that is built on past lessons learned and best practices but is flexible in accommodating the unique needs of each issue, region and program.

Introduction

Acknowledgement

Western Economic Diversification Canada (the department) would like to thank all of the key informants, case study participants and survey participants who generously gave of their time and knowledge to take part in the Evaluation of the Mountain Pine Beetle Program.  Without their participation and their insights, this report would not have been possible.  The evaluators also acknowledge the work done by Ference Weicker & Company in collecting key informant interview and survey data.

Background

Western Diversification Program

The Western Diversification Program (WDP) is the department’s major program and provides support to projects that develop, strengthen and diversify the Western Canadian economy.  The WDP facilitates the department’s collaborative opportunities and responses to economic priorities.¹

As an umbrella program authority, the WDP has a number of sub-components operating under its authority.  Both the Community Economic Diversification Initiative (CEDI) and the Airport Improvements Initiative (AII), using the existing WDP authority, were intended to support economic development and infrastructure projects in British Columbia communities affected by the mountain pine beetle infestation.  These two initiatives offered the federal government the opportunity to provide economic options to affected communities, some of which derived over half of their income base from the forest industry.

Federal Mountain Pine Beetle Program

The mountain pine beetle epidemic decimating British Columbia’s pine timber stands began in the mid-1990’s and has since grown to one of the largest insect infestations on record in North America.  The massive timber depletion caused by the infestation has devastated British Columbia’s forest industry and forest-dependent rural communities; the social and economic upheaval will be felt for decades. In 2007, the federal government announced the $200 million Federal Mountain Pine Beetle (MPB) Program designed to address both the short-term and long-term impacts of the beetle infestation.

The Federal Mountain Pine Beetle Program, lead by Natural Resources Canada, comprised three key components delivered by different federal departments:  1) communities: targeting long term economic impacts affecting communities 5 to 8 years in the future.  Delivered by Western Economic Diversification; 2) infrastructure. Delivered by Transport Canada; and 3) forests: targeting short-term biological impacts affecting forests and trees.  Delivered by Natural Resources Canada.  The $200M in funding was allocated as follows:

• Natural Resources Canada $100M;
• Transport Canada $44M; and
• Western Economic Diversification Canada $56M.

The $56M allotted to Western Economic Diversification was divided between the Community Economic Diversification Initiative ($36.6M) and the Airport Improvements Initiative ($19.4M).

Funding for the Community Economic Diversification Initiative (CEDI) and the Airport Improvements Initiative (AII) was approved in March 2007 with the Minister of Western Economic Diversification accountable for delivering the initiatives.  At that time, the department identified the risk that the tight timelines could require re-profiling of unused funds into 2008-09 and 2009-10 as outlined in Table 1.1.  In November 2009, the program’s end date was extended to March 31, 2011 and $7.74M reprofiled from 2009-2010 to 2010-2011.

Community Economic Diversification Initiative and Airport Improvements Initiative

Table 1.1 summarizes initial and proposed reprofiling of Grants and Contributions (G&C) funding for the Community Economic Diversification Initiative and the Airport Improvements Initiative.

*the total budget of $36.6M for CEDI included $3.48M for management costs including operating costs and a Public Works and Government Services Canada accommodation charge of 13% of salaries
**the total budget of $19.4M for AII included an additional $1.05M for operating costs.

CEDI was to be guided by the Mountain Pine Beetle Ministerial Council composed of the BC Minister of Forests and Range, the Natural Resources Canada Minister and the Western Economic Diversification Minister and supported by a senior-level Canada-BC Steering Committee.  The Mountain Pine Beetle Ministerial Council appointed an advisory board of interested parties from academia, First Nations, industry and the community.  The Minister of Western Economic Diversification was to advise the Mountain Pine Beetle Ministerial Council on the Airport Improvements Initiative.

Community Economic Diversification Initiative

The Community Economic Diversification Initiative was initially a two year $36.6M federal contribution program addressing long term impacts of the mountain pine beetle on approximately 180 forest-dependent communities in British Columbia.  CEDI funding eligibility was based on three sets of criteria:

• applicant criteria: eligible applicants comprise a range of legal entities including non-profit organizations, community and sector associations, local governments, small business ventures, Aboriginal groups and regional alliances.
• community criteria: eligible communities must be located in one of two “immediate risk” provincial zones².  Among eligible communities, additional priority was given to those that were:  Aboriginal; located in zone 2, which was targeted to receive at least 70% of the funding; forest-dependent; experiencing negative community impacts resulting from the mountain pine beetle.
• project criteria:  funded projects were to address one of four programming objectives: community capacity building, economic diversification, economic infrastructure or value-added forestry.

Not-for-profit organizations were eligible for non-repayable funding; small and medium sized enterprises proposing for-profit projects involving innovative forest and value-added wood were eligible for unconditionally repayable contributions.

According to the department’s project database (as of July 2010), 144 projects (104 clients) were approved and proceeding: 61 projects (42%) were complete and 83 were ongoing and scheduled to end by March 31, 2011..  Per project, committed departmental assistance ranged from a low of $5,117  to a high of $2.12M with half the projects being under $121,000.

The department delivered the Community Economic Diversification Initiative in partnership with 19 Community Futures Organizations and other community stakeholders.  Based on the Softwood Industry Community Economic Adjustment Initiative, a pre-launch effort promoted the program and prepared potential proponents for the request for proposal process.  Approximately 100 (56%) of the 180 affected communities were First Nations, prompting the department’s development of the “Aboriginal Engagement Strategy” to facilitate Aboriginal uptake and proposal development.  The two-step application process included an initial request for expressions of interest followed by a request for proposals from applicants submitting the top-ranked expressions of interest.   The department ranked the expressions of interest according to proponent eligibility, location, partnership, benefit, viability, timing and concurrence with Treasury Board conditions for the program.  A dedicated departmental team in the department’s Vancouver office assumed delivery responsibilities including: extensive support for proposal development, environmental assessment reviews, Aboriginal consultations and compliance with the official languages act. The team and funding partners developed a list of alternative funding sources (a “Pathfinding Resource”) to assist proponents not funded under CEDI.  Projects that addressed the objectives of both CEDI and the Pine Beetle Recovery Program (Northern Development Initiative Trust³) were invited to submit a common application to both Western Economic Diversification and the Northern Development Initiative Trust.

The Initiative operated within tight timelines, allowing six weeks for the Expressions of Interest phase  (September 7 to October 19 of 2007).  Expressions of Interest were evaluated against a pre-established set of assessment criteria.  By November 26, 2007, the top ranked Expression of Interest applicants were invited to develop a more detailed proposal.  The proposal submission deadline was December 17, 2007.  Many projects were incomplete as of the initial deadline for project completion (March 2009), prompting two deadline extensions to March 2010 and then to March 2011.

The department implemented special intake, review and approval processes to handle the very large volume of CEDI applications received. There was an electronic application process at the intake stage, a dedicated Unit to undertake support for project development and due diligence at the review stage, and a batching system to group applications for ministerial approval.

Airport Improvements Initiative

Recognizing that airport improvements represent an early and immediate opportunity to stimulate development and growth across large numbers of impacted communities, the two-year Airport Improvements Initiative funded three airport expansions: Prince George ($11.3M), Kamloops ($6.6M) and Kelowna ($1.5M). Operating funds for delivery costs were also funded under this initiative.  The province of British Columbia had targeted the three airports for infrastructure funding under its Transportation Partnerships Program and agreed to match the federal funding for two of the airport expansions (Prince George and Kelowna) and contributed $4M towards the Kamloops airport expansion.   Although at least ten BC local and regional airports were seeking funds at the time the initiative began, only airports located in and adjacent to affected regions were eligible under this program.

Prince George Airport:  located within Prince George timber supply area where some communities derive up to half their income base from forestry.  The airport first obtained departmental funding in 2004 under the Softwood Industry and Community Economic Adjustment Initiative (SICEA) to expand its terminal building and was again successful in obtaining an additional $11.3M under the Mountain Pine Beetle Initiative, announced in January 2007.  The $11.3M would fund an expansion of the airport runway, purchase of a snow-clearing vehicle, re-construction of taxiway/ apron and other improvements such as lighting and the navigational system. The project was scheduled to begin in September 2007.  Upgrades were expected to increase international air cargo and fuel stops.  Prince George would become a competitive fuel stop and air cargo option to Anchorage for trans-pacific air traffic between North America and Asia.

Kamloops Airport:  located within Kamloops timber supply area where many communities rely on forestry for 10% to 20% of their income base.  The proposed airport expansion included a runway extension, taxiway/apron expansion, renovations to the air terminal building and improved navigational aids.  The upgrades were expected to support transcontinental charter operations, making Kamloops an international all-season ski and golf destination. Construction start was scheduled for December 2007 but was delayed to spring 2008 while the Kamloops Airport Authority completed the Environmental Impact Assessment.

Kelowna Airport:  located within the Okanagan timber supply area where communities rely on forestry for between 10% and 30% of their income base.  The proposed airport expansion included a runway extension, improvements to navigational aids and a doubling of existing facilities, making Kelowna an international ski tourist destination for Europeans. The upgrades were expected to increase tourism.  While one local First Nations band strongly endorsed the project, the project was not supported by the Okanagan Indian Band and a Justice Canada review concluded contractual obligations and mitigation measures assumed by the Kelowna Airport reasonably addressed the issues identified by the Okanagan Indian Band.  An official groundbreaking ceremony occurred in February 2008.

According to the department’s project database (as of July 2010), the Kelowna Airport project is complete, final claim has been approved on the Prince George Airport and first claim has been approved for the Kamloops Airport.

Evaluation Mandate

Using the existing WDP authority, the Community Economic Diversification Initiative and the Airport Improvements Initiative will be evaluated based on the Performance Measurement Framework of the WDP. The evaluation will focus on the degree to which projects achieved their intended objectives and contributed to the development and diversification of the Western Canadian economy. The 144 projects funded under the Community Economic Diversification Initiative and the three airports funded under the Airport Improvements Initiative were examined as part of the evaluation. The CEDI and AII evaluation covers the fiscal years 2007-08 to 2009-10.

CEDI and AII are one-time initiatives that are not being renewed. Therefore, the evaluation will address the performance and the past/current, rather than ongoing, relevance of the programs.

Evaluation Scope and Objectives

The 2009 Treasury Board Policy on Evaluation requires departments develop a five-year evaluation plan to cover 100% of program spending over a five-year period. The Community Economic Diversification Initiative and the Airport Improvements Initiative were evaluated in the context of the department’s approved Five-Year Evaluation Plan (2009-14). This evaluation of the Community Economic Diversification Initiative and the Airport Improvements Initiative included various data collection methods outlined below. The objectives of the evaluation and the core evaluation issues are presented in Table 1.2.

Evaluation Approach and Methodology

Evaluators consulted with program managers to develop the logic model underlying this theory-based evaluation.  The evaluation was planned as a quasi-experimental design involving a non-equivalent control group. To maximize the objectivity and relevance of the conclusions, evaluators sought feedback from program staff throughout the evaluation process.

Evaluation Study Activities

Preliminary Consultations

Preliminary consultations were conducted with departmental officers in the development of the evaluation framework and the interview guides and also to discuss data availability. The officers were also consulted in developing the list of key informant interviewees and case studies. Through these consultations, some preliminary evaluation information was also obtained.

Documents and Literature Review

The evaluation included the review of documents and literature containing information relevant to the initiatives.  Three main types of documents were assessed and analyzed during the evaluation:

• General Background documentation (e.g. TB Submissions, documents that describe CEDI and AII history, rationale, theory, etc.);
• Program & Policy Documentation (e.g., Departmental Performance Reports, departmental database, project files); and
• Literature on community adjustment programs, best practices in economic adjustment program design and the mountain pine beetle.

File Review

Using a customized data extraction template, the evaluation team analysed all information contained in the department’s databases (Project Gateway and the GX financial system) and other department data collection systems including paper files.  The database review was completed in July 2010 and the file review in British Columbia was completed in August 2010.  Initially, 140 of the 144 CEDI projects listed community development as the sub activity while the other four listed community planning (one project) or community adjustment (3 projects).  The sub activity for the community adjustment projects was later changed to community development and new Program Activity Architecture indicators were added.   All of the Airport projects linked to the community development sub activity.

Of the 144 CEDI projects funded, 83 (58%) were ongoing and scheduled to complete by March 31, 2011.  Eighty-six projects (60%) focused on community capacity building, 37 (26%) on economic diversification, 19 (13%) on economic infrastructure and the remaining two on value-added forestry.  There were relatively few value-added forestry projects funded under the CEDI because softwood lumber manufacturers and exporters were not eligible for funding as a consequence of the Softwood Lumber Agreement restrictions imposed on those companies and their subsidiaries.  Fifty of the CEDI funded projects were tourism related (mainly tourism development studies and feasibility studies). Other common project types included community economic development studies (19), forestry industry development projects (17), industrial land development projects (15), training projects (10), and agriculture industry development projects (9).

At the time of data collection for this evaluation, the departmental liaison officers for the airport projects had been reassigned to other duties within the regional office.  Therefore, the paper files in Vancouver were the only source of information on these projects.

Comparative Analysis

The evaluation included a review of comparable programs to both initiatives, including:

1. A literature review of similar programs targeting pine beetle affected areas;
2. Fourteen interviews with officials that are involved or have been involved in similar economic adjustment programming;
3. Analysis of the results to develop a profile of similar programming, review the CEDI and AII against these programs, and to identify best practices and potential implications for future programming; and
4. A review of the literature regarding best practices in economic adjustment program design.

The review concluded there is no ideal economic adjustment program for all situations; however, a review of comparable programs identified success factors for economic adjustment programming.  As indicated later in the report, CEDI and AII addressed some of these success factors but key informants claimed that others, such as monitoring, were more difficult or not addressed.

Interviews with Internal and External Key Informants

The evaluators completed 49 key informant interviews.  The 49 key informants included:

• 13 interviews with departmental staff involved in the CEDI/AII.  These 13 staff were chosen because they were identified as the most involved and familiar with the programs;
• 19 interviews with staff from 16 of the 19 Community Futures Organizations involved in the CEDI.  Although input from all 19 Community Futures Organizations was sought, three of the 19 declined participation;
• 10 interviews with representatives of other government departments and stakeholders involved in/familiar with the CEDI/AII.  These 10 representatives were chosen based on literature research and snowball sampling; they were involved in the delivery of other programs and services targeting similar communities; and
• 7 interviews with experts in the field of community adjustment/diversification.  These experts were academics and professionals, selected because they were knowledgeable about community adjustment/diversification issues in BC, particularly in the context of the mountain pine beetle infestation.

Proponent Survey

The proponent survey attempted to contact at least one proponent from each funded project.  The survey successfully contacted representatives of 105 of the 144 (79%) CEDI projects and all 3 (100%) of the airports.  The proponent survey also contacted proponents from non-funded projects including some: 1) from each zone; 2) from each of  the four categories of funding applicant:  independent, small/medium sized enterprise, aboriginal and municipal; and 3) who were rejected at the Expression of Interest stage and some who were rejected at the full proposal stage.

Consultants developed and pre-tested the questionnaire and conducted telephone interviews with 118 proponents of funded projects including:

1. Community Economic Diversification Initiative Proponents:  a representative sample of 114 proponents of 105 CEDI-funded projects (38 led by independent organizations, 36 led by Aboriginal organizations, 31 led by municipalities and 67% from zone 2);
2. Airport Improvement Initiative Proponents: four proponents of the three funded airports (one respondent from Kamloops, one respondent from Kelowna and two respondents from Prince George).

Consultants developed and pre-tested the questionnaire and conducted telephone interviews with 98 proponents of projects which did not receive funding including: 48 projects led by independent organizations, 22 projects led by small/medium size enterprises, 16 projects led by Aboriginal organizations and 12 projects led by municipalities.  The evaluators attempted to contact all proponents submitting Expressions of Interest; however, only 98 representatives were willing and available for interview at the time of the evaluation.

Data collection spanned a period of four weeks between early November through early December 2010 with original contact information acquired through the departmental project database. Most proponents contacted were managers, directors, owners, or senior staff in municipal organizations, First Nations bands and other community organizations

Key informant interviews and survey results reported in Section 3 and Section 4 of this report include average respondent ratings to questions aimed at assessing the relevance and performance of the initiatives using a scale of 1 to 5, where 1 is no impact at all, 3 is somewhat of an impact and 5 is major impact.

Case Studies

Case studies for CEDI were selected by the program officer to represent a variety of challenges and obstacles encountered across both zones and the four program objectives.  A total of eight case studies were completed for CEDI.  In the absence of a program officer during the data collection stage for the AII, the evaluation team chose the Prince George Airport as the largest of the three airports.  A total of 17 stakeholders were interviewed for the 8 CEDI case studies and 2 stakeholders were interviewed for the one AII case study.

Outcome Assessment

To build stronger evidence of the socio-economic impact of CEDI/AII, the evaluation explored  the feasibility of two options:

• Option 1: using pre- and post-initiative community-based socio-economic indicators for affected BC communities from sources such as Statistics Canada or the existing mountain pine beetle literature.  The indicators would measure the socio-economic impact of CEDI/AII.
• Option 2: Including control communities for comparison purposes.

The report-based nature of many of the projects and the lack of timely regional data from existing data sources precluded option 1.  The following methodology was developed for option 2:

1. Thirty-one CEDI projects were identified under the following seven sectors:  1) Tourism Development (7 communities); 2) Energy Plan Feasibility Studies (7 communities); 3) Community Economic Development Strategies (5 communities); 4) Downtown Revitalization (3 communities); 5) Forestry Related Feasibility Study (3 communities); 6) Land Development Study (3 communities); and 7) Agriculture Feasibility Study (3 communities).
2. The community associated with each of the 31 CEDI projects was matched with a comparator community that had experienced some level of Pine Beetle devastation but had not received funding for that sector under CEDI (although they may have received funding for another sector).  The communities were matched based on region, population, Aboriginal status and level of forest dependency.
3. Four comparator airports, not funded under AII, were matched to the 3 AII funded airports based on similar volumes of passengers enplaning and deplaning in 2008. The four airports were:  Victoria, Abbotsford, Comox and Fort St John.  During the time period of the Mountain Pine Beetle Program, these airports had undertaken various upgrades including improved taxiways or lighting and equipment purchases.
4. Interview representatives of the 31 comparable communities and all of the comparator airports.

Focus Groups

All departmental staff responsible for the initiatives and all Community Futures Organization’ staff involved in the delivery of the initiatives were invited to attend one of the two focus groups conducted in Vancouver in early January 2011.  The sixteen focus group attendees  included six departmental staff and ten Community Futures Organizations’ staff. The departmental staff attended in-person while the Community Futures Organizations’ staff participated via videoconference.

A consultant presented the field research findings at the focus groups and then facilitated group discussions. The objectives of the focus groups were to review and confirm the field research findings and explore ways to enhance the effectiveness and efficiency of similar programming in the future.

Limitations of the Methodology

Project File Review:  it was difficult to identify which of the 144 CEDI projects were incomplete as of July 2010:  the departmental database reported 83 incomplete projects while the British Columbia office reported 51 incomplete projects.  There were some financial inconsistencies with the three Airport Improvement Initiatives.  Specifically, 1) Kamloops Airport:  the most recent client report (September 2009) estimated the project cost at $24.34M, however the financial system and the departmental database still had the original value of 18,500,000; 2) Kelowna Airport: a small discrepancy between the updated project cost on the project gateway information page ($7.82M) and the original cost of $7.97M indicated in the rest of the project gateway documentation including the amended contribution agreement.  The total project costs differed in the financial system and the departmental database for 71 amended CEDI projects and one AII project, likely because the financial system had not been updated to reflect the amendments.  Therefore, the departmental database data were used to calculate leveraging.  Also, the amount disbursed on the Kelowna airport according to project gateway ($1.35M) disagreed with that shown in financial system ($1.12M) (June 30, 2010).

Case Studies: Case studies enable in-depth analyses that are otherwise impossible. Given the expense and time involved in completing a case study, the goal was to purposefully select cases for contextual descriptive potential. While quantitative samples aim to be representative and generalizable, case study sample selection is non-random and intended to provide insight into the complex processes underlying a small number of projects.  The case study interviews relied on respondent experience, perception and recall at the time of the evaluation and results may be biased accordingly.  Some respondents, for example, became involved with the project after the completion of the application process, impeding collection of application-related information for that project. There was also overlap with the proponent surveys as the same 19 proponents interviewed for the case studies were interviewed for the survey.  In addition to recall bias, there is the potential for measurement error related to using questionnaires that were not tested for validity or reliability.  For example, the meaning of terms such as community capacity and economic infrastructure were open to respondent interpretation.  To compensate for potential bias, the case studies in this evaluation serve as one line of evidence to complement the other lines of evidence.

Key Informant Interviews:  Representatives of some funded projects had moved to different organizations, retired or did not precisely remember the project details.  Every effort was made to update contact information and give interviewees time to review documentation prior to the interview. Many of the funded projects were still ongoing and therefore the interviewees estimated the impacts whenever possible.  Of the 49 key informants interviewed, six were able to provide limited information on the Airport Improvements Initiative and their information was limited to a particular aspect of the initiative: one of the airports, one time period (such as the end of the program) or one activity (i.e. reporting, monitoring).  As a result, the key informant information reported in section 3 often relates to both CEDI/AII and, because it is already in section 3, is not repeated in section 4.   The one departmental staff member who was very knowledgeable about the Airport Improvements Initiative was unavailable for interview during the regular data collection period; this individual completed a key informant interview late in the evaluation process and the information was integrated into section 4.

Proponent Surveys:  The main challenges included: 1) there were eight instances where information on multiple projects was obtained through a single interview, which could potentially bias and/or generalize the responses. For this reason, each respondent was counted as only one data point regardless of the number of projects in which they were involved; 2) For non-funded projects, some companies/organizations no longer existed at the time of the evaluation. This implied that some of the respondents most affected by not receiving funding were excluded from the survey. A broad sample of non-funded projects was contacted to minimize this risk; 3) many projects were either recently completed or not completed yet, leaving too little time to realize economic impacts.  Questions were therefore worded to elicit opinions of respondents in regards to the potential economic impacts for projects which were not complete (or recently completed).

Outcome Assessment:  The size of the sample reporting a similar project was relatively small:  26 of 31 comparator communities agreed to an interview, of which 16 had undertaken a similar project.  Most of these projects are still ongoing and few reported impacts.  Nine of the projects were studies with a report being the only output.  The recommendations of these reports will need to be implemented before impacts are measurable.  Furthermore, most projects began recently with only six of the 16 projects being complete at the time of the interview.  Even in the situations where the project was completed, it is still too early for tangible results to be evident.  The projects may not be directly comparable:  the projects that were chosen in the comparable communities were from the same sector but often had a different focus in terms of, for example, the nature of the study or research or the type of infrastructure being developed.

Attribution:  Determining the net impacts of the department’s activities is challenging because comparable baseline data is lacking and, over the long term, it is difficult to disentangle the contributions of the CEDI/AII from that of its funding partners and the many other factors influencing the outcomes over a large number of years.  This evaluation uses contribution-focused analysis to ascertain whether the initiatives’ activities contributed to the achievement of objectives.  Contribution-focused analysis is not intended to establish the unique impacts of the program’s activities, but instead to indicate whether and to what extent they played a role in achieving strategic outcomes.

Community Economic Diversification Initiative

Relevance

Need for Program

The communities targeted by CEDI were highly forest-dependent and felt their well-being was significantly threatened by the mountain pine beetle.  A study, published in January 2007, assessed vulnerability in four rural BC communities; vulnerability was defined as susceptibility and ability to adapt/cope with the mountain pine beetle threat.  The study concluded that the high level of “risk awareness” in the communities created a potential and readiness for institutional change and political action.⁴ This readiness for action was echoed by first nations communities in the First Nations Mountain Pine Beetle Impact Assessment (2006) involving 99 bands representing 58,682 people:  31% of survey participants identified economic development, training and employment as priorities in their communities.  Many affected communities, being small, rural, Aboriginal and forest-dependent, lacked the capacity and tax base to address a complex and lasting problem like the mountain pine beetle infestation.  Furthermore, the department and the Community Futures Organizations’ staff noted that some CEDI-funded proponents were seeking additional funding for follow-on projects, indicating there was and continues to be a need for CEDI. In fact, focus group participants felt that one-time initiatives like the CEDI and the AII should be coordinated with other programs so that the affected communities were funded from the CEDI/AII for the first phase of the projects with subsequent funding coming from other sources.  Key informants who questioned the need for the CEDI/AII acknowledged the need for funding and support in the affected communities but indicated that the CEDI/AII was designed and delivered under pressure and may not have been the best approach to addressing the needs of the affected communities in a meaningful and lasting way.

The CEDI/AII complemented other government and private sector programs. The department used several approaches to minimize funding duplication and maximize leverage opportunities: 1) representatives from other relevant funding sources were briefed at a funders’ table organized during the launch phase of the CEDI; 2) Expression of Interest screening criteria assessed whether or not each proposed project could obtain funding from a different source; 3)  departmental and Community Futures Organizations’ staff referred project proponents to alternate funding sources.

The program filled a funding gap not addressed by other funders.  Twenty-seven percent of projects were developed in response to availability of CEDI funding and another 21 projects were completed earlier because of CEDI funding.  Most project proponent respondents said there was a 25% or less likelihood that their project would have gone ahead without assistance from the department (82%) and they were unaware of other funding programs, although a few cited programs such as the Northern Development Initiative Trust, Community Futures and various beetle coalitions groups. Almost half of the respondents (49%) of non-funded projects reported that their projects were no longer pursued, cancelled, or suspended and felt that the CEDI/AII complemented the other available funding programs particularly via non-financial support and assistance. The case studies support the finding that CEDI/AII funds were moderately to highly incremental. Of the eight case study projects, four were found to be totally incremental (that is, the project would not have proceeded in the absence of CEDI/AII) and four of the projects would likely have proceeded, but only after a delay or with a reduced scope (partially incremental).  The comparator communities examined in the outcome assessment were somewhat successful in procuring alternative funding; the comparator communities that did not obtain other funding did not place a high priority on the project.

Alignment with Departmental and Federal government priorities

The initiative’s objectives supported the department’s strategic outcome at the time, Community Economic Development, and aligned with the economic focus of the department’s mandate.  The initiative’s activities and outcomes align with the federal priority of Strong Economic Growth.

The majority of departmental and Community Futures Organizations’ staff reported that CEDI was consistent with departmental priorities as well as priorities of the Government of Canada.

Consistency with Federal Roles and Responsibilities

Residents of areas most affected by the mountain pine beetle believed the federal government should be involved.  A 2004 survey of approximately 2800 households in 13 rural communities in BC and Alberta asked respondents: “Should this organization be responsible for mountain pine beetle activity” and “How much trust do you have in the organization to properly manage for mountain pine beetle activity”.  Eighty-one percent of respondents thought the federal government should be responsible but they had little trust in the federal government (2.1 on a scale of 1 being no trust to 5 being complete trust).⁵

The majority of departmental and Community Futures Organizations’ staff, stakeholders and experts indicated that the CEDI/AII was aligned with an appropriate role for the federal government.  The federal government has a mandate to ensure that all regions in Canada including the rural communities are prosperous. The federal government also has a role in addressing any funding/support gaps not met by provincial/regional/local governments to deal with the mountain pine beetle infestation.

Performance: Achievement of Expected Outcomes

General Findings

Overall, the CEDI was somewhat successful in achieving its intended objectives.  However, there were implementation issues and many projects were incomplete as of the March 2009 deadline.  The projects funded under the CEDI enabled the affected communities to deal with the consequences of the mountain pine beetle infestation by enhancing their economic capacity and infrastructure, improving their business climate, creating economic stability, generating economic opportunities, facilitating the establishment of new businesses/industries, and diversifying the local economies.  All of the case studies demonstrate progress towards diversifying the local economy into areas such as tourism, technology, manufacturing and value-added forestry operations.

Key informants rated CEDI as moderately successful in achieving intended objectives.  Those who felt CEDI was less successful indicated that the funded projects only laid the foundation/groundwork and CEDI would have been more successful if planned as part of a comprehensive, coordinated and long-term strategy to address the impacts of the mountain pine beetle infestation with secured funding for the subsequent phases.  The efficacy of CEDI was limited by:  the lack of a proactive monitoring strategy; the absence of support from dedicated departmental staff for project implementation, particularly for proponents who lacked in-house capacity/expertise to manage their projects independently; and the global economic downturn.

Key informants identified the following critical success factors contributing to the efficacy of CEDI:

• a dedicated departmental team during the launch phase of the CEDI;
• solid partnerships/rapport with provincial government and other federal government departments;
• consultation with industry, academia/experts, and the communities affected by the mountain pine beetle infestation;
• the Aboriginal Engagement Strategy to facilitate Aboriginal CEDI application intake and assist Aboriginal project proponents with proposal development contributed to the efficacy of the CEDI.
• a two-step application process to facilitate the submission and screening of a large volume of CEDI applications within a short period of time; and

Case study proponents agreed that the two-stage application process was easy to understand and follow and found the electronic application process easy to use.

The CEDI faced some implementation issues.

The long approval time resulted in projects experiencing loss of momentum, missing the construction season, and losing leveraged funds.  There are several reasons for the long approval times: 1) the Request for Proposal process resulted in an unexpectedly large response creating volume delays throughout the approval process;  2) some clients lacked the capacity to refine proposals into projects and needed extensive, sometimes time-consuming, officer assistance; 3) the amount of due diligence required for projects with low levels of risk was the same as the due diligence required for projects with high levels of risk; and 4) the approval time after due diligence was complete frequently exceeded service standards.  Projects requiring environmental assessments, consultation with First Nations communities or the services of specialists/experts not available locally also faced extended timelines.

The dedicated departmental staff did not continue after March 31, 2009; instead, staff were responsible for CEDI and non-CEDI activities.  The usual departmental project monitoring began, leaving some project proponents without the support/monitoring they needed to complete their projects.

The CEDI faced some constraining factors.

Although the department provided advances to some CEDI proponents, many project proponents were reimbursed only after expenditures were made.  Proponents who did not receive advances often faced difficulties carrying out project work, particularly the volunteer-supported organizations that did not have adequate collateral/expertise to raise funds.

Due to the absence of a streamlined and customizable project database system to process applications and generate reports using application data, departmental staff had to work overtime to manually produce the reports needed for effective decision-making, which was not efficient in terms of  cost or time.

The global economic downturn also affected some of the CEDI funded projects. For example, one of the project proponents that received repayable assistance went into receivership due to sharp decline in sales and as a result, the project withdrew its funding request. In addition, some newly developed/expanded industrial parks found it challenging to attract new businesses during a time of declining economic activities.

Performance Measurement

The Performance Activity Architecture performance indicator for most of the projects was “number of instances of increased community stability”.  According to the departmental Program Activity Architecture Performance Plan, the definition of this indicator is:

“The number of instances of increased community stability where the CFDC, FEDO or project proponent played a key role that resulted in an outcome during the reporting period such as the development of a community needs assessment study; development of a labour force adjustment initiative; a business retention and expansion program; a “buy local” campaign; a youth retention strategy, etcetera.  It can also include making a loan to an “essential service” business in the community.”⁶

The definition of the indicator is very broad and raises questions as to what is actually being measured.  Two officer reports stated that:  “For the MPB program, simply completing the project indicated that this indicator was met” and “The MPB program was restricted to only using this indicator.  Therefore, it is used solely to determine if the project was successful”.  However, 15 projects did not use this indicator.  The case studies also reflect confusion as to what is being measured:  one of the case studies indicated the number of instances of increased community stability would increase from zero to ten, with no explanation as to which ten communities would be affected or how; another case study indicated the “outcome will be determined at the time of project completion”.

CEDI realized its key program outputs

The large volume of applications indicates the program achieved its key program outputs of high awareness, understanding, response and funding to target communities. The program also had a variety of horizontal impacts including: aboriginal (63), environmentally sustainable development (10) and youth (10).

The CEDI funded projects have generated a wide range of impacts.

CEDI funded projects addressed four programming objectives:

• Increasing community capacity to cope with the mountain pine beetle infestation;
• Diversifying community economies away from forest-dependence;
• Creating opportunities for diversified and value-added forestry
• Increasing economic infrastructure.

Expected Outcome: Increased community capacity

Definition: positioning communities to exploit new diversification opportunities.  Results: 1) completion of studies facilitating exploitation of diversification projects 2) leadership mobilizing stakeholder action towards community diversification plans/projects.

The nature of community capacity building projects made them vulnerable in several ways:

• The projects often involved consultants who, when the project approval was delayed, took other work.  When the project was approved, the proponent then had to find a consultant.
• Some proponents needed significant departmental support to implement their projects at a time when the dedicated CEDI staff was being reassigned to non-CEDI tasks and the department’s focus was shifting to the Economic Action Plan.
• The success of these projects was dependent on follow-up funding for implementation.

Key informants noted that CEDI funded projects galvanized the communities affected by the mountain pine beetle infestation and brought together a diverse group of people and organizations in each community for the first time to collectively come up with solutions to address the impacts of the infestation (e.g. collaboration between tourism operators and First Nations communities). In addition, the projects served as good learning exercises for many individuals and organizations that did not have experience in undertaking similar projects but now have the capabilities to design and implement follow-on/spin-off projects.  Project proponent respondents agreed that the projects were successful in increasing community capacity and ability to cope with the mountain pine beetle infestation (3.9/5).

Expected Outcome:  Diversifying Community economies

Definition:  communities restructuring economic fundamentals away from forest-dependency and into new sustainable engines of economic growth.  Results:  economic diversification into new or emerging non-traditional sectors such as tourism, manufacturing, entertainment and high-technology sectors. 

One of the key lessons learned at the 2008 Mountain Pine Beetle conference regarding employment in tourism as an alternative to the forestry sector was:  “Increased tourism could partially or fully offset the impact on employment.  This would further offset the negative effects on monetary indicators.  Increased traditional agricultural activity will minimally offset the impacts on employment and monetary indicators”⁷  However, First Nations participants at the same conference indicated tourism may not be a viable alternative for First Nations communities:   “community members have attempted to diversify by expanding into tourism.  Small business operators soon discovered the cost of insurance is too high to make a living at tourism-based jobs.”

Key informants reported that while a number of CEDI funded projects worked toward diversifying the local economic base, these projects were seen as only the beginning of a long-term undertaking and therefore, it was generally considered too early to realize any meaningful impacts in this area.   Project proponent respondents felt projects were successful in diversifying the local economy away from forest dependence (3.7/5).

Expected Outcome:  Creating Opportunities for Diversified and Value-Added Forestry

Definition:  communities capitalizing on niche opportunities to create value from forest resources.  Results: 1) new or expanded businesses created or maintained; 2) wealth generated by value-added wood products and tradeable services derived from forest resources.

Key informants indicated that relatively few CEDI projects received funding in this area because softwood lumber manufacturers and exporters were not eligible for funding. Some projects designed to explore the feasibility of utilizing forestry resources for alternate uses such as bioenergy were highlighted as projects that generated impacts in this area.  Project proponent respondents agreed that projects were less successful in creating opportunities for diversified value-added forestry (2.9/5).

Expected Outcome: Increasing Economic Infrastructure

Definition:  communities exploiting new systemic growth opportunities that attract incremental investment.  Results: improvements in transportation, communication, commerce, energy, skills, technology or knowledge. 

Key informants stated that projects supporting new economic infrastructure including the AII projects tended to be larger value projects. Apart from airport expansion/upgrading, these projects supported the development/expansion/upgrading of a wide range of infrastructure such as industrial parks, visitor information/interpretive centres, walking/hiking trails, fibre optic/broadband networks, connector roads/bridges etc. Project proponent respondents indicated projects had some success in creating new economic infrastructure.

Ultimate Outcome:  Developing and diversifying the Western Canadian Economy

Literature on the role of community development in achieving a diversified and developed regional economy

Under terms such as “community adjustment”, “community transition” and “community economic development”, governments have pursued various approaches to developing rural community economies towards long-term goals of sustainable regional economic diversification.  Key lessons pertaining to regional development policy show, for example:  “a range of policy instruments and measures will need to be brought into play, including financial assistance for developmental projects; seed monies; support for innovation and technology; support for community development and capacity building; resources for consultation, planning and community empowerment and involvement; and expert and technical assistance”. ⁸

Evidence that Community Economic Diversification Initiative Developed and Diversified the Canadian Economy

The CEDI laid a strong foundation but did not secure funding for the long term

CEDI was intended to be the first step in the federal programming response to the beetle infestation.  CEDI supported many feasibility studies that would diversify the economy when implemented through follow-up funding. However, the recession forced the diversion of the follow-up funding into economic stimulus rather than the mountain pine beetle infestation.  Among the four categories of projects, some, such as economic infrastructure, had longer term adjustment outcomes than others.  It is still too early to determine the long-term success of the initiative and some key informants claim the funded projects only laid the foundation/groundwork and CEDI would have been more successful if planned as part of a comprehensive, coordinated and long-term strategy to address the impacts of the mountain pine beetle infestation with secured funding for the subsequent phases.  Yet some projects - the case studies for example - show early evidence of progress towards diversifying the local economy and therefore contributing in a small way to diversifying the Western Canadian economy.  Although the outcome assessment component of the evaluation was meant to provide insight on the extent to which CEDI contributed to diversifying the Western Canadian economy, it ultimately failed to improve the weak evidence-base underlying the measurement of CEDI’s impacts.

Unintended Impacts

The projects mobilized communities but delivery problems adversely affected the department’s image in the communities.

Under the CEDI, the department provided funding to many small-scale community-based organizations that previously had not received funding from the department or worked with the department directly since typical funding recipients tend to be larger, more established organizations with highly developed skills/expertise to manage large-scale, complex projects. Working directly with a new type of funding recipient enabled the department to raise its profile at the grassroots community level and cultivate new relationships.  The projects were catalysts, generating community interest and support, building capacity for future projects, generating employment and developing cooperation and partnerships.  On the negative side, the tight application deadlines combined with the long approval time lead to a sense of mismanaged expectations and disillusionment in the communities, adversely affecting the image of the department and the Community Futures Organizations.  For example, one case study proponent understood approval would be done in two months and it took 14 months, creating significant stress and frustration for the client and leaving 13 months to complete the project.

Some projects resulted in impacts on a much greater scale than originally anticipated and project proponent respondents claim the projects will likely lead to long-term impacts on the communities. For example, the creation of a community garden generated widespread interest in local and sustainable food production/farming in a community, which, in turn, initiated the process to establish a local agricultural distribution venture as a way to diversify the local economy. The community garden became a symbolic beacon of hope for a community going through tough times.  Several of the case studies proponents indicated their projects garnered media attention and affected the public through increased awareness, understanding and raising the community profile locally/internationally.  One of the case study projects was invited to set up a booth at the 2010 Olympics in Vancouver.

Performance: Demonstration of Efficiency & Economy

Economy

For the most part, CEDI was well-structured but design changes could have improved effectiveness

CEDI Design and Launch

The designing and launching of the CEDI were executed effectively and efficiently, particularly in light of the tight timeline.  The department had a dedicated team set up to launch CEDI, allocated the necessary resources to the CEDI/AII, and was able to leverage the resources/expertise of the provincial government and other federal government departments as well as academics/experts and relevant representatives from industry and the affected communities including First Nations bands.  As a result, key components of the CEDI/AII such as the types of assistance provided, the eligibility criteria, promotion and outreach, initial screening of applications, and the provision of assistance/support to applicants in preparing Expressions of Interest and proposals was designed and delivered in an effective and timely manner.

Most respondents of non-funded projects said the CEDI was well-structured but some experienced delays in getting responses, found the application timeline difficult to meet and indicated there could be a clearer focus on primary objectives. The project proponent survey respondents found the assistance and seminars provided by the department and Community Futures Organizations to be helpful.  Proponents of non-funded and funded projects felt there could have been more promotion of the program.  Focus group participants stated that promotion of the CEDI and the AII was done during the summer months when many potential project proponents were away on vacation, reducing the effectiveness and reach of promotional activities.

CEDI Request for Proposal and Approval Process

An interim review of CEDI’s Request for Proposal process⁹ concluded:

• The large number of Expressions of Interest overwhelmed both project officers and the department’s technical systems, making it difficult to meet service standards and proponent expectations.
• The program neglected to complete a value for money comparison to select projects with the best potential outcomes because it was not oversubscribed at the proposal stage.
• The program “generated a high degree of awareness, program understanding, quickly identified projects, and, while formal in structure, also allowed for project officers to undertake developmental work with proponents and support capacity building in communities.”

The departmental key informants raised several issues surrounding the approval process.  The CEDI projects involved file reviews by more than 10 staff within the various departmental offices in Vancouver, Edmonton and Ottawa.  Each project underwent extensive due diligence review and some staff noted that the information required from applicants was onerous given the relatively low value of funding sought. In order to fast-track the approval process, department staff batched the more detailed or complete proposals together and sent them out for approval while they worked with applicants to fine-tune the proposals that required additional work.  This approach worked well for some batches; however, the entire batch could be stalled if it contained one questionable application.

The lengthy approval process combined with a lack of communication with applicants affected the projects in several ways:

• The relatively brief period of time applicants had to apply for funding signalled that the federal government was committed to disbursing the CEDI funds on a priority basis to assist the affected communities.  But the long approval time, in stark contrast to the time given to applicants to apply for funding, did not deliver on the raised expectations and resulted in a sense of disillusionment in the communities. There was also a resultant loss of momentum as some project proponents became engaged in other projects/priorities while awaiting approval;
• Projects involving construction work missed the construction season and required extension.  This problem was mentioned in several of the case studies and sometimes meant that the proponents were unable to adequately source for the best price because they had only a few months to implement the project;
• Projects requiring environmental assessment or consultation with First Nations communities did not have a realistic timeline to complete such complex procedures. The lengthy approval process further pushed their timeline;
• Some projects required the services of specialists/experts (e.g. geomorphologist) not available locally and required additional time to secure such services from outside their respective regions.  The delay in approval further pushed their timeline; and
• Some projects lost the funding leveraged from other sources while they awaited departmental approval because the other funders could not remain committed.
  The extent and impact of the long approval process was a common theme throughout the case studies.  As previously mentioned, one case study proponent waited 14-months for approval.  An interval spanning six months or more from application receipt to project approval appeared to be the norm rather than the exception among the case studies:
• One proponent underwent a 16 month approval process because the department wanted presales.  The proponent’s funding was eventually approved in April 2009 with project completion in March 2010.  Since work could not be done in the winter, the proponent was essentially faced with completing the project in six months.
• Another project was initially approved for less funding than requested, leading to several months of negotiation before obtaining the funding needed to do the project.  The proponent was then notified of funding approval but had to wait six to seven weeks for the announcement by the federal government before they could start the work.  Three other case study proponents experienced approval delays of between six months and one year, including, in one case, a six-week delay while waiting for the funding announcement.

Some case study proponents indicated they didn’t know deadlines would be extended and worked very hard, under considerable stress and additional expense, to meet the original deadlines – in effect they were punished for meeting deadlines.  The deadline extension was also stressful for departmental staff who couldn’t announce an extension until it was approved.  The long approval time, combined with the prevalence of a relatively high number of project proponents who lacked in-house capacity/expertise to manage projects independently and the absence of proactive and consistent departmental follow-up and support, resulted in many projects not completing as per the original deadline and requiring an extension. The extensions of the CEDI program was cited as the major reason as to why department and Community Futures Organizations’ staff felt that the CEDI was not implemented as planned.

CEDI project monitoring and follow-up was lacking

The dedicated departmental team set up for the CEDI did not continue after the administrative funding for CEDI ended on March 31, 2009.  Departmental staff were then assigned both CEDI and non-CEDI responsibilities.  The monitoring for the CEDI projects was the same as for other projects under other programs and did not provide capacity or expertise to help clients manage their projects.  This was problematic because many project proponents lacked in-house capacity/expertise to manage their projects independently and needed proactive and frequent follow-up/support.  The lack of monitoring may partially explain why 25% of projects failed to submit their first progress report within the 12 months following their acceptance of the letter of offer.

Almost all (78%) projects were delayed for a combination of reasons most often involving the slow approval process, weather conditions and delays related to consultants/contractors;  however, respondents said that since there was flexibility on the implementation timeline side, these delays did not significantly affect the success of the projects.  The final reports for 48 of the 61 completed projects indicated that the department could not have engaged the client more effectively although some funded and non-funded project proponents felt communication with project officers could have been improved.  The case study proponents were generally pleased and grateful for the help from the department; problems most commonly cited by case study proponents related to staff going on vacation without someone to fill in for them, the large amount of paperwork required for reporting purposes, delays in processing claims and high departmental staff turnover.  A clear succession plan was not always in place following the departure of a number of departmental staff involved in the CEDI.  The initial extension of CEDI created problems related to a lack of a clear extension strategy along with the allocation of additional resources to ensure adequate departmental oversight of the outstanding CEDI funded projects.

Project outputs and outcomes were moderately cost-effective and the CEDI/AII obtained good value with respect to the use of public funds.

The majority of staff and project proponent respondents felt CEDI/AII was moderately cost-effective.  Staff indicated that conditions such as the use of competitive bidding and supplier quotes that project proponents were contractually obligated to meet ensured that project outputs and outcomes had been generated at the lowest possible cost.  Contractual obligations were not strictly followed in every case because some proponents were forced to pay more than the lowest possible cost in order to meet project completion deadlines. The administrative costs of delivery were also seen as consistent with the costs of delivering similar initiatives/programs. However, the assessment of administrative costs was based on the initial allocation and did not take into account the administrative funds spent on the CEDI subsequent to the extension.

Efficiency

CEDI incorporated some best practices but efficiency could have been improved.  The following discussion compares CEDI to comparable programs and best practices in economic adjustment.

CEDI delivery costs were reasonable.  The CEDI program operating costs are within the normal costs of program delivery of the reviewed programs.  The total budget of $36.6 million for CEDI, includes $3.48 million for management costs including operating costs and a Public Works and Government Services Canada accommodation charge of 13% of salaries. Thus delivery costs were 9.5% of program funding.  The average delivery costs for the reviewed programs were under 10%.

CEDI requirement to leverage funds appeared to be reasonable.  The department did not provide 100% funding to any CEDI project.  On average, the department funded 73% of project costs; however the departmental contribution varied considerably across projects (ranging from 15% to 92%).  Collaboration between stakeholders is a recognized success factor¹⁰ ; the CEDI funded projects leveraged funding from a wide range of sources including provincial/regional/municipal governments, other federal government funding programs, economic development organizations/trusts, Beetle Action Coalitions, First Nations/Aboriginal organizations, the private sector (e.g. credit unions, chambers of commerce etc.), and the project proponents themselves (Table 3.1). Table 3.2 summarizes the department’s contribution to the total project costs and the dollars contributed by other funders for each departmental dollar invested (dollars leveraged).  Although leveraging varied across zones and program objectives, overall each department dollar was matched by fifty cents from other contributors.  A leveraging ratio of fifty cents is low compared to the $1.14, $1.25 and $2.00 leveraged by three similar programs¹¹.  However, the requirement for leveraging in an adjustment program can have a negative impact, especially in depressed areas.  Conversely, some staff indicated that the department had a more rigorous set of screening/assessment criteria to approve funding compared to other funding sources, which meant that securing funding from the CEDI/AII helped open doors to other funding sources. Project proponent respondents agreed that CEDI funding helped leverage funding from other sources.

*All numbers in millions CAD$
 

Note: All numbers in millions CAD$

The top down design had some advantages.  A top down design is a design largely imposed by the funding agency.   Although the priorities of the program were based on recommendations resulting from a one-year community and public consultative process, the program design was top-down in terms of project selection, delivery and monitoring.

The best practices literature notes that the advantage of the top down approach is that it ensures that aggressive lobbyists do not control the program to the detriment of vulnerable social groups.  CEDI made a point of preparing First Nations clients for the program and making the involvement of First Nations an advantage for rankings of projects. The top down approach enabled the process to be more inclusive. The CEDI program had developed very clear guidelines and criteria that were comprehensive and easily understood. The program also used a call for an “Expression of Interest” from potential project proponents to reduce the amount of unnecessary effort on the part of proponents that had no chance of success. Successful programs follow this approach to maintain a higher level of client satisfaction; however, some project proponent respondents felt the application process could be simplified for small projects.

Another feature of the program design that was made possible by the top down approach was the targeting of communities that were suffering the most from the pine beetle infestation.  However this approach can also mean that communities with low capacity for carrying out meaningful projects are targeted.  Other programs have focussed on increasing capacity before and during an economic adjustment initiative.  This is particularly important when an initiative is centrally administered.

CEDI did not advance money to all proponents who might have benefitted from an advance. Approximately one third of proponents received advance payments and the remainder were reimbursed after expenditures were made.  Reimbursement can create hardships for organizations in severely economically depressed areas, especially given the lengthy waits for reimbursement that occurred.  Some project proponent respondents found it difficult to undertake their project without advanced funding.  Other programs, recognizing the financial situation of their clients, will advance funds.  The Nechako-Kitamat Development Fund Society Grant Program advances proponents 50% of the value of the project on approval.  Case study proponents agreed that this caused significant cash flow problems for them.

The department partnered with the local Community Futures Organizations in order to provide information and assistance with project development and help with promotion.  According to the literature and program administrators, complementing centralized delivery with a local presence can slow the delivery process but can have the following positive impacts.

• Enhanced quality of applications generated and approved;
• Improved ability to consider the true capacity of communities to do projects or help to build that capacity;
• Increased flexibility in response to individual community circumstances; and
• Greater likelihood of reaching small communities or maximizing the fit of projects to priorities.

At least half of the case study proponents described their local Community Futures Organization as instrumental in promoting CEDI and the application process.  Project proponent respondents felt the program could be improved by engaging more local government involvement or the involvement of local organizations so that the funding agencies are more aware of the communities’ needs.

The centralized monitoring was unlikely to be as effective as local monitoring.  Local monitoring of project progress more accurately reflects the circumstances of the proponents efforts than centralized paper-based analysis.  Centralized monitoring often results in more paper and effort as the arms-length monitoring officer seeks to be reassured by the success of a project that has not been seen.

The short time frame between the program announcement and the application date created issues for applicants.  The program had at least the appearance being rushed into implementation.  This affected the quality of the applications and reduced the ability of communities with little local capacity to participate.

The term of the program limited its ability to mitigate pine beetle infestation impacts.
Although CEDI was meant to be the first step in the federal programming response, the follow-up funding for implementation was not secured and did not materialize.  At the time CEDI was implemented, community economies were booming but the long-term economic impacts of the pine beetle infestation were expected in 5-8 years. CEDI was a time limited program with little potential to effect long term adjustment without follow-up funding.  This is supported by evidence that shows that a time limited program restricts the opportunity for significant adjustment and reduces the potential for highly effective strategic programming with truly sustainable adjustment techniques.  In fact, one of the recommendations emerging from the evaluation of the two-year program upon which CEDI was based was to “set a longer program duration from the beginning.”¹²

Possible Improvements and Lessons Learned

There were several processes during the CEDI implementation that took too long for a two year program.  The database did not capture the lengthy amount of time involved in proposal development that occurred prior to receipt of the proposal. The departmental database indicates that the time between the receipt of the proposal and completion of the Due Diligence Report was at least 231 days for half of the projects (ranging  from 83 to 566 days).   Other time periods throughout the project lifecycle included:  1) an average of 14 days from time of completion of the Due Diligence Report to regional approval; 2) an average of 89 days from time of regional approval to ministerial approval; and 3) an average of 5 days from ministerial approval to issue of the letter of offer.

Focus group participants suggested a few changes that, if implemented in the design phase of the process, could improve the effectiveness of the program:

• Develop a template for one-time and time sensitive initiatives like the CEDI and the AII that could fast track the design and delivery processes. The template should be flexible enough to accommodate the unique needs of each program.   Technical system requirements should also be considered.
• Consult local delivery partners early in the design phase to pinpoint specific community needs and fund the most qualified proponents in each community.
• Adjust funding criteria to promote regional collaboration and discourage competition for limited funding among the individual communities located in the target region.
• When funding a large number of projects under different categories, it is necessary to recognize that the same timeline may not be realistic or appropriate for all projects.  For example, projects involving environmental assessment or consultation with First Nations communities tend to require more time.
• Allocate some funding to project monitoring/follow-up early in the process to avoid future problems (e.g. early monitoring/follow-up can ensure that funded projects are completed on time and avoid the need to extend project timelines).  An important consideration in developing monitoring strategies is the capacity of the funding recipients/project proponents.
• While staff may not be able to control the approval time associated with centralized decision-making, departmental staff can fast track the design and delivery of some processes including: 1) clarifying eligibility criteria early on to ensure that potential applicants receive information in a timely manner; 2) developing strategies/IT systems to deal with the high volume of applications; and 3) streamlining information requirements for applicants and working with them closely to ensure timely development/submission of applications.

Key informants suggested that the Community Futures Organizations and/or other economic development organizations could have played a more direct and effective role in assessing the project proposals as well as the capabilities of project proponents and providing the support/monitoring needed for successful implementation of the projects.  The focus group participants supported a more prominent assistance and monitoring role for local community economic development organizations.  A competitive bidding process to select local community economic development organizations for the provision of project monitoring services should assess not only cost but also the organizational capacity, community presence and quality of services required.

Project proponents felt more local government involvement or involvement of local/regional organizations would make funding agencies more aware of the communities’ needs.  Proponents suggested two other improvements:  1) incorporate a mechanism for timely dissemination of funding or turnaround time from claims submissions (e.g. staggered or advance funding); 2) shorten approval time or lengthen the time to complete projects to lessen the “hurry up and wait” feeling on the part of the proponent.

Airport Improvements Initiative

As previously indicated, most of the key informants were familiar with CEDI and had limited knowledge of AII and therefore most of the key informant information is included in Section 3 although it applies to both CEDI/AII.  The one departmental staff member who was very knowledgeable about the Airport Improvements Initiative completed a key informant interview late in the evaluation process and the information was integrated into the following section.  Additional information contained in the following section was collected from project proponents, the comparative analysis and the outcome assessment.

Relevance

Need for Program

The AII did not duplicate similar programs/initiatives but rather played a complementary role.  Every AII-funded respondent said there was a major need for programming such as the AII. Respondents explained that the scope of the department is more diverse than most funding operations, allowing more diverse types of projects to apply for funding.  Most AII-funded respondents said there was a 25% likelihood their project would have proceeded without departmental funding.  The four comparator airports selected in the outcome assessment component of the evaluation noted that runway expansions are impossible without public funds, which they were currently considering or actively seeking.  Although the comparator airports had made other improvements during the study period, the main reason why none had extended runways or expanded air terminals was lack of funding.

Alignment with Departmental and Federal government priorities

The federal government has a responsibility to ensure all areas of Canada are prosperous and the AII offered the targeted, smaller airports an opportunity to increase traffic flows and thus tourism for their respective regions.  Key informants felt that this initiative was well within the government's range of priorities (80% for experts, and 100% for stakeholders and department employees).  The objectives of the initiative aligned with the federal priority of Strong Economic Growth and supported the department's strategic outcome of Community Economic Development.  The department's economic mandate and past experience in delivering federal infrastructure programs in Western Canada positioned it to deliver the Airport Improvements Initiative.

Consistency with Federal Roles and Responsibilities

Federal investment in small airports contradicts the National Airports Policy that requires all incremental airport infrastructure be financially self-sustaining.  However, the federal government can, and has, invested in airports when it is in the national interest to do so.  For example, funding of the Moncton Airport (Sommet de la Francophonie in Moncton in 1998) and the Quebec Airport (the 400th anniversary of Quebec City in 2006) was deemed to be in the national interest.  The Airport Improvements Initiative addressed economic issues of public interest by funding airports in areas affected by the mountain pine beetle.  The economic viability of each project was demonstrated in business cases submitted by the proponents and through appropriate due diligence.

Performance:  Achievement of Expected Outcomes

General Findings

AII was economically driven and projects sought to support long-term structural adjustments to regional economies.  Project proponent respondents reported that the AII-funded projects were very successful in achieving their objectives related to extending runways and expanding airport capacity and ability to operate year-round.  Proponents said that the projects are likely to be very successful in the long run.  Long run success depends on the airports expanding their commercial activities and diversifying their lines of revenue and business, thus enhancing economic stability and replacing lost forest sector activity.  Projects successfully leveraged funding from other sources (4.8/5), increased community capacity and ability to cope with the mountain pine beetle infestation (4.5/5), and created a new economic infrastructure (4.3/5).  The AII projects had less of an impact in terms of diversifying the local economy away from forest dependence (3.3/5) and creating opportunities for diversified value-added forestry (2.9/5) since the nature of these projects did not allow for direct impacts in these areas.  In terms of impacts on local businesses, most respondents said it was too soon to tell.

Project proponents identified community and departmental support as the most important success factors.  Departmental involvement with First Nations bands was also critical, as was the power of large project to unite the community.  Furthermore, a commercially viable and solid business case supported each project and due diligence was appropriate to the project.  With the international appeal that comes with an airport, multiple communities also began to take pride in their increased international profile and made renewed efforts in other local initiatives.

The AII faced some constraining factors.

Two years is an insufficient amount of time to deliver a program, particularly when there are environmental and native rights issues involved.  Proponents highlighted two major factors constraining the success of the AII-funded projects.  The first was a delay to an aircraft parking area in one of the projects that came about as a result of an environmental impact assessment. Lengthy communications between the department and Environment Canada ended up delaying the construction by several months.  Departmental management disagrees with the proponent's assessment of the reasons for the delay and claims Environment Canada acted promptly and the delays were caused by the proponent's alterations to their project. The second challenge was to convince one of the local communities that the project was worthwhile.  They did not originally see the airport expansion as profitable, but were convinced to support it eventually.

Performance Measurement

The Program Activity Architecture indicator for all three projects was "the number of enhanced community services or facilities".  The unique indicators focussed primarily on increased flights or passengers through the airport and increased employment.  The Kelowna airport was the only complete project at the time of the file review and the final report indicated that the project could not have been improved, a follow-on project could enhance outcomes, the department could not have engaged the client more effectively and the project was within budget.

• Kelowna Airport:  according to the final report, results in terms of the completed runway extension have exceeded expectations.  The project achieved its performance target of an enhanced community service/facility.  The project has three other performance indicators that it expects to achieve by December 31, 2011:  1) an annual increase of 78 flights into the airport from Europe; 2) an annual increase of 19,000 passengers traveling through the airport from Europe; and 3) an annual increase of 88 person years of direct employment.
• Prince George Airport:  the final claim has been approved and the final reporting date is February 28, 2011.  The most recent progress report states that there is now an 11,400 foot runway indicating the achievement of its performance indicator of an enhanced community service/facility.  The project has three other performance targets it has yet to meet:  1) annual increase of 500 international cargo trans-shipment flights at Prince George; 2) an annual increase of 1600 fuel stops at Prince George Airport; and 3) an annual increase of 50 tourism-related international charter flights.  The project was delayed by the slow approval to implement a mitigation plan for an "at risk" species of toad and the weather.
• Kamloops Airport:  the first claim has been approved and the expected completion date is December 31, 2012.  The most recent progress report states that the runway is complete and therefore the project has achieved its indicator target of an enhanced community service/facility.  The project has yet to achieve its remaining performance targets of: 1) increasing the annual number of passengers traveling through the airport from 200,611 (pre-initiative) to 467,478; 2) increase direct employment in the region by 744 jobs; and 3) increasing the number of aircraft movements from 13,333 (baseline) to 15,085.  The project was delayed by the late completion of the environmental impact assessment/ archaeological impact assessment.

Expected Outcome:  Incremental Creation of Economic Benefits

Kelowna:  An air service market study conducted by consultants in 2008 found that the runway extension improved operational and safety conditions for current air carriers flying into Kelowna by:  reducing weight penalties associated with runway contamination, increasing safety margins for takeoff and landing, providing greater obstacle clearance on takeoff and reducing aircraft noise impact.

Prince George:  Case study data indicate the airport increased the community's:
1) international profile; 2) ability to attract new business; and 3) hope and optimism.

Ultimate Outcome:  Developing and diversifying the Western Canadian Economy

Literature on diversifying and developing the regional economy

Research supports investment in "asset building", including improvements to community infrastructure and transportation, as a means of stimulating regional development:  "In Canada and internationally, asset building is increasingly being recognized as an important strategy for social and economic development; in the United Kingdom, for example, it has been described as the 'third pillar' of the social welfare system, along with education and income security".¹³

Evidence that Airport Improvement Initiative Developed and Diversified the Canadian Economy

The AII-funded projects were very successful in achieving their objectives (average rating of 4.8/5) which included: to extend the runway length to accommodate international flights, to expand the airport to the point where it could accommodate major cargo traffic, and to allow the airports to continue operating year-round instead of only in the summer months.  Project proponent respondents said that the projects are likely to be very successful in the long run as well although that remains to be seen.

What is evident now is that the AII has provided airports with the potential to realize long term goals.  Specifically, the AII-funded airports now have the potential to increase their activity levels whereas the four comparator airports, upgraded during the same time period, still lack the potential to significantly increase activity levels.

Unintended Impacts

AII-funded respondents indicated that several positive impacts were generated as a result of these projects:  less noise traffic because of the longer runway, an increase in community spirit around such a large infrastructure project, and the introduction of Canadian Border Services Agency into the area because of new international flights.  One key informant noted that, in Kelowna and Kamloops, first nations Bands negotiated agreements with the airports that improved their relationship; the Bands publicly credited the Federal government for changing their local relationship with the airports.  No negative impacts were mentioned.

Performance: Demonstration of Efficiency & Economy

Economy

One key informant acknowledged that AII was delivered at very low cost – AII fit into the department's existing delivery capacity and the incremental cost to the department was one staff  for 1.5 years and 25%  of a manager's time for one year; outcomes and outputs likely could not have been generated at lower cost, making AII good value for the money spent.  Every AII-funded respondent said the AII is very cost-effective (100%; average rating of 5/5) and half of the respondents felt the program was well-structured (50%).  Some respondents had cash-flow issues that rendered pre-payment very difficult.  Respondents also said that although the departmental staff were careful to minimize delays, there were red tape issues that ended up resulting in several months of delays.  The first was a delay to an aircraft parking area in one of the projects that came about as a result of an environmental impact assessment.  Lengthy communications between the department and Environment Canada ended up delaying the construction by several months.  The second challenge was to convince one of the local communities that the project was worthwhile.  They did not originally see the airport expansion as profitable, but were convinced to support it eventually.  Respondents said that intended results would have been achieved more effectively if they had been provided more funding to complete the project; as a result the community had to invest much more of their own funding to make up for the gap. The department indicated that the first delay was caused by the proponent's alterations to their project and the second delay relates to future expansion plans and not to the project funded under AII.

Efficiency

The following is a discussion of the AII program in relation to the programs and best practices that have been reviewed.

The experience of one community with AII was complicated by a perceived funding shortfall.

One community was initially reluctant to support their AII project.  When they became convinced of its efficacy, AII did not provide all necessary funding and the community had to invest their own resources to make up the funding gap.  Some infrastructure programs do require joint funding.  However, the difference in the expectation and the reality of funding levels can reduce local support for the initiative and for a program in general.

The lack of advance funding can create issues for proponents.

Some participants in the program did not have sufficient cash flow to cover the up-front project costs and were forced to obtain bridge funding.  The Canadian Federal government airport infrastructure programs (and many other jurisdictions) focus on smaller remote airports and often provide 100% of program costs.  The previous BC Air Transport Assistance Program offered 100% funding for small airport projects while larger ones are expected to cost share.

Coordination between stakeholder agencies is essential to success.

Most often there are two or more levels of government (and potentially more than one department of one level) involved in the project.  Coordination is crucial not only for cost sharing issues but for areas of administrative and regulatory issues. Consultation and coordination with the involved communities is important for ensuring a smooth project process.

While there is some variation across airports, overall the department funded 29% of total project costs; every departmental dollar was matched by $2.45 from other contributors.  A leveraging ratio of 2.45 is respectable in comparison to the $1.65 leveraged by the Western Economic Partnership Agreements which funded the Cranbrook runway expansion.¹⁴ Internationally, the leveraging ratio of 2.45 compares favourably with the $2.88 leveraged from a large transportation infrastructure project funded in Dundee Scotland.¹⁵

Note: All numbers in millions CAD$
 

Note: All numbers in millions CAD$
 

Infrastructure projects like airports require sophisticated management systems.
These projects need to incorporate best practices in strategic planning, information management, asset management systems, consultation, priority setting and business case approaches.  They also include cost and benefit analysis of the project and account for social and environmental impacts.

Like CEDI, AII applicants had concerns regarding short time frames.

The program had at least the appearance being rushed into implementation.  Applicants said they would have liked to have more time for implementation.  Most infrastructure programs are of longer duration, usually multiple years.

Possible Improvements and Lessons Learned

• Project proponents learned they should know more about all the requirements before undertaking the project in order to plan more effectively.
• AII-funded respondents recommended that funding should be available in advance, explaining that project managers had to seek out bridge funding to fill in until the government repaid their costs.  Departmental management explained that all three airports were consulted and declined advances as outlined in the Due Diligence Report;  however, the Due Diligence Report for one of the airports lacked this information and bridge funding was very costly for this particular airport.
• One key informant listed a number of lessons learned/best practices including:  utilization of experienced departmental staff; availability of appropriate processes and systems to support delivery; sufficient project implementation time; timely due diligence and funding processes; collaboration with other funding partners; leadership with respect to environmental and first nations consultations; legal support; and strict stacking requirements and requirement of provincial government matching funding to ensure cooperation of all partners.

Recommendation

The evidence gathered in the evaluation suggests the following recommendation:

Recommendation #1: The department should develop a template for the design and delivery of economic diversification programs that is built on past lessons learned and best practices but is flexible in accommodating the unique needs of each issue, region and program.

Although CEDI/AII are one-time initiatives, there will continue to be community adjustment issues mandating federal intervention.  In this sense, programs like CEDI – one-time and time sensitive initiatives – are ongoing.  A template could:

• ensure projects are assessed for their long-term sustainability;
• target promotional activities and specific client groups;
• speed up design and approval processes, particularly under high volume conditions;
• ensure incorporation of appropriate monitoring practices based on the capacity of the clients;  and
• assist collaboration and coordination activities both within regions and with other funding programs.

A template would also benefit inexperienced staff:  a couple of key informants stated that staff in these short term programs are often inexperienced and there is high turnover because the jobs are offered as terms and therefore unattractive to the experienced staff that are needed to deliver such fast paced programs.