Introduction

This report presents the findings of an evaluation of the Community Futures program in Western Canada. This evaluation was conducted between November 2007 and August 2008 by Government Consulting Services (GCS) in collaboration with Western Economic Diversification Canada (WD) Audit, Evaluation and Disclosure Branch.

Background

The Community Futures (CF) Program was authorized in 1985 as part of the Canadian Jobs Strategy and the first community selection was announced in February 1986. The Program introduced a structure for the creation of, and support for, community-based development and/or adjustment initiatives in non-metropolitan areas of significant economic stress across Canada. The focus of the CF program has evolved over the years. The program transitioned from being housed in what is now called Human Resource and Social Development Canada (HRSDC), to being housed in WD. The program then took on integrated client service delivery focus, which then evolved into more of a social economic focus and currently has more of a business economic focus. ¹

The national CF Program is currently administered by four Regional Development Agencies (RDAs): the Atlantic Canada Opportunities Agency (ACOA), Canada Economic Development for Québec Regions (CED-Q), Western Economic Diversification Canada (WD), and the Federal Economic Development Initiative for Northern Ontario (FedNor) under Industry Canada (IC).

The national program's aim is to support local rural communities and small and medium-sized enterprises (SMEs) in meeting their economic needs, to help rural communities to develop and implement long-term community strategic plans leading to the sustainable development of their local economies, and to provide resources to local Community Futures organizations (CFs) to build community capacity to adapt to and manage change.

The purpose of the CF Program in Western Canada is to:

• support Community Economic Development (CED);
• diversify the economy;
• support the creation and expansion of SMEs;
• maintain and create new employment; and
• maintain the Minister's capacity to deliver business services to rural communities in Western Canada.

Financial support provided to the corporations under this Project is for:

• community economic development and strategic community planning;
• business advisory services including business planning, business information and entrepreneurship training to SMEs; and
• delivery of an investment fund.

The key activities, outputs and outcomes of the CF program can be found in the logic model (see Appendix A).

In Western Canada, the CF program is delivered through a network of 90 non-profit CF organizations (34 in British Columbia (BC), 27 in Alberta (AB), 13 in Saskatchewan (SK) and 16 in Manitoba (MB) that are supported by four CF associations (one per province) and a Pan-West CF Network.

According to information provided by WD, the CF program has received a total of $158M in operating costs over the last 6 fiscal years to support CF operations, the CF Associations and WD administration of the program. The breakdown per year can be found below.

Table 1: Operating Costs of the CF Program (000s)

Fiscal Year	CFsAssociations	Associations	WD Admin	Total
2002-2003	21,303	840	1,827	23,970
2003-2004	20,435	840	1,827	23,102
2004-2005	23,390	840	1,827	26,057
2005-2006	24,711	850	1,840	27,401
2006-2007	25,202	1,227	1,840	28,269
2007-2008	26,313	1,244	1,840	29,397

top of page

Evaluation Objectives

An evaluation of the CF Program is required as part of the Department's responsibilities under the Treasury Board of Canada Evaluation Policy and theFederal Accountability Act. This evaluation was conducted across Canada in the four RDAs (ACOA, CED-Q, WD, and FedNor). The evaluation was conducted to ensure that the end product meets Treasury Board Secretariat standards for evaluations, as outlined in the current evaluation policy and the guide for the review of evaluation reports.

The outcomes and impacts of WD's Community Futures program were considered as part of this evaluation. The remaining RDAs are covered in separate evaluations. A pan-Canadian evaluation rollup report will be written once all of the individual RDAs have completed their evaluations.

This evaluation covers the period of fiscal year 2002/03 to 2007/08. Case studies, however, covered the period of fiscal year 1999/00 to 2007/08, in order to better assess long-term outcomes and trends. The scope of the evaluation is to cover issues relating to CF Program relevance, design and delivery, program impact (short-term, medium-term and long-term), and cost-effectiveness/alternatives, as described in the CF program Result-based Management and Accountability Framework (RMAF).

Approach

The work undertaken by GCS was managed by the Evaluation Unit under the Audit, Evaluation and Disclosure Branch, WD. The work was overseen by an Advisory Committee, comprised of representatives from WD headquarters (program office and evaluation office), each of the four WD regions and each of the four CF associations.

A framework was developed for a pan-Canadian evaluation of the CF program in the fall of 2007. The pan-Canadian matrix outlined a set of common questions and indicators that all RDAs were to use as the basis for each of their evaluations. Since WD conducted its evaluation in tandem with the evaluations conducted in the other regions of the country, it was imperative that WD's evaluation utilizes the same approach and methodology as the other evaluations to ensure validity and reliability of the pan-Canadian evaluation rollup report. WD supplemented the national questions with additional questions that addressed WD specific issues. The evaluation issues and questions are described in Table 2 below.

Table 2: Summary of Evaluation Issues and Questions

Evaluation Issue	Evaluation Question
Relevance	Is there a continued need for the Community Futures program? Does the Program complement, duplicate or overlap other government programs?  Other private sector services? Are local CF objectives and activities consistent with the national CF Program?  Are CF program objectives consistent with departmental objectives?
Design and Delivery	Are the CF networks (national, provincial, regional, and sub-regional) networks working effectively? What factors impact or facilitate the achievement of program results? Are the CF investment funds well managed?  Are the number, level and loss rates of the loans meeting the needs? Are the departments and CFs gathering the necessary data for evaluation and measurement purposes? What is the level of awareness of the CF? What activities have been undertaken to increase awareness? What strategies have been undertaken to recruit competent employees and board members?
Program Impact	Short Term To what extent has the CF Program provided appropriate information, referrals and counselling to clients? To what extent has the CF Program improved business knowledge and skills of clients? To what extent has the CF Program created new business start-ups or strengthened existing businesses? To what extent is the CF Program serving the needs of Official Languages Minority Communities (OLMCs)? Intermediate Have CFs been involved in developing community strategic plans?  To what extent are CF activities linked to community plans? To what extent has the CF Program: supported community economic development; assisted communities to develop and diversify their economies; and strengthened community capacity? Long-Term To what extent has the CF Program contributed to long term goals - i.e., economic growth and stability, diversification and development of local rural communities, sustainable communities, and survival of business assisted by CFs. Has the CF program produced unintended positive and/or negative outcomes?
Cost-effectiveness and Alternatives	To what extent is the CF Program cost-effective? Are there other more cost-effective/ efficient approaches or alternatives to be considered that would achieve CF Program objectives?

top of page

Methodology

The evaluation was guided by the national evaluation matrix and was conducted in two phases: 1) methodological planning and 2) conduct of the evaluation. The methodological planning phase commenced in November 2007 and the Methodology Report, accompanied by a WD-specific evaluation matrix and data collection tools, was completed in February 2008.  For a copy of the evaluation matrix see Appendix B

The evaluation methodology integrates the use of multiple lines of evidence and complementary quantitative and qualitative research methods as a means to ensure the reliability of results being reported and validity of information and data collected. The research methods are described in detail below.

Document and Literature Review

The review of documents was used primarily to assess program relevance, program design and delivery and program impact. 

Three main types of documents were assessed and analyzed during the evaluation:

• General Background documentation (e.g. TB Submission, Business Case, documents that describe the program's history, rationale, program theory, etc.)
• Program & Policy Documentation (e.g., Reports on Plans Priorities, Performance Reports, WD website, Terms of Reference, information on relevant committees, meeting minutes, guidelines, operational documents, manuals, etc.)
• Evaluations & Past Studies (e.g. previous evaluations, reports, surveys, research, etc.)

The document review was conducted using a customized template that facilitated extracting relevant information from the documents and organizing to indicators and evaluation questions. For a full list of documents reviewed see Appendix C.

Database Review and Analysis

Administrative data is stored in two WD databases.  Quantitative data for fiscal year (FY) 2002/03 to FY 2005/06 inclusively is contained in a Lotus Notes database.  A new web-based measurement tool was implemented for FY 2006/07.  WD provided a cross walk table that showed which variables remained the same from the old database to the new database. Where possible, trends were analyzed over a 5-year period (2002/03 to 2006/07). Templates were developed to facilitate the gathering and analysis of the information according to the performance indicators and evaluation questions identified in the evaluation plan. 

top of page

Interviews

Interviews served as an important source of information for the evaluation by providing qualitative input on relevance, design and delivery, program impact and cost-effectiveness/alternatives of the CF Program. A total of 50 interviews were completed during the course of the evaluation (see Table 3 below).  Interviewees included:

• Senior Managers (RDAs) - a WD senior manager at HQ and in each of the four provinces
• CF Program Managers - a WD program manager at HQ and one in each of the four provinces
• CF Sample - a representative from each of the 4 provincial associations and a randomly selected sample of 15 CFs (5 in BC, 4 in AB, 3 in SK and 3 in MB). The number of CFs to be included in each province was chosen to be proportionate to the distribution of CFs per province.
• External Stakeholders and Experts - individuals in the community who are knowledgeable about CF activities but who do not have a direct link to a CF. The CF provincial association representatives each provided a list of potential contacts with a rational for including them in the interviews. At least 2 stakeholders/experts per region were interviewed.

Table 3: List of Interview Groups

Interview Group	Number of Interviews Conducted
Senior Managers	5
CF Program Managers	5
CF Sample CF Chairpersons or Directors CF Managers Provincial Association Representatives	12 15 4
External Stakeholders/Experts	9
Total	50

All interviews were conducted via telephone.  The interviews lasted between one to two hours. All interviewees were contacted in advance to schedule an appropriate interview time.  An interview guide was provided to all persons in advance of the interview (for a list of interviewees see Appendix D, for the interview guides see Appendix E).

Surveys

Web-based surveys were administered to two groups: CFs and CF clients.  The results of these surveys provided input into the relevance and success of the Program. 

CF Survey - All 90 CFs were invited to participate in a web-based survey.  The survey was directed to the Managers of the CFs and the Chairs of their Boards of Directors. A total of 179 emails were sent and no bounce backs were received. The survey had a total of 118 respondents providing a 95% confidence interval of +/- 5.3%.  The CF survey had representation from all provinces (for a breakdown of respondents see Table 4 and Table 5 below, the survey questions can be found in Appendix F).

top of page

Client Survey - Clients that have used the services of a CF in the past five calendar years (i.e., January 2003 - December 2007) were invited to participate in the web-based survey.  Due to privacy restrictions, each of the CFs sent an email to their clients requesting their participation and CFs were also responsible for sending 2 follow-up reminder emails to their clients. The survey questions can be found in Appendix F.

Three categories of clients were included in the survey:

• Investment clients: include those who applied to a CF for financial help (this includes both recipients and non-recipients of loans)
• Business services clients: include those who accessed the information services of a CF (e.g., counselling, advice, training)
• Community Economic Development clients: include those partners and organizations with which CFs may engage for the development and promotion of the CF service area, including the development of community strategic plans.

A total of 1,114 individuals responded to the survey. The CFs tracked the number of surveys that were sent out and the number of surveys that did not make it to intended recipients (email bounce backs). In total the survey was sent to 9,963 (5980 in BC, 2274 in AB, 866 in MB and 843 in SK) potential respondents. The 1,114 responses results in a 99% confidence interval of +/- 3.7. The client survey had a high proportion of respondents from BC. Therefore the survey's results were weighted to ensure that the proportion of respondents were reflective of the proportion of CFs in each region (BC 38%, AB 30%, SK 14%, MB 18%).

Clients were asked to identify which group they fell into. They were allowed to choose more than one answer. Table 6 (below) outlines the number of respondents who chose each group.

The respondents had businesses in a variety of sectors and their current annual gross revenues ranged from less than $100K to more than $10M. The highest proportion of clients was in the less than $100K category (48%) followed by the $100K to $499K category (30%).

Case Studies

The case studies were completed by the evaluation group at WD and the findings were incorporated throughout the evaluation report.

While not statistically representative of the entire program, case studies in this context are intended to provide a context and vivid illustration of how the program is achieving or failing to achieve the intended outcomes and identify any best practices or lessons learned in order to enhance program effectiveness and cost efficiency.

WD Evaluation group worked with the evaluation advisory committee to first determine a total of 22 CFs, from which WD selected 10 critical case studies. The 10 case studies (i.e. 10 CFs) were selected to obtain a range of geographic locations, population size, success or lack of, and to some extent service focus (business vs. community economic development and planning). WD Evaluation group visited all 10 CFs.

Case studies involved a review of the relevant case-related documents provided by the CF and WD staff; administrative data including electronic WD database; interviews with the CF manager, the CF business analyst, the WD officer working directly with the CF (interview guides can be found in Appendix G); Statistics Canada socio-economic data specific to each of the selected CF generated through census data from 1996, 2001, and 2006; and one focus group in each CF (please refer to focus group guide in Appendix G) . In total, 89 people participated at the 10 focus groups, an average of 9 participants per focus group. Participants represented CF clients, community partners and stakeholders, municipality representatives, and some CF board chairs.

Information on the selected CFs is summarized in Table 7 below.

Table 7: CFs Included in the Case Studies

Community Futures Development Organization	Province	City	Size of population*	Location	Date established
Crowsnest Pass	AB	Blairmore	5665	South West	1986
East Parkland	AB	Mirror	103760	Central	1989
Grande Prairie & Region	AB	Grande Prairie	72605	Northwest	1988
Boundary Area	BC	Grand Forks	11995	South West Kootenay	1992
Cariboo - Chilcotin	BC	Williams Lake	41355	North East- Remote	1995
Okanagan - Similkameen	BC	Penticton	78480	South	1984
North Central	MB	Thompson	30230	Northern Remote	1997
North Red	MB	Selkirk	29405	South East	1990
Beaver River	SK	Meadow Lake	24970	North West-Remote	1989
Sagehill	SK	Bruno	52635	East Central	1987

* Source: Statistics Canada: 2006 census data

top of page

Detailed case study write-ups are included in Appendix H. These write-ups were forwarded to the selected CF for verification of fact and accuracy.

Labour Market Data

Labour market data was available through WD, who engaged Statistics Canada to produce customized data profiles for each of the CF regions.  This data, compiled from the 1996, 2001, and 2006 censuses, provided key information required for the evaluation, including; labour force participation rates, unemployment rates, average household income, and employment by industrial sector.  This information was used primarily in the impact section.

Strengths and Limitations of the Methodology

• Data Analysis: Administrative data covered a five-year period (2002/03 to 2006/07). A new database to track ongoing performance information was implemented by WD in 2006/07. Discrepancies between the types and/or accuracy of data provided through the various data sources were identified. WD worked closely with GCS to determine the accuracy of the information and outline any limitations of the administrative data. Limitations are noted throughout the evaluation report, where appropriate.
  
  
• Client Survey: The evaluation advisory committee discussed at length, which mechanism would be used to conduct the client survey. A number of barriers were identified including privacy restrictions, access to email (particularly in remote areas), cost and time constraints. In the end, it was determined that a web-based survey was feasible and the CFs could distribute the survey to their clients thereby avoiding any privacy concerns. CFs were also asked to post a memo in their offices alerting clients that the survey was underway and providing them with a link to the survey. In some instances, clients who did not have access or difficulty accessing the internet, completed the survey at the CF's office. If the response to the survey was not deemed adequate and did not have responses from a variety of communities (e.g. northern, remote, large, small etc.) then mail based surveys would have been sent to underrepresented communities. The web survey did prove to have a high response rate and included respondents from a variety of communities. CF associations were provided a list of communities that responded to the survey. The associations felt comfortable that a wide breadth of communities were represented in the survey. It was, therefore, determined that a targeted mail based survey was not required. 
  
  
• Case Studies: The case studies enable an in-depth analysis that would not be possible with more general approaches. Case studies are typically expensive and time consuming to carry out. As explained above it is not possible to analyse a statistically reliable sample and, consequently, it is not expected to make statistical generalization of the results. Case study should be looked at as only one line of evidence to complement the other lines of evidence. The list of participants at the focus group was provided by the CFs. It is likely that this process will introduce a selection bias that may influence the responses obtained. To partially address this limitation, we urged the CFs to include some participants who do not have a direct vested interest in the CF program.
  
  
• Finally, estimating the net impacts of the CF program ideally requires that comparisons be made between samples of recipients and non-recipients from similar populations to ensure that impacts can be attributed to the intervention (CF program) and not to differences between the groups. Identifying a comparison group proved to be very difficult. This challenge, coupled with the fact that the comparison group was not significantly appropriate, and a potentially lengthy and costly process for contracting a firm to run the survey, led us to eliminate the "non-client" survey as line of evidence. To address this weakness, we have included in our client survey sample those who have applied for and received loans from CFs as well as those who have applied for and did not receive loans (non-recipients). Furthermore, we have included in the interviews experts who did not have any stake in the CF program. As well, the focus groups conducted as part of the case studies, included people who did not benefit from the CF program.

---