Western Economic Diversification Canada
Symbol of the Government of Canada

Common menu bar links

Evaluation and Methodology

This section describes the design, methodology and limitations of the evaluation.

Evaluation Logic Model and Framework

A formal logic model was developed for the Community Futures Program (CFP) as part of the national Performance Measurement Strategy (PMS). The CFP logic model is a diagram that depicts the inputs, activities, and outcomes of the CFP and the relationship between them. The model also makes the distinction between the activities and outputs of the government departments that administer the program, and those of the CFs.

Text Version, Figure 1: Logic Model – Community Futures Program

The figure shows the Community Futures Program logic model as a diagram that depicts the inputs, activities, and outcomes of the program and the relationship between them.

The evaluation framework for this study combines the national CFP evaluation framework developed as part of the national PMS and feedback received during the evaluation planning process to reflect coverage of five core issues as per the Treasury Board Secretariat Directive on Evaluation.3

Evaluation Approach and Methodology

A literature review of approaches to evaluating community development programming4 suggests that an evaluation of the CFP requires a mixed-methods research design involving multiple lines of evidence gathered through both quantitative and qualitative methods. Quantitative methods such as data analysis and surveys can be relied upon when describing activities, outputs and short-term impacts. Qualitative methods such as interviews and document review play a critical role in studying intermediate and longer term outcomes. A mixed methods approach allows for triangulation (i.e. convergence of results across lines of evidence) and complementarity (i.e., developing better understanding by exploring different aspects of an evaluation issue).

CFP Evaluation Advisory Committee

An advisory committee was established to provide the evaluation team with input, advice and suggestions regarding the conduct of the study, resolve ambiguous or contentious issues and ensure that the Evaluation Team produces a balanced, objective and credible report. The members were drawn from the department and external sources (CFs and CF Associations). A Terms of Reference guided the operation of the advisory committee.

Preliminary Consultations

Preliminary consultations were conducted with WD corporate and regional staff to develop the Terms of Reference for the evaluation. The terms of reference, including the evaluation framework, was reviewed by the advisory committee, the department’s Directors General of Operations Committee and Executive Committee. The Terms of Reference was finalised after these consultations.

Document and Literature Review

A review was conducted of the department’s documents as well as literature relevant to the CFP. The review of documents was used to assess program relevance and performance. Three main types of documents were assessed and analyzed during the evaluation:

  • General background documentation (e.g. WD documents that describe the program’s history, rationale and program theory, etc.).
  • Industry Canada reports on SMEs in Canada.
  • Program and policy documentation (e.g., Terms and Conditions of the CFP, CF performance reports, CF’s websites, information on CF revitalization process, CF operational documents and manuals, the national PMS, etc.).
  • Past evaluations of the CFP and literature related to community development programming.

Performance Data

The evaluation project team undertook a detailed analysis of project data from 2008–09 to 2012–13 to create a profile of activities implemented through the program. The performance data provided a comprehensive perspective of the outputs and activities of the program (such as statistics on program activities, loans, business support services, etc.). This data is reported by the CFs into a database hosted by CommonGoals5 on a quarterly and/or annual basis. Project data was supplemented by data from the department’s financial and human resource systems where possible.

Statistics Canada Business Number Review

As per a contract established with Statistics Canada, the department was provided with client data on employment, sales and survival rates to be used to report on CFP indicators such as business entry and exit, employment growth and business survival rates. Statistics Canada completed four data runs which were used as the basis for establishing performance targets by the department. In order to facilitate this process, the department provided Statistics Canada with a list of business numbers of their CF clients. Data was received from Statistics Canada at an aggregate level and the department analysed and reported on the data for the evaluation. Comparable6 firms to CF-assisted firms were identified based on postal codes or census division in areas serviced by CFs. The evaluation relied on results from the 2000 to 2010 run of data.

Census Data

Statistics Canada produced customized data profiles for each of the CF service areas. This data, compiled from the 2006 and 2011 census, provided key information required for the evaluation such as population in each CF service area, labour force participation rates, unemployment rates, average household income and employment by industrial sector. The following table shows the distribution of the population served by the CFP in the various regions of Western Canada.

Province Number of CFs Population Served
Manitoba 16 530,670
Saskatchewan 13 616,725
Alberta 27 1,588,310
British Columbia 34 1,726,740
Total 90 4,993,115

Key Informant Interviews

Semi-structured interviews were completed with 60 internal and external stakeholders. Key groups are identified in the table below:

Stakeholder Type Planned Completed
Internal WD management and staff 10 11
Representatives from CF network organizations 10 5
Account managers of financial institutions 10 9
Organizations involved in similar activities as the CFP 10 9
Community groups/leaders 10 14
External stakeholders/subject matter experts 10 12
Total 60 60

Case Studies

Case studies are an effective evaluation method to demonstrate program successes and challenges. While not representative of the entire program, selected case studies are useful for providing examples and demonstrating the effectiveness of the program. Case studies may also be used to illustrate the complexities surrounding the CFP and the contextual elements in which it operates.

A total of 8 case studies (2 per region) were selected for the evaluation based on size, service focus (business vs. community economic development and planning), geographic location and consultation with the CFP Evaluation Advisory Committee. The case study methodology included site visits to each of the 8 CFs to review documents and interview the CF Manager, a Board Member and at least 3 CF clients. The list of CFs selected for the case studies are below.

Selected CF Province City Size of Population*
Central Okanagan CF BC Kamloops 179,840
Central Interior First Nations CF BC Kamloops 9,325
CF Lethbridge AB Lethbridge 115, 605
CF Peace Country AB Peace River 45,140
South Central CF SK Moose Jaw 54,540
Newsask CF SK Tisdale 45,315
CF Greenstone MB Flin Flon 7,330
Cedar Lake CF MB The Pas 15,965

Return to the top of this pagetop of page

Surveys

Survey of CF Organizations

To gain insights on the relevance and performance of the CFP, a survey was conducted of the CFs not included in the case studies. For each CF, two survey instruments were circulated: one for the CF Managers and one by the CF Board Chairpersons. A total of 65 CF Managers (79 percent) and 40 CF Chairs (49 percent) completed the survey (response rate of 64 percent). The estimated response rates by province and by respondent type are as below.

CF Manager and Chair Survey Response Rate by Province
  Alberta British Columbia Manitoba Saskatchewan Total
Valid contacts 51 63 28 22 164
Non-response rate 33% 33% 54% 27% 36%
Response rate 67% 67% 46% 73% 64%

 

CF Manager and Chair Survey Estimated Response Rate by Respondent Type
  CF Manager CF Chair Total
Valid contacts 82 82 164
Non-response rate 21% 51% 36%
Response rate 79% 49% 64%

CF Client Survey

A survey was conducted with CF clients (including non-recipient loan clients) to collect information on the activities and outcomes of the CFP. Clients were surveyed for insights on program delivery and outcomes. Surveys were expected to inform relevance and performance of the CFP.

In building the sampling frame, CFs were asked to provide the names, positions, telephone numbers and email addresses of all of their clients. Information related to loan amounts, types of services received, time frame for receipt of services and supports for clients was collected through the questionnaire.

The response rates and actual completions were monitored closely after survey implementation to ensure that the survey was being implemented appropriately. The survey was sent by email to 5,743 clients. Emails of 1,195 clients were invalid. A total of 1024 clients started the survey and 893 clients completed it for a response rate of 20%. The estimated response rates by province and by respondent type are as below.

Client Survey Estimated Response Rate by Province
  Alberta British Columbia Manitoba Saskatchewan Total
Valid contacts 1,032 2,999 294 223 4,548
Non-response rate 78% (806) 83% (2494) 70% (207) 66% (148) 80% (3655)
Response rate 22% (226) 17% (505 ) 30% (87) 34% (75) 20% (893)

 

Client Survey Estimated Response Rate by Respondent Type
  Loan Clients Non-loan Clients Total
Valid contacts 2,148 2,400 4,548
Non-response rate 71% (1,528) 89% (2,127) 80% (3,655)
Response rate 29% (620) 11% (273) 20% (893)

CF Client Profile: The majority of CF clients (83 percent) indicated that their organizations or businesses were currently operating. About 6 percent indicated that their organizations or businesses were still in the planning stage but not yet operating. About 5 percent indicated that their organization or business had permanently stopped operations. About 7 percent of the CF Clients indicated that their organizations or businesses had temporarily stopped operations, been sold or will not proceed. About 12.4 percent of the CF clients were in the retail trade and 7 percent were in the accommodation and food services sector. The remaining was split among various other sectors as shown below.

Text Version, Figure 2: Business Sector

This figure shows the business sector for Community Futures clients surveyed for the evaluation.

Most clients (68.8 percent) received loans from the CF and assistance in the development of a business plan (36.6 percent).

CF Partner Survey

CFs provided administrative data on names, position, telephone numbers and email addresses of partner organizations. Due to time and resource constraints, most CFs elected to provide a sample of 10 – 25 Partners. In the event a region was over-represented, the survey data was weighted accordingly.

The survey was sent by email to 1,329 partners based on the contact information provided by CFs. Emails sent to 106 of the addresses were invalid. A total of 301 clients completed the survey for a response rate of 24.6 percent. The estimated response rates by province are below.

  Alberta British Columbia Manitoba Saskatchewan Total
Valid contacts 279 727 151 66 1223
Non-response rate 67% (186) 78% (565) 85% (129) 64% (42) 75% (922)
Response rate 33% (93) 22% (162) 15% (22) 36% (24) 25% (301)

Evaluation Challenges and Limitations

  • Potential for respondent bias. The evaluation findings are based, in part, on the views of key informants and survey respondents with a vested interest in the CFP program and potentially biased in their responses regarding outcomes. Several measures were taken to reduce the effect of respondent biases and validate interview results, including the following: (i) ensuring that respondents understood the strict confidentiality of responses; (ii) including key informants and survey respondents who do not have any vested interest in the CFP or were unsuccessful in their loan applications to the CF; and (iii) asking respondents to provide a rationale for their ratings, including a description of specific activities which contributed to reported outcomes.
     
  • Surveys: The ability of the evaluation to conduct a survey was impacted by a) the completeness and quality of contact information provided by CFs; and b) the willingness of respondents to participate in the surveys. In the event a region was over-represented, the survey data was weighted accordingly. If a group was under-represented, targeted follow-up calls and/or emails were made to enhance the representativeness of the survey responses.
     
  • Case Studies: Case studies are typically expensive and time consuming to carry out. As explained above, it is not possible to analyze a statistically reliable sample and consequently, case studies are not expected to make statistical generalization of the results. To partially address this limitation, the results from the case studies are used to support findings in other lines of evaluation evidence. In addition, documents reviewed during the case studies included those obtained directly from the CF. Because this may include those documents the CFs believe are important for the evaluation, there is a potential for a selection bias. Information was therefore analyzed together with internal departmental information to reduce this bias.
     
  • Attribution/Contribution: As with most of WD programming, other factors will contribute to the expected results/outcomes of CFP. This challenge is particularly relevant for community based programming, given that multiple factors (i.e., multiple funding partners, influence of volunteer boards and other initiatives delivered by the CFs on behalf of other organizations) can influence outcomes while trying to isolate WD’s specific influence on these outcomes. The evaluation team addressed this challenge of attribution by using Statistics Canada data to compare CF-assisted firms versus non-assisted firms.
     
  • Data availability for assessing efficiency/economy: Accurate data on the operational costs of administering programs at WD is often limited or not available which prohibits an extensive analysis of program efficiency and economy. Alternative ways of assessing efficiency and economy was undertaken, such as seeking out opportunities for more optimal use of resources in program governance and delivery through CFP performance data and key-informant interviews.

 


[3] Treasury Board Secretariat. Directive on Evaluation, 2009 http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681

[4] Community Development Evaluation Research, Published by Community Waitakere, New Zealand, August 2012 http://www.communitywaitakere.org.nz/images/research1.pdf.

[5] CommonGoals is a company that developed The Exceptional Assistant (TEA) software used by the CFs to report on performance to the department.

[6] "Comparable firms"? those that are of similar age, size, sector and geographic region, but that have not received a direct monetary contribution from CFs.