Western Economic Diversification Canada
Symbol of the Government of Canada

Common menu bar links

Methodology

Scope of Work

The logic model underlying this theory-based evaluation was developed in consultation with the advisory group. The evaluation used a non-experimental design because a control group was unavailable; however, when used with multiple lines of evidence, the design met the objectives of the evaluation. This evaluation of the department’s investments in the EDP included a wide range of data collection methods, each of which is described below. The scope of the evaluation included the following evaluation issues: relevance and performance (efficiency, effectiveness and economy). The objectives of the evaluation and the core evaluation issues are presented in table 3-1.

Table 3-1  Core Evaluation Issues
  Evaluation Issues
Relevance 1. Is there a continued need for the EDP?
2. Is there a continuing demand for the EDP?
3. Is the program aligned to WD & federal government priorities?
4. Is the program consistent with federal roles and responsibilities?
Performance (Achievement of Outcomes, and Demonstration of Efficiency and Economy) 5. To what extent is the EDP effective in meeting its objectives?
6. What factors facilitate or impede the achievement of program results?
7. To what extent has the EDP realised its intended outputs?
8. To what extent have intended outcomes been achieved as a result of the program?
9. To what extent did the EDP contribute to its long-term outcomes?
10. To what extent did the operating funds impact on the delivering of EDP services (loans, advice etc) in rural and urban sectors?
11. Is the program design (rural and urban) appropriate for achieving expected program results?
12. Have there been any unintended (positive or negative) outcomes?
13. Is the program undertaking activities and delivering products in the most efficient manner?
      • How could the efficiency of the program’s activities be improved?
      • Are there alternative, more efficient, ways of delivering the program?
14. Is the program achieving its intended outcomes in the most economical manner?

 

Evaluation Study Activities

Preliminary Consultations and Evaluability Assessment

Preliminary consultations were conducted with departmental officers responsible for the agreements in the development of the evaluation framework and also to discuss data availability. Through these consultations, the scope of the evaluation was established and final terms of reference developed for the evaluation. An Advisory Committee made up of departmental representatives from each region was set up to provide support to the evaluation in generating useful and relevant information by contributing to improve the effectiveness of the evaluation, and ensuring quality assurance of evaluation products. Based on the evaluability assessment, a consultant was engaged to collect some of the data for the evaluation. The final report was written using internal evaluation resources.

Document and Literature Review

The evaluation included the review of documents and literature containing information relevant to the EDP. The review was used primarily to assess relevance and performance of the agreements.

The main types of documents assessed and analyzed during the evaluation were:

  • General background documentation (e.g., program agreement documents, websites, media reports, EDP Program documentation including funded project descriptions, program guidelines, status reports, performance reports, records held by rural and urban partners of the EDP etc.); and
  • Specific documentation (e.g., departmental website, Terms of Reference, previous evaluation reports and other studies related to the EDP).

Information from the review was used in developing the logic model for the evaluation and to complete the Terms of Reference for the evaluation.

Project File Review and Data Analysis

The evaluation team reviewed the project files related to the EDP and conducted an analysis of project data available through the department’s project management database, financial system, and the TEA4 database used by the Community Futures Organizations. Because Saskatchewan’s information was missing from the department’s project management database, hard file reviews were completed during fieldwork in Saskatchewan. Results from previous evaluations were used as baseline data for this study. The evaluation team also consulted with departmental EDP Officers and stakeholders from the delivery agents to validate data collected through the file and database review. Data was collected for all 7 urban delivery agents, 4 provincial associations, and 90 Community Futures Organizations.

Return to the top of this pagetop of page

Key Informant Interviews

The department’s EDP program managers took part in a focus group held in Edmonton (in-person and by video conference), and the other respondent groups provided input by way of a telephone or in-person interview. Groups represented included: Provincial Community Futures Associations, Urban Delivery Agents, Community Futures Organizations and community stakeholders.

In all, 51 individuals representing 38 organizations provided their input on the key research questions. Question guides were distributed to participants prior to the interview/focus group. Interviews lasted on average 1.25 hours and the focus group lasted two hours. One organization supplemented their interview with a written response that represented the input of their Board, approximately eight additional individuals (not counted in the total numbers below; however, the written response was included in the data roll-up).

Many of the interviewees had a long history of involvement with the EDP, and were able to provide a historical perspective. For most interviewees the EDP is a part of their overall job responsibility and duties. This analysis was completed by the consultant.

Table 3-2   Stakeholders Providing Input
Stakeholder group No. of organizations represented No. of individuals participating
Western Diversification, Program Managers, (Focus Group) 4 8
Community Futures Associations (Interviews) 4 7
Urban Service Delivery Organizations (Interviews) 7 13
Community Futures Organizations (Interviews) 11 11
Stakeholders (Interviews) 12 12
Total 38 51

Sampling approach for selecting key informant participants:

  • Departmental Program Managers: Representatives from each province were selected based on their familiarity with and their role in overseeing the EDP.
  • Community Futures Associations: The three associations from each province were chosen to represent different types of communities and levels of engagement in the Entrepreneurs with Disabilities Program.
  • All Urban Delivery Agents participated.
  • Stakeholders: A representative sample of community stakeholders was chosen based on geographic spread and their knowledge of services to entrepreneurs and services to people with disabilities.

Comparative Analysis

The comparative analysis included performing a comparative analysis of the EDP services to other programs, organisations, departments and services available to entrepreneurs with disabilities to access capital in all four western Canadian provinces, and the federal government.

A survey of ten related organizations/programs/services to the EDP was conducted, in addition to a scan of web-based resources available to entrepreneurs. The surveyed organizations serve people with disabilities, providing a range of services including support for self-employment. Information was collected based on website content, articles available, and in some cases, direct consultation with a member of the organization. This analysis was completed by the consultant. The organizations surveyed included:

  • Opportunities Fund for People with Disabilities: A Canada-wide program offered by the department of Human Resources and Skills Development, established in 1997 to assist people with disabilities in preparing for, obtaining and keeping employment or becoming self-employed, and increasing their economic participation and independence.
  • Canadian Society for Social Development: An independent Canada-wide organization, providing employment and skills training to persons facing challenges to employment: persons living with a disability, youth, stay at home parents, older workers, Aboriginal Peoples, individuals living in rural areas.
  • Canadian Council on Rehabilitation and Work: A Canada-wide organization with a mandate to promote and support meaningful and equitable employment for people with disabilities.
  • SEED Winnipeg: A Winnipeg-based program established in the late 1980s to fight poverty and help renew Winnipeg's inner city by helping entrepreneurs start small businesses and save money for future goals.
  • Employability Assistance for People with Disabilities: A Saskatchewan based program, providing funding to assist adults with disabilities to prepare for, secure and maintain employment.
  • EmployAbilities: A community-based, non-profit organization operating in Alberta to provide employment programming and learning opportunities for people with disabilities
  • Disabilities Related Employment Supports: An Alberta based program providing education; workplace and job search to support people with disabilities.
  • Vancouver Foundation Disability Support for Employment Fund: A Vancouver based program offering grants to assist eligible organizations throughout British Columbia who are working to increase employment opportunities and ultimately the rate of employment for people with disabilities in their communities.
  • Employment Program for People with disabilities (EDP): A program through British Columbia’s Ministry of Housing and Social Development, providing a range of specialized services to help individuals with disabilities participate in their communities, pursue their employment goals as they are able, increase their self-reliance, and build skills and experience that may lead to further employment or volunteer opportunities.
  • BC Coalition of People with Disabilities: A BC-based organization, championing issues impacting the lives of people with disabilities through direct services, community partnerships, advocacy, research, and publications.
    The analysis also included the services offered by the department’s Community Futures Program to entrepreneurs through the Community Futures Organizations. The organizations were compared with the EDP based on factors including:
    • Regions served;
    • Loan capacity;
    • Services provided - business planning, training, research assistance, business advisory, referral, networking and mentoring;
    • Overall service delivery model - including rural vs. urban services;
    • Target clients; and
    • Performance.

Return to the top of this pagetop of page

Client Survey

As part of the evaluation, a survey was conducted of EDP clients who have accessed the program in the past 5 years to share their experiences. When all EDP client contacts were accounted for, the total sample was 544 clients. In order to compile an appropriate sample, each service delivery agent (both urban and rural) was asked to provide their client lists.

The delivery agents provided a total of 354 individual EDP contacts. Of the total provided, only 270 were found to be usable. Some 84 contacts could not be used due to out of service phone numbers, bounced e-mails and outdated information. In total, 86 successful contacts were made, 27 of which were done via email and 54 of which were done via telephone. An additional 5 survey responses were received manually following the completion of the analysis. This is a 32% response rate for the usable sample of 270 client contacts provided for the evaluation.

The following table illustrates the actual distribution of clients who completed the survey across each province and organization and the response rate:

Table 3-3  EDP Client Survey Respondents by Province – Actual*
BC # of People Invited to Participate in Survey of Respondents Response Rate
Urban 17 5 29.41%
Rural 80 25 31.25%
Subtotal 97 30 30.93%
AB      
Urban 43 10 23.26%
Rural 109 10 9.17%
Subtotal 152 20 13.16%
SK      
Urban 141 23 16.31%
Rural 14 2 14.29%
Subtotal 155 25 16.13%
MB      
Urban ** 126 0 0.00%
Rural 14 11 78.57%
Subtotal 140 11 7.86%

Total

544 86 15.81%

*Response rate is calculated by dividing # of Respondents by # of People Invited to Participate in Survey.
**For confidentiality reasons, the Urban agent did not provide a client list to the evaluators; instead, the Urban agent sent letters to their clients requesting their participation. None of the clients participated.

The total distribution of urban vs. rural response to the survey is shown below.

Table 3-4   Urban vs. Rural
Provider EDP Clients Percent of Total Volume
Urban 38 44.2%
Rural 48 55.8%
Total 86 100.0%

Return to the top of this pagetop of page

Focus Group

Validation focus groups for the EDP evaluation were held in Vancouver, Edmonton, Saskatoon and Winnipeg. In all, 37 individuals (Table 3 – 5) participated in the focus groups either in person, by videoconference, or teleconference. The consultant presented an overview of findings, and discussion questions to stimulate input from participants. The sampling approach outlined for selecting the key informants was also applied to selecting the focus group participants.

Table 3-5   Numbers of attendees by category
Region Community Futures Organizations Urban Delivery Agents Other Stakeholders WD Staff Provincial Community Future Associations Total
Manitoba 2 1 2 2 1 8
Saskatchewan 1 6 1 2 - 10
Alberta 2 5   3 2 12
British Columbia 1 1 2 2 1 7
TOTAL 6 13 5 9 4 37

Limitations of the Evaluation

  • Data Analysis: The evaluation used data reported by the Community Futures Organizations in the TEA database, and data reported by the Urban Delivery Agents and the Provincial Community Futures Associations in the department’s project management database and files. The interviews and focus groups indicated that data reporting is not consistent from region to region. In addition, it was unclear if the delivery agents had a consistent definition regarding how they define and track clients and results. Data generated from each of the databases was validated with the provincial associations and the urban delivery agents. In cases where the data could not be reconciled through the validation process, it was not used in the analysis.
  • Survey: Selection bias in the survey was a potential threat to the validity of findings and the ability to apply findings to all entrepreneurs with disabilities. Clients invited to participate in the survey likely directly participated in the EDP program and who may have experienced greater success with their business. Those who failed may not have responded or may be under-represented in the findings. In addition, the survey sample was not representative of the total population. Results of the survey were therefore used to support other lines of evidence in the analysis.
  • Comparative Analysis: Since the EDP is a small component of a larger suite of programs and services offered by the delivery agents, it was difficult to isolate the EDP’s effect on a program participant. The use of an experimental design to assess incremental impact was not feasible for the EDP. Matched groups could not be identified and tracked to be compared to program participants. EDP delivery agencies did not record client descriptor information that would allow for comparability assessment, nor did they track data on program participants who chose not to participate. As a result of these limitations and the size of the sample of clients who responded to the survey, survey results could not be cross-tabulated with a satisfactory level of reliability to demonstrate differences among sub-groups of respondents. The evaluation, however, used the input of participants through key informant interviews and the focus group to validate on findings on the comparative analysis.
  • Key informant Interviews: All the representatives from the Provincial Community Futures Associations including representatives from the Community Futures Organizations and the Urban Delivery Agencies were interviewed. This group represents 100% of organizations funded annually by for the EDP. These interviews typically involved more than one person and included a historical perspective on the EDP. There is the potential for bias in their responses as these stakeholders have a strong vested interest in the EDP. The results of these interviews of stakeholders not involved with the EDP were influenced by the low overall awareness of and direct experience with the program. However, the interviewer was able to describe the EDP and to solicit responses based on a description of its program design.
  • Challenges in using EDP performance indicators to measure results: Since the inception of the EDP, a Performance Measurement Strategy has not been developed for the program. The 2005 evaluation recommended the need to develop a Performance Measurement Strategy for the program. A series of performance indicators was developed for the EDP during the renewal process in 2005. EDP partners have been using these indicators to report on results to the department; however, there were divergent definitions of indicators by reporting agencies. It was not clear how some of the indicators were linked to departmental outputs and outcomes.