Language selection

Evaluation of Entrepreneurs with Disabilities Program

Table of Contents (March 2011)

Executive Summary

Introduction

The Entrepreneurs with Disabilities Program (EDP) and the Urban Entrepreneurs with Disabilities Initiative (UEDI) was created in 1997-98 in response to the Access to Business Opportunities project and the 1996 report of the Federal Task Force on Disabilities which identified employment as a major issue that can alleviate the high incidences of poverty among people with disabilities. Both the EDP and the UEDI were consolidated into one program during the 2005-06 renewal process based on the recommendations from a 2005 program evaluation study.

The EDP provides western Canadians with disabilities with access to business information, training and development, mentoring with one-on-one counselling services, and financing in their pursuit of self-employment and entrepreneurship.

The EDP is delivered in both rural and urban communities in Western Canada. Since the initial loan funds were established at the beginning of the program, continued funding from WD has mainly been in the form of operating funds to third parties to assist in the delivery of the program. Collectively, the Community Futures Organizations currently have a balance of approximately $8 million in funds available to lend; theUrban Delivery Agents were not required to report their balance of funds available to lend. In rural communities, services for EDP clients are provided through the Community Futures Organizations. Provincial Community Futures Associations receive operating funds to help the Community Futures Organizations deliver on their EDP commitments. In urban areas in Vancouver, Victoria, Edmonton, Calgary, Saskatoon, Regina, and Winnipeg, the department works with a number of organizations by providing them with operating funds to deliver the program. Clients are provided with specialized counseling and support, as well as access to small business loans.

Under the five year agreement, the department provided a maximum $7.7 million dollars in operating funds to the urban and the Provincial Community Futures Associations to help administer the EDP. The EDP program was evaluated in 1999 and 2005. The scope of the current evaluation was from 2005-06 to the third quarter of 2010-11 fiscal years and covered both urban and rural models of the program. This evaluation focused on the EDP’s relevance and performance (efficiency, effectiveness, and economy) with regards to its activities, outputs and outcomes. The evaluation methodology included document/literature review, file and database review, comparative analysis, 51 key informant interviews, 86 client surveys, and a focus group in each of the 4 western provinces.

 

Relevance

The 2010 Federal Disability Report stated that many people with disabilities have to overcome work-related barriers and challenges that people without disabilities do not. The report concluded that despite the Employment Equity Act and a growing need for skilled workers, the statistics reveal that many people with disabilities remain underemployed and discriminated against.

Key informant interviews defined the need for the EDP mostly in terms of the extra supports for people with disabilities, beyond what is available to non-disabled entrepreneurs. The need was attributed to a number of factors such as people with disabilities not always well-served by mainstream commercial services, and the decline in the economy which supports an increased level of interest in the feasibility of the entrepreneurial option. Several key informants however questioned whether additional loan funding from the EDP was needed considering other sources of funding and financing available. Key informants viewed EDP-funded services as likely to help reduce the number of business failures, either by discouraging a start-up that is likely to fail, or by providing business supports as the start-up grows and stabilizes.

The 2005 evaluation of the EDP noted that the total number of EDP loans issued by the Community Futures Organizations since 1999 was 615 and the annual number of loans declined from 152 in 1999-00 to 88 in 2003-04. Over the period of the current evaluation (2006 to 2010), the total number of loans issued by the Community Futures Organizations was 385, with the average annual number of loans issued being 77. About 48% of all the Community Futures Organizations did not issue any loans in 2009 and most of the Urban Delivery Agents reported carrying low or no loan portfolios. This could reflect either: 1) a declining trend in the use of the EDP loan service, although there is approximately $8 million in Community Futures Organizations available for lending; or 2) high level of losses. Key informants indicated that there was a fairly high level of referrals, but a small portion of clients continue past an initial formal or informal screening to apply for loans and other business related services.

The EDP was seen as the only service that specifically target entrepreneurs with disabilities by key informants. In providing EDP services, key informants indicated that there was some potential for duplication and overlap with other initiatives. Duplication and overlap exists with parts of the EDP (some disability organizations support entrepreneurship; many small business-serving organizations serve people with disabilities), but not directly with the EDP as a whole. The EDP also overlaps to an extent with WD’s micro-loan program; and with other sources of financing and business support that could be available for entrepreneurs with disabilities (e.g. Women’s Enterprise loans; Youth loans; Business Service Centres; Incubators; many web-based resources). Other areas of duplication and overlap mentioned by key informants included: (a) BC’s two funded credit unions overlap geographically with each other and with the Community Futures Organizations; (b) urban and rural delivery organizations may overlap when clients cross geographic boundaries. These situations were not viewed as problematic, because organizations coordinate to achieve efficiency and continuity in client service. In addition, internet-based resources for entrepreneurs with disabilities are increasingly available.

The EDP falls within the departmental program activity of Business Development under the sub-activities of Improve Business Productivity and Access to Capital. The EDP priorities are also linked to federal priorities as outlined in the Government of Canada’s document, “Advantage Canada: Building a Strong Economy for Canadians”. The goal of the program reflects the Government of Canada’s longstanding commitment to uphold and protect the rights of people with disabilities and to help them participate fully in society.

 

Performance

In general, the interview data indicate that entrepreneurs with disabilities are benefiting from the program, are satisfied with the program and are achieving success as a result of the program. After the 2005 evaluation of the program, a series of performance indicators were developed to assess the results of the program. The link between some of the unique performance indicators and current departmental Program Activity Architecture (PAA) outputs and outcomes is unclear (e.g. the link between the PAA and # of partnerships or # of events sponsored). Indicators, such as number of business created/expanded/maintained, are lacking consistency in how they are reported between Urban Delivery Agents and Community Future Organizations. In addition, the EDP has been integrated into the normal operations of the Community Futures Organizations and the Urban Delivery Agents in most areas. These issues affected the data on performance that could solely be attributed to the EDP. For future renewal or extension of the EDP, the department should develop a comprehensive Performance Measurement Strategy with clearly defined performance indicators and explicit protocols for reporting.

Overall the EDP issued a total of 5751 loans from 2006 to the third quarter of fiscal 2010. Reliable data on loan default rates were not routinely collected; however, key informants and focus group participants indicated a high default rate of approximately 60%. Only 39% of survey respondents expressed a positive opinion that the EDP prevents client’s businesses from failing. Some key informants questioned the value of continuing the loan portfolio under the program.

Overall, the EDP was estimated to have created 1,614 jobs over the period evaluated. The majority of client survey respondents who started a business reported employing at least one person in addition to themselves, with the average being 3.4 employees. The majority of those who expanded their business through the EDP were able to employ at least one additional person, with an average of 4.9 employees.

The Urban Delivery Agents reported about 270 businesses created/expanded as a result of the EDP. The client survey respondents indicated that over 40% of respondents were unemployed or derived their income from other sources prior to accessing the EDP. Following participation in the EDP, roughly half of survey respondents indicated that they either started or expanded on a business which continues to operate. 37% of survey respondents indicated the EDP had a significant impact on their business.

The most significant result of the EDP was described as the “pre and post care” services offered by the EDP; the program reported almost 11,000 hours of counselling services. This service was viewed by almost all key informants as a very valuable aspect of the EDP supported advisory/training/ business services. Focus group participants supported this and further stated that as a result of this service, a high number of clients that applied for loans and went through with their applications were able to receive the loans. The approval rate for loans, as indicated by client survey respondents was quite high at 91%.

The EDP does not consume significant departmental resources for oversight or program management; the program was described as being efficient. Key informants indicated that the best feature of the program design was its flexibility. The most frequently identified challenge of the design, identified by key informants, was the lack of clarity around expectations, performance management, and inconsistency of program delivery from region to region. In comparing the EDP to other initiatives, the program was seen as a small program, with an annual overall budget of approximately $1.6 million. The comparative analysis concluded that the EDP has limited resources, and a fragmented service delivery network compared to other initiatives. It does not have a strong identity as an independent program due to the integration with other core services offered by the Community Futures Associations and the Urban Delivery Agents.

The EDP was found to be economical from several perspectives; it capitalized on the existing departmental network of Community Futures Organizations, and enhanced capacity to serve people with disabilities by way of the support and funding. The use of operating funds to support EDP activities was generally viewed as an effective use of resources by key informants.

Based on the decline in the loans service of the program and the continued need for business services, the department should determine its continued support and focus for the program. Although the findings question the need for the loan fund, the qualitative interview data indicates there is a need for the business services and support available only through the EDP. However, better performance measures are needed to quantify the utilization and impact of these business services.

 

Recommendation

The following recommendations have been made based on the findings from all lines of evidence utilised in the evaluation.

Recommendation 1: The department should determine the extent and focus of any future renewal of the Entrepreneurs with Disabilities Program.

Recommendation 2: If the department plans to renew the Entrepreneurs with Disabilities Program, then the department should develop clear quantitative performance indicators for the program.

 

Introduction

Acknowledgement

Western Economic Diversification Canada (the department) would like to thank all the key informants, EDP clients and focus group participants who generously gave their time and knowledge to take part in the study. Without their participation and their insights, this report would not have been possible. The department also acknowledges the work done by Meyers Norris Penny in conducting the key informant interviews, client surveys, comparative analysis and focus groups for the evaluation.

Entrepreneurs with Disabilities Program (EDP)

The Entrepreneurs with Disabilities Program (EDP) and the Urban Entrepreneurs with Disabilities Initiative (UEDI) was created in 1997-98 in response to the Access to Business Opportunities project and the 1996 report of the Federal Task Force on Disabilities which identified employment as a major issue that can alleviate the high incidences of poverty among people with disabilities.

The EDP and the UEDI programs were consolidated into one program during the 2005-06 renewal process based on the recommendations from a 2005 program evaluation study. The evaluation study concluded that there was significant need for these programs and that the programs had been successful. The study recognized that since the program’s inception, 765 loans totaling $16.2 million had been issued to clients under both programs across the West.

The EDP provides western Canadians with disabilities with access to business information, training and development, mentoring with one-on-one counselling services, and financing in pursuit of self-employment and entrepreneurship. The EDP makes it easier for entrepreneurs who have a disability to pursue their business goals and contribute to the economic growth of their communities. Entrepreneurs meeting the following basic criteria are considered for support:

  • have been unsuccessful in acquiring funding for business from other sources;
  • are restricted in the ability to perform at least one of the basic activities of entrepreneurship or self-employment;
  • are disabled due to physical or mental impairment;
  • have a viable business plan and are a new or current small business owner with a disability; and
  • reside in Western Canada.

The program provides entrepreneurs with disabilities who are unable to obtain financing from a traditional financial institution access to business loans. The types of projects supported by this program include:

  • starting or expanding a business;
  • purchasing and applying new technology;
  • upgrading facilities and equipment;
  • developing marketing and promotions materials; and
  • establishing working capital for anticipated sales increases.

Since the initial loan funds were established at the beginning of the program, continued funding from the department has mainly been in the form of operating funds to third parties to assist in the delivery of the program.

The EDP is delivered in both rural and urban communities in Western Canada. In rural communities, small business loans and services for EDP clients are provided through the Community Futures Organizations. The department provides the four Provincial Community Futures Association2 with operating funds to support Community Futures Organizations with resources that enable them to provide clients with specialized support, training/coaching and loans, and to undertake specific projects to improve overall capacity in rural areas.

In urban areas in Vancouver, Victoria, Edmonton, Calgary, Saskatoon, Regina, and Winnipeg, the department works with a number of organizations by providing them with operating funds to deliver the program. Clients are provided with specialized counseling and support as well as small business loans.

 

2005 EDP Program Renewal

Based on the results of the 2005 evaluation, consultations with the delivery organizations, and WD’s analysis, a decision was made to:

  • consolidate UEDI and EDP into one;
  • extend the program for five more years;
  • increase funding for the program from $775,000 to $1.5 million annually - $7.7 million over five years (operating funds);
  • adopt common data/performance management indicators; and
  • put a management information system in place to track performance data.

The increase in funding was meant to enable partners to enhance the awareness of the program, screen candidates, improve loan pre and post-care (to reduce loan loss rates), and provide training and mentoring. The five-year renewal was approved in 2006-07.

Scope and Objectives of the Evaluation

The 2009 Treasury Board Policy on Evaluation calls for departments to develop a 5-year plan to cover 100% of programs spending in a five-year life cycle. It is in this context that an evaluation of the EDP was included in the approved departmental Five-Year Evaluation Plan (2009-14). This engagement conforms to the 2009 Evaluation Policy, Directive and Standards contained therein as approved by the Treasury Board of Canada.

The EDP program was evaluated in 1999 and 2005. The scope of the current evaluation encompassed:

  • The time period from 2005-06 to the third quarter of 2010-11 fiscal years;
  • All departmental investments in the EDP within the evaluation period;
  • Both models of rural and urban delivery of the EDP; and
  • All of the major activities, operations and systems utilized in the management and administration of the EDP.

This evaluation focused on the EDP’s relevance and performance (efficiency, effectiveness, and economy) with regards to its activities, outputs and outcomes.

Organization of the Report

The report is organized as follows:

Executive Summary

Section 1: Introduction
Section 2: Overview of EDP Partners and EDP Funding
Section 3: Methodology
Section 4: Relevance
Section 5: Performance (Achievement of Objectives)
Section 6: Performance (Demonstration of Efficiency and Economy)
Section 7: Recommendations

Overview of EDP Partners and EDP Funding

Introduction

The department has contracted with organizations in each of the four western provinces to deliver the EDP. The organizations include 90 Community Futures Organizations and their associated Provincial Community Futures Associations, and 7 Urban Delivery Agents in western Canada.

Province RURAL DELIVERY AGENTS URBAN DELIVERY AGENTS
British Columbia
  • 34 Community Futures Organizations
  • Community Futures Development Association of BC
  • Coast Capital Savings
  • Vancouver City Savings Credit Union
Alberta
  • 27 Community Futures Organizations
  • Community Futures Network Society of Alberta
  • Distinctive Employment Counseling Services
  • Momentum Community Economic Development Society
Saskatchewan
  • 13 Community Futures Organizations
  • Community Futures Partners of Saskatchewan
  • North Saskatchewan Independent Living Resource Centre
  • South Saskatchewan Independent Living Resource Centre
Manitoba
  • 16 Community Futures Organizations
  • Community Futures Partners of Manitoba
  • Independent Living Resource Centre

Community Futures Organizations

Community Futures Organizations are non-profit organizations established to deliver the Community Futures Program. They are independent, locally autonomous corporations that each operate under the direction of a volunteer board. The mandate of the organizations is community and business development. Each Community Futures Organization delivers a variety of services ranging from strategic economic planning, technical and advisory services to businesses, loans to small and medium-sized businesses, self-employment assistance programs, and services targeted to youth and entrepreneurs with disabilities.

Initially, each Community Futures Organization received $200,000 in loan funds and $50,000 in operating funds to deliver the EDP. The organizations tend to use the same process to manage their EDP loan portfolio as they use for their general loan fund. The loan applications are adjudicated by a loans committee, which consists of volunteers from the local business community. The staff and loans committee also works to refer applicants to commercial lending sources.

In recent years, the Community Futures Organizations have not received any direct operating funding related to the delivery of EDP but rather are expected to fund the services provided to entrepreneurs with disabilities through their core operating funding, with support from the projects and initiatives implemented by the Provincial Community Futures Associations.

 

Provincial Associations for Community Futures

Associations have been established in each of the four western provinces to represent the Community Futures Organizations. The Community Futures Development Association of BC, Community Futures Network Society of Alberta, Community Futures Partners of Saskatchewan, and Community Futures Partners of Manitoba. The four associations are not involved in delivering services directly to EDP clients but rather work to support the Community Futures Organizations in delivering services. More specifically, each provincial association administers a pool of annual funding provided by the department in accordance with the following principles:

  • Funding is to be used to support entrepreneurs with disabilities throughout non-metropolitan areas of the province.
  • Funding is to be used to support dedicated human resources with skills specific to helping people with disabilities.
  • Funding will flow through the associations in each province rather than directly to the resources contracted to deliver the services.
  • Activities supported should be consistent and coordinated with the urban EDP initiatives in each province.

Each provincial association consults with the department, Community Futures Organizations, entrepreneurs with disabilities, and various groups who represent people

Urban Delivery Agents

The department has contracted with 7 delivery agents in western Canada to deliver the EDP. At the beginning of implementation of the EDP, the delivery agents received a mixture of capital funding, which was conditionally repayable (initial disbursement of $500,000 each3) and funding for operations ($100,000 each annually) which was not repayable. Over time, the agencies received approval to transfer some of the loan funds to operating funding. Current funding from the department is in the form of operating funding only. The agencies receive EDP funding to help clients;

  • start or expand a business;
  • purchase and apply new technology;
  • upgrade facilities and equipment;
  • develop marketing and promotional activities; and
  • establish working capital.

 

British Columbia

In Vancouver and Victoria, Vancouver City Savings Credit Union and Coast Capital Savings Credit Union offer the Vancouver and Victoria components of the EDP, which is known as ABLED (Advice and Business Loans for Entrepreneurs with Disabilities). Currently the program in BC is delivered under the department’s Loan and Investment Program agreements. Under this agreement, the department provides a loan loss reserve to the financial institutions to reduce their exposure in this area of business financing considering high risk. The department also provides annual operating funds to the institutions to help offset the cost of administering the program.

The program is specifically designed to improve access to business services and other support mechanisms to entrepreneurs with disabilities who are thinking of stating or expanding a small business. The program provides pre- and post-business launch support tailored to the realities of entrepreneurs with disabilities. Eligible loan applicants can receive funding for business planning and training, business coaching or mentoring services, and business counseling.

Alberta

In Edmonton, the EDP is administered by the Distinctive Employment Counselling Services of Alberta under their Ventures Program, and in Calgary, it is administered by Momentum under their FundAbility Program.

The Ventures Program in Edmonton provides entrepreneurial training and business support to people with disabilities who want to pursue self-employment. Both programs offer assistance in development of business plans and business skills for entrepreneurs with disabilities. Participants may access loans of up to $75,000. The FundAbility Program at Momentum in Calgary offers access to loans of up to $10,000. The FundAbility Program represented 9.45% of all lending fund programs at Momentum.

Saskatchewan

In Saskatchewan, the North Saskatchewan Independent Living Centre in Saskatoon and the South Saskatchewan Independent Living Centre in Regina manage the EDP on behalf of the department.

The two centers take a lead role in providing mentoring and counselling services, support in business development stages, as well as business loans to entrepreneurs with disabilities. Through the EDP, the Independent Living Centers: promote entrepreneurship and self-employment; provide mentorship and peer support; provide access to information resources and community services; provide support in developing a business plan, financial forecasts and researching capital options; provide a mechanism for follow-up and aftercare; and provide an option for capital (and the possibility of leveraging additional money) through an investment fund.

Manitoba

In Winnipeg, the Independent Living Resource Centre delivers the EDP on behalf of the department. It provides a range of business support services to entrepreneurs with disabilities, including: coordinating partnerships to better assist entrepreneurs with disabilities; assistance in developing business skills and business plans; networking opportunities; and consulting services to further support entrepreneurs with disabilities.

 

Departmental Funding to EDP Partners

Under the five year agreement approved in 2006/07 fiscal year, the department provided a maximum of $1.5 million annually ($7.7 million dollars over the five-years) in operating funds to the Urban Delivery Agents ($1 million annually) and the Provincial Community Futures Associations ($500,000 annually) to help administer the EDP. Table below indicates funding disbursed to the partners since 2007 from the departmental financial database.

Table 2-1: Departmental Financial Disbursement by Fiscal Year, EDP
Name 2006 -2007 2007 - 2008 2008 - 2009 2009 - 2010 2010 - 2011 Total
Coast Capital Savings $75,000 $89,534 $83,470 $64,601 $69,669 $382,274
Community Futures Development Association of BC $1,519 $124,161 $173,552 $198,536 $180,000 $677,768
Vancouver City Savings Credit Union $58,218 $84,089 $175,400 $147,368 $142,743 $607,818
BC $134,737 $297,784 $432,422 $410,505 $392,412 $1,667,860
Community Futures Network Of Alberta $151,697 $148,303 $137,646 $147,462 $163,455 $748,563
Distinctive Employment Counseling Services of AB $95,901 $204,099 $150,000 $149,688 $135,753 $735,441
Momentum Community Economic Development $96,952 $114,660 $128,659 $143,740 $146,392 $630,403
AB $393,709 $461,126 $600,684 $635,527 $381,317 $2,472,363
CF Partners of SK $80,000 $90,000 $100,000 $100,000 $100,000 $470,000
North Saskatchewan Independent Living Centre $94,822 $115,000 $135,000 $150,000 $150,000 $644,822
South Saskatchewan Independent Living Center $95,000 $114,929 $125,048 $148,178 $150,000 $633,155
SK $269,822 $319,929 $360,048 $398,178 $400,000 $1,747,977
CF-MB $88,012 $94,551 $99,000 $93,430 $108,583 $483,576
Independent Living Resource Center $112,083 $140,493 $152,761 $150,000 $150,000 $705,337
MB $200,095 $235,044 $251,761 $243,430 $258,583 $1,188,913
Grand Total $998,363 $1,313,883 $1,644,915 $1,687,640 $1,432,312 $13,796,270

Source: Departmental Financial System accessed May 5, 2011

Methodology

Scope of Work

The logic model underlying this theory-based evaluation was developed in consultation with the advisory group. The evaluation used a non-experimental design because a control group was unavailable; however, when used with multiple lines of evidence, the design met the objectives of the evaluation. This evaluation of the department's investments in the EDP included a wide range of data collection methods, each of which is described below. The scope of the evaluation included the following evaluation issues: relevance and performance (efficiency, effectiveness and economy). The objectives of the evaluation and the core evaluation issues are presented in table 3-1.

Table 3-1  Core Evaluation Issues
  Evaluation Issues
Relevance 1. Is there a continued need for the EDP?
2. Is there a continuing demand for the EDP?
3. Is the program aligned to WD & federal government priorities?
4. Is the program consistent with federal roles and responsibilities?
Performance (Achievement of Outcomes, and Demonstration of Efficiency and Economy) 5. To what extent is the EDP effective in meeting its objectives?
6. What factors facilitate or impede the achievement of program results?
7. To what extent has the EDP realised its intended outputs?
8. To what extent have intended outcomes been achieved as a result of the program?
9. To what extent did the EDP contribute to its long-term outcomes?
10. To what extent did the operating funds impact on the delivering of EDP services (loans, advice etc) in rural and urban sectors?
11. Is the program design (rural and urban) appropriate for achieving expected program results?
12. Have there been any unintended (positive or negative) outcomes?
13. Is the program undertaking activities and delivering products in the most efficient manner?
      • How could the efficiency of the program's activities be improved?
      • Are there alternative, more efficient, ways of delivering the program?
14. Is the program achieving its intended outcomes in the most economical manner?

 

Evaluation Study Activities

Preliminary Consultations and Evaluability Assessment

Preliminary consultations were conducted with departmental officers responsible for the agreements in the development of the evaluation framework and also to discuss data availability. Through these consultations, the scope of the evaluation was established and final terms of reference developed for the evaluation. An Advisory Committee made up of departmental representatives from each region was set up to provide support to the evaluation in generating useful and relevant information by contributing to improve the effectiveness of the evaluation, and ensuring quality assurance of evaluation products. Based on the evaluability assessment, a consultant was engaged to collect some of the data for the evaluation. The final report was written using internal evaluation resources.

Document and Literature Review

The evaluation included the review of documents and literature containing information relevant to the EDP. The review was used primarily to assess relevance and performance of the agreements.

The main types of documents assessed and analyzed during the evaluation were:

  • General background documentation (e.g., program agreement documents, websites, media reports, EDP Program documentation including funded project descriptions, program guidelines, status reports, performance reports, records held by rural and urban partners of the EDP etc.); and
  • Specific documentation (e.g., departmental website, Terms of Reference, previous evaluation reports and other studies related to the EDP).

Information from the review was used in developing the logic model for the evaluation and to complete the Terms of Reference for the evaluation.

Project File Review and Data Analysis

The evaluation team reviewed the project files related to the EDP and conducted an analysis of project data available through the department's project management database, financial system, and the TEA4 database used by the Community Futures Organizations. Because Saskatchewan's information was missing from the department's project management database, hard file reviews were completed during fieldwork in Saskatchewan. Results from previous evaluations were used as baseline data for this study. The evaluation team also consulted with departmental EDP Officers and stakeholders from the delivery agents to validate data collected through the file and database review. Data was collected for all 7 urban delivery agents, 4 provincial associations, and 90 Community Futures Organizations.

Key Informant Interviews

The department's EDP program managers took part in a focus group held in Edmonton (in-person and by video conference), and the other respondent groups provided input by way of a telephone or in-person interview. Groups represented included: Provincial Community Futures Associations, Urban Delivery Agents, Community Futures Organizations and community stakeholders.

In all, 51 individuals representing 38 organizations provided their input on the key research questions. Question guides were distributed to participants prior to the interview/focus group. Interviews lasted on average 1.25 hours and the focus group lasted two hours. One organization supplemented their interview with a written response that represented the input of their Board, approximately eight additional individuals (not counted in the total numbers below; however, the written response was included in the data roll-up).

Many of the interviewees had a long history of involvement with the EDP, and were able to provide a historical perspective. For most interviewees the EDP is a part of their overall job responsibility and duties. This analysis was completed by the consultant.

Table 3-2   Stakeholders Providing Input
Stakeholder group No. of organizations represented No. of individuals participating
Western Diversification, Program Managers, (Focus Group) 4 8
Community Futures Associations (Interviews) 4 7
Urban Service Delivery Organizations (Interviews) 7 13
Community Futures Organizations (Interviews) 11 11
Stakeholders (Interviews) 12 12
Total 38 51

Sampling approach for selecting key informant participants:

  • Departmental Program Managers: Representatives from each province were selected based on their familiarity with and their role in overseeing the EDP.
  • Community Futures Associations: The three associations from each province were chosen to represent different types of communities and levels of engagement in the Entrepreneurs with Disabilities Program.
  • All Urban Delivery Agents participated.
  • Stakeholders: A representative sample of community stakeholders was chosen based on geographic spread and their knowledge of services to entrepreneurs and services to people with disabilities.

Comparative Analysis

The comparative analysis included performing a comparative analysis of the EDP services to other programs, organisations, departments and services available to entrepreneurs with disabilities to access capital in all four western Canadian provinces, and the federal government.

A survey of ten related organizations/programs/services to the EDP was conducted, in addition to a scan of web-based resources available to entrepreneurs. The surveyed organizations serve people with disabilities, providing a range of services including support for self-employment. Information was collected based on website content, articles available, and in some cases, direct consultation with a member of the organization. This analysis was completed by the consultant. The organizations surveyed included:

  • Opportunities Fund for People with Disabilities: A Canada-wide program offered by the department of Human Resources and Skills Development, established in 1997 to assist people with disabilities in preparing for, obtaining and keeping employment or becoming self-employed, and increasing their economic participation and independence.
  • Canadian Society for Social Development: An independent Canada-wide organization, providing employment and skills training to persons facing challenges to employment: persons living with a disability, youth, stay at home parents, older workers, Aboriginal Peoples, individuals living in rural areas.
  • Canadian Council on Rehabilitation and Work: A Canada-wide organization with a mandate to promote and support meaningful and equitable employment for people with disabilities.
  • SEED Winnipeg: A Winnipeg-based program established in the late 1980s to fight poverty and help renew Winnipeg's inner city by helping entrepreneurs start small businesses and save money for future goals.
  • Employability Assistance for People with Disabilities: A Saskatchewan based program, providing funding to assist adults with disabilities to prepare for, secure and maintain employment.
  • EmployAbilities: A community-based, non-profit organization operating in Alberta to provide employment programming and learning opportunities for people with disabilities
  • Disabilities Related Employment Supports: An Alberta based program providing education; workplace and job search to support people with disabilities.
  • Vancouver Foundation Disability Support for Employment Fund: A Vancouver based program offering grants to assist eligible organizations throughout British Columbia who are working to increase employment opportunities and ultimately the rate of employment for people with disabilities in their communities.
  • Employment Program for People with disabilities (EPPD): A program through British Columbia's Ministry of Housing and Social Development, providing a range of specialized services to help individuals with disabilities participate in their communities, pursue their employment goals as they are able, increase their self-reliance, and build skills and experience that may lead to further employment or volunteer opportunities.
  • BC Coalition of People with Disabilities: A BC-based organization, championing issues impacting the lives of people with disabilities through direct services, community partnerships, advocacy, research, and publications.
    The analysis also included the services offered by the department's Community Futures Program to entrepreneurs through the Community Futures Organizations. The organizations were compared with the EDP based on factors including:
    • Regions served;
    • Loan capacity;
    • Services provided - business planning, training, research assistance, business advisory, referral, networking and mentoring;
    • Overall service delivery model - including rural vs. urban services;
    • Target clients; and
    • Performance.

 

Client Survey

As part of the evaluation, a survey was conducted of EDP clients who have accessed the program in the past 5 years to share their experiences. When all EDP client contacts were accounted for, the total sample was 544 clients. In order to compile an appropriate sample, each service delivery agent (both urban and rural) was asked to provide their client lists.

The delivery agents provided a total of 354 individual EDP contacts. Of the total provided, only 270 were found to be usable. Some 84 contacts could not be used due to out of service phone numbers, bounced e-mails and outdated information. In total, 86 successful contacts were made, 27 of which were done via email and 54 of which were done via telephone. An additional 5 survey responses were received manually following the completion of the analysis. This is a 32% response rate for the usable sample of 270 client contacts provided for the evaluation.

The following table illustrates the actual distribution of clients who completed the survey across each province and organization and the response rate:

Table 3-3  EDP Client Survey Respondents by Province – Actual*
Region # of People Invited to Participate in Survey of Respondents Response Rate
BC      
Urban 17 5 29.41%
Rural 80 25 31.25%
Subtotal 97 30 30.93%
AB      
Urban 43 10 23.26%
Rural 109 10 9.17%
Subtotal 152 20 13.16%
SK      
Urban 141 23 16.31%
Rural 14 2 14.29%
Subtotal 155 25 16.13%
MB      
Urban ** 126 0 0.00%
Rural 14 11 78.57%
Subtotal 140 11 7.86%

Total

544 86 15.81%

*Response rate is calculated by dividing # of Respondents by # of People Invited to Participate in Survey.
**For confidentiality reasons, the Urban agent did not provide a client list to the evaluators; instead, the Urban agent sent letters to their clients requesting their participation. None of the clients participated.

The total distribution of urban vs. rural response to the survey is shown below.

Table 3-4   Urban vs. Rural
Provider EDP Clients Percent of Total Volume
Urban 38 44.2%
Rural 48 55.8%
Total 86 100.0%

 

Focus Group

Validation focus groups for the EDP evaluation were held in Vancouver, Edmonton, Saskatoon and Winnipeg. In all, 37 individuals (Table 3 – 5) participated in the focus groups either in person, by videoconference, or teleconference. The consultant presented an overview of findings, and discussion questions to stimulate input from participants. The sampling approach outlined for selecting the key informants was also applied to selecting the focus group participants.

Table 3-5   Numbers of attendees by category
Region Community Futures Organizations Urban Delivery Agents Other Stakeholders WD Staff Provincial Community Future Associations Total
Manitoba 2 1 2 2 1 8
Saskatchewan 1 6 1 2 - 10
Alberta 2 5   3 2 12
British Columbia 1 1 2 2 1 7
TOTAL 6 13 5 9 4 37

Limitations of the Evaluation

  • Data Analysis: The evaluation used data reported by the Community Futures Organizations in the TEA database, and data reported by the Urban Delivery Agents and the Provincial Community Futures Associations in the department's project management database and files. The interviews and focus groups indicated that data reporting is not consistent from region to region. In addition, it was unclear if the delivery agents had a consistent definition regarding how they define and track clients and results. Data generated from each of the databases was validated with the provincial associations and the urban delivery agents. In cases where the data could not be reconciled through the validation process, it was not used in the analysis.
  • Survey: Selection bias in the survey was a potential threat to the validity of findings and the ability to apply findings to all entrepreneurs with disabilities. Clients invited to participate in the survey likely directly participated in the EDP program and who may have experienced greater success with their business. Those who failed may not have responded or may be under-represented in the findings. In addition, the survey sample was not representative of the total population. Results of the survey were therefore used to support other lines of evidence in the analysis.
  • Comparative Analysis: Since the EDP is a small component of a larger suite of programs and services offered by the delivery agents, it was difficult to isolate the EDP's effect on a program participant. The use of an experimental design to assess incremental impact was not feasible for the EDP. Matched groups could not be identified and tracked to be compared to program participants. EDP delivery agencies did not record client descriptor information that would allow for comparability assessment, nor did they track data on program participants who chose not to participate. As a result of these limitations and the size of the sample of clients who responded to the survey, survey results could not be cross-tabulated with a satisfactory level of reliability to demonstrate differences among sub-groups of respondents. The evaluation, however, used the input of participants through key informant interviews and the focus group to validate on findings on the comparative analysis.
  • Key informant Interviews: All the representatives from the Provincial Community Futures Associations including representatives from the Community Futures Organizations and the Urban Delivery Agencies were interviewed. This group represents 100% of organizations funded annually by for the EDP. These interviews typically involved more than one person and included a historical perspective on the EDP. There is the potential for bias in their responses as these stakeholders have a strong vested interest in the EDP. The results of these interviews of stakeholders not involved with the EDP were influenced by the low overall awareness of and direct experience with the program. However, the interviewer was able to describe the EDP and to solicit responses based on a description of its program design.
  • Challenges in using EDP performance indicators to measure results: Since the inception of the EDP, a Performance Measurement Strategy has not been developed for the program. The 2005 evaluation recommended the need to develop a Performance Measurement Strategy for the program. A series of performance indicators was developed for the EDP during the renewal process in 2005. EDP partners have been using these indicators to report on results to the department; however, there were divergent definitions of indicators by reporting agencies. It was not clear how some of the indicators were linked to departmental outputs and outcomes.

Relevance

Need for Services for Entrepreneurs with Disabilities

The 2010 Federal Disability Report5 notes that the average employment income for working-age adults with disabilities is $29,393, which is 22.5% lower than the average of $37,994 for working-age adults without disabilities. Severity of disability affects income: the average employment incomes for those with severe disabilities and very severe disabilities are $23,786 and $19,447 respectively, compared to $33,427 for those with mild disabilities. The Statistics Canada’s 2006 Participation and Activity Limitation Survey indicates that the overall labor force participation rate for working-age adults with disabilities is 59.6% (1,379,325 people). In comparison, the participation rate for working-age adults without disabilities is 80.2% (15,163,250 people).

From the report, many people with disabilities have had to overcome work-related barriers and challenges that people without disabilities do not. People with disabilities may be limited in the amount of work they can do in the workplace, or they may require workplace modifications or flexible working arrangements that employers are sometimes reluctant to provide. In addition to accommodation challenges, workers with disabilities face the possibility of employer discrimination and social exclusion. People with disabilities were more likely to be unemployed than people without disabilities. The unemployment rate for working-age adults with disabilities is 10.4%, compared to 6.8% for working-age adults without disabilities.

The Federal Disability Report concludes that workers with disabilities make up a significant portion of the workforce, and a number of unemployed working-age adults with disabilities are qualified and ready to work. However, despite the Employment Equity Act and a growing need for skilled workers, the statistics reveal that many people with disabilities remain underemployed and discriminated against. The report highlighted the department’s Entrepreneurs with Disabilities Program (EDP) as one of the means to help people with disabilities overcome employment related challenges.

In the key informant interviews, the need for services for entrepreneurs with disabilities was defined mostly in terms of the extra supports for people with disabilities, beyond what is available to non-disabled entrepreneurs. The need to support other organizations that provide services to entrepreneurs with disabilities was also emphasized. Key informants attributed the need to various factors such as:

  • People with disabilities are still not always well-served by mainstream services,
  • A broader range of disabilities now considered within the realm of “disabled” (the generally accepted definition of disability has expanded),
  • The decline in the economy which support an increased level of interest in and feasibility of the entrepreneurial option for people with disabilities.

Key informants viewed entrepreneurs with disabilities as having unique challenges that cannot always be met by mainstream organizations. The unique challenges included managing health-related issues, and planning the business within consideration of disability limitations. People with disabilities may also need specific accommodations to contribute to the feasibility of a business start-up. These accommodations are most often not provided by mainstream organizations. Interviewees reflected an overall philosophy that supported “levelling the playing field” for people with disabilities through providing additional assistance where it leads to successful outcomes for that individual, and for the community overall (e.g. business start-up benefits).

Need for EDP Program

Overall, the EDP is a valued program. While the majority of key informants indicated that both loan and supportive business services meet the needs of, and are beneficial to, entrepreneurs with disabilities, approximately one quarter of respondents questioned the extent to which the loan fund was needed and its benefits. Several key informants questioned whether additional loan funding was needed considering the other sources of financing available to entrepreneurs with disabilities. However, most organizations provide EDP loans at the same terms as other loans but with more support and flexibility in repayment.

Key informants mentioned that EDP funding may be used to help other organizations serve specific clients, work generally with people with disabilities, and market the EDP services. Key informants indicated that providing business advisory services through the EDP may also reduce the number of business failures, either by discouraging a start-up that is likely to fail, or by providing business support.

The comparison with related programs indicates the following about the continued need for the EDP:

  • Need for capacity-building and specialized supports: For some of the related program/service areas, the EDP funding was viewed as indirectly increasing capacity to serve entrepreneurs with disabilities by partnering with other community organizations.
  • Need to promote the entrepreneurial option: EDP funding may promote the entrepreneurial option, by way of funded initiatives that promote self-employment as an option for people with disabilities.
  • Need for more intensive supports for entrepreneurs with disabilities: The most valued aspect of the EDP was identified as the “pre and post care” service. This service was recognized as most needed to ensure that businesses supported have a high likelihood of being successful.

The majority of respondents in the client survey indicated that they had applied for loan financing through the EDP. The approval rate for the EDP, as indicated by survey respondents, was quite high at 91%. Focus group participants attributed this high approval rating to the “pre and post care” service offered by the EDP. Through this service, only businesses that had a high potential for success were able to complete the application for funding.

Figure 4-1   Percentage of Start-up Cost Covered by EDP Loan In this figure, key informants indicate amount of start-up funding provided by EDP loan.

Text Version Figure 4-1: Percentage of Start-up Cost Covered by EDP Loan

Nearly half of the survey respondents indicated that greater than 75% of their costs were financed through the EDP program. Where shortfalls in funding occurred, the majority of respondents used their own personal resources, or sought help from family and friends, to finance the endeavour. Survey respondents did not mention EDP funding as being used to leverage other funding.

Traditional commercial lending was not considered a viable option for either supplementing or replacing EDP loans, as roughly half of respondents indicated that they were not confident of being approved by traditional lenders. Where financing external to the EDP program was a viable option, 72% of respondents indicated that the EDP provider was somewhat or very helpful in coordinating access to other resources.

 

Trend and Demand for the Entrepreneurs with Disabilities Program

The 2005 evaluation of the EDP noted that the total number of EDP loans issued by the Community Futures Organizations was 615 and the annual number of loans declined from 152 in 1999-2000 to 88 in 2003-2004. Over the period of the current evaluation (2006 to the third quarter of 2010), the total number of loans issued by the Community Futures Organizations was 407, and the average annual number of loans issued was approximately 78. About 45% of all the CF’s did not issue any EDP loans in 2009-10 or the first three quarters of 2010. Most of the urban delivery agents reported currently carrying low or no loan portfolios in the key informant interviews. These numbers indicate a declining trend in the use of the loan service aspect of the EDP. Data collected for the EDP indicates a large number of clients were served by the EDP delivery agents (16,567)6 ; however, this is in contrast to the small number of loans provided (572) from 2006 to 2010. These findings then indicate that most demand for the EDP has been in services other than for loans and financing.

Key informants identified two trends that would indicate a possible decrease in the need for the EDP services in loans: 1) the reluctance of people with disabilities to reveal their disabilities; and 2) the ability of mainstream organizations to meet the needs of people with disabilities. In addition, key informants mentioned that EDP clients may have become disinterested after initial contact if: loan or grant funds were not available, or they were required to complete a business plan or meet other requirements. Key informants indicated that there was a fairly high level of referrals (for educational and promotional activities, etc.) but a small portion of clients continued past an initial formal (or informal) screening to apply for loans and other services. This probably accounted for the high number of clients served compared to the small number of loans issued.

About 42% of client survey respondents indicated access additional external, non-financial support. The most cited sources of additional financial support by respondents are indicated in the figure below.

Figure 4-2   Sources of Additional FundingIn this figure, key informants indicate additional sources of funding for their business ventures.

Text Version Figure 4-2: Sources of Additional Funding

The comparative analysis noted that the demand for EDP services was affected by the small size of the entrepreneurs with disabilities market. Other programs and organizations that targeted distinct groups of entrepreneurs such as women, Aboriginal, or youth entrepreneurs were mentioned as having a significantly larger market share than the EDP.

The 2010 Federal Disability report noted that people with very severe disabilities are least likely to desire full-time work (20.5%) but are more flexible in working either part time or full time (42.0%). The report continued that 65.0% of people with disabilities who are not in the labour force are completely prevented from working, including 76.1% of people with severe disabilities and 83.9% of people with very severe disabilities. Additionally, 12.1% of people with disabilities who are not in the labour force are limited in their ability to look for work. This further reduces the market for the EDP.

 

Presence of Other Initiatives that serve Entrepreneurs with Disabilities

Key informants stated that there are other initiatives that assist people with disabilities to obtain employment or assisted living support. There are also other initiatives that target entrepreneurs in general. However, the EDP is unique in specifically targeting entrepreneurs with disabilities. In providing EDP services, key informants indicated that there may be some potential for duplication and overlap with other programs. Urban Delivery Agents noted that there was some overlap in program offering given that they were in the same geographical region. In addition, some key informants mentioned that the EDP could overlap with some departmental programs such as the Loan and Investment Program, and the Women’s Enterprise Initiative.

The comparative analysis indicated the following findings about related programs that provide similar services to the EDP in western Canada and are targeted generally to people with/without disabilities:

Many related programs, services, and organizations provide services to entrepreneurs. Many organizations, programs and services (including many Internet-based resources) support entrepreneurs in both rural and urban settings, including programs that target segments of the entrepreneurial population, such as women, Aboriginal entrepreneurs, and youth. These may be funded by one, or a combination of federal, provincial and municipal governments, non-governmental organizations, or they may be private enterprises. These programs may offer access to loans, and/or may provide business start-up information, supports and linkages to other sources of financing. Examples of such programs, services and organizations include banks and credit unions that provided services to entrepreneurs in general.

Many related programs, services and organizations serve people with disabilities. Many organizations, programs and services also provide a range of programs and services targeted specifically to people with disabilities in both rural and urban settings. Initiatives that target people with disabilities provide a range of services, including: services to assist with daily living, access to funding for disability-specific costs, supporting further education and employment, acting as an advocate on behalf of people with disabilities, and promoting and support of people with disabilities to employers.

Examples of such initiatives include:

  • The Opportunities Fund for People with Disabilities: This initiative is administered by the federal government and it seeks to assist people with disabilities in preparing for, obtaining and keeping employment or becoming self-employed, and increasing their economic participation and independence. The program also encourages employers to provide opportunities and work experience programs for people with disabilities.
  • Employability Assistance for People with Disabilities: This initiative is a Saskatchewan-based program, providing funding to assist adults with disabilities to prepare for, secure and maintain employment. Various supports are offered, including: on-the-job-training, vocational and work assessments, psycho-educational assessments, job coaching, support for employers and disability-related costs in a wide variety of post-secondary education and training programs.

Employment programs are also widespread. Some targeted toward persons with disabilities, some for a broader client group. For clients who are unemployed and looking for work, many supports are available through federal/provincial funded employment programs. Some of these are targeted toward people with disabilities, some toward a broad base of eligible clients, and some are targeted toward self-employment. The self-employment courses often have eligibility criteria relating to unemployment, and are comprehensive for individuals interested in starting a business. As a result of EDP funding, an EDP-funded organization may work with or supplement the services of the organization that provides the employment /self-employment programs, to better meet the needs of the entrepreneurs with disabilities.

Example:

  • BC’s Ministry of Housing and Social Development’s Employment Program for Persons with Disabilities (EPPD): The EPPD is an example of an organization with a wide-reaching span of services targeted towards employment seekers, persons with disabilities, and entrepreneurs. The EPPD provides a range of specialized services to help individuals with disabilities participate in their communities; pursue their employment goals as they are able, increase their self-reliance, and build skills and experience that may lead to further employment or volunteer opportunities. It is intended to assist persons with disabilities to achieve their economic and social potential to the fullest extent possible. Offered throughout the province of BC, the EPPD provides programs with individualized services provided through Service Provider contracts.

Increased accessibility of internet-based resources for entrepreneurs with disabilities. To the extent that the EDP funds organizations to provide information and resources to people with disabilities considering entrepreneurship, resources are increasingly available on the internet for these services. Service providers are developing web-based instructional courses, information sources, and self-assessment tools.

An example of such an organization that provides internet services for people with disabilities is the Canadian Society for Social Development. The organization provides employment and skills training to persons facing challenges to employment such as people with disabilities, youths, Aboriginals, and individuals living in rural areas. The organization’s Business Abilities website provides resources for entrepreneurs with disabilities seeking to start their own business. The following table below compare the services of the EDP with a few other initiatives in terms of their service provision.

Table 4-1 Comparison of Services offered by EDP to Other Initiatives Serving People with Disabilities
Organizations Loans Business Planning Training Business Advisory Referral Services Networking/
Mentoring
Entrepreneurs with Disabilities Program x x x x x x
HRSDC Opportunities Fund for People with disabilities   x x x   x
Canadian Society for Social Development   x x   x x
Canadian Council on Rehabilitation and Work     x x x  
SEED Winnipeg   x x   x x
Employability Assistance for People with Disabilities     x x x  
EmployAbilities     x x x  
Disabilities Related Employment Supports     x x x  
Vancouver Foundation Disability Support for Employment Fund       x    
Ministry of Housing and Social Development’s Employment Program for People with disabilities (EPPD)     x      
BC Coalition of People with Disabilities     x     x
Women Enterprise Initiative x x x x x x
Community Futures Program x x x x x x

The EDP has been in existence since 1999 when the first agreements were signed. The department needs to determine its continued support for the program and determine whether the EDP best serves the needs of entrepreneurs with disabilities. This recommendation is due to: the decline in demand for loan service of the program, the existence of other organizations and initiatives that have similar objectives, and the potential for overlap and duplication with other organizations and initiatives. Although the need for the loan fund is questionable, interviewees indicate a continuing need for the business services and support. To validate the continued demand for services, the department needs to revise the program’s performance measurement strategy to include performance measures that capture the non-loan, service support components of the program that are currently measured only qualitatively.

Alignment with Departmental, and Federal Priorities

The EDP falls under the department’s Business Development program activity and addresses the two sub-activities of “Improve Business Productivity” and “Access to Capital”. The EDP priorities are also linked to federal priorities as outlined in the Government of Canada’s “Advantage Canada: Building a Strong Economy for Canadians” document7.

The goal of the EDP is to make it easier for Canadians who have a disability to pursue entrepreneurship and contribute to the economic growth of their communities. This goal reflects the Government of Canada’s longstanding commitment to uphold and protect the rights of people with disabilities and to help them participate fully in society. The Government’s commitment includes removing obstacles and creating opportunities for Canadians with disabilities. There are approximately 4.4 million people with disabilities in Canada representing about 14.3 percent of the population. The Government reaffirmed this ongoing commitment to supporting people with disabilities on March 11, 2010, when Canada ratified the United Nations Convention on the Rights of People with Disabilities.

Most key informants viewed the EDP as having linkages to both economic and social policy objectives, with the strongest link to social policy objectives. The benefits of the program are viewed most strongly as improving quality of life, and greater independence (along with increased self-esteem and self-confidence). The economic benefits to individuals and to communities are also viewed as fitting with WD’s overall economic development mandate.

Key informants strongly supported federal government funding for assisting entrepreneurs with disabilities. The funding to the Provincial Community Futures Associations was viewed in most provinces as being well-spent and appropriately accountable. Some key informants, however, questioned whether the federal government should continue to provide loan funding under the EDP separately from its other entrepreneurial funding streams.

 

Performance (Achievement of Outcomes)

The objective of the EDP is to provide western Canadians with disabilities with easy access to business information, training and development, mentoring and one-on-one counselling services, and financing in their pursuit of self-employment and entrepreneurship. In general, the interview data indicate that entrepreneurs with disabilities are benefiting from the program, are satisfied with the program and are achieving success as a result of the program. After the 2005 evaluation of the program, a series of performance indicators were developed to capture the results of the program. The link between these performance indicators and current departmental Program Activity Architecture outputs and outcomes is not clear. Urban Delivery Agents reported on the number of business created/expanded/maintained; however, this indicator was not reported by the Community Futures Organizations. In addition, some of the indicators such as number of hours spent in coaching activities, are not linked to any of the outcomes in the Program Activity Architecture.

Some of the Community Futures Organizations are involved in delivering other initiatives for the provincial and federal governments; as a result they indicated that data in the database on the performance of the EDP includes results from other initiatives. Focus groups participants validated that client counting has been very inconsistent from region to region. In addition, the operation of the EDP has been integrated into the normal operations of the delivery agents in most areas. In British Columbia, urban EDP operations have been integrated into the operations of the Loan and Investment Program. Other Urban Delivery Agencies and the Community Futures Organizations have largely integrated the EDP loan funds with their core loan portfolios. Because of the issues with data integrity, it is exceedingly difficult to assess actual performance indicators of the EDP. For future renewal or extension of the EDP, the department should develop clear quantitative performance indicators for the program.

Factors that Impeded the Measurement of Results

The most common challenges identified by key informants on the achievement of results for the EDP were:

  • Program accountability:
    • Performance measures gathered through the indicators did not reflect relevant outcomes (especially those relating to loans);
    • Integration of EDP services with the core services of the delivery agents, which made data reporting challenging;
    • Performance measures not defined adequately (service providers counted differently); and
    • Overall expectations of the EDP as described in relation to performance measure are not clear.
  • Achieving consistent levels of service; eligibility criteria; definitions; terms and conditions:
    • The EDP is viewed as somewhat fragmented and lacking consistency;
    • The urban delivery agencies have wide ranging approaches and usage of funds, even though their contracts with the department look very similar;
    • The Provincial Community Futures Associations reported that engaging all Community Futures Organizations in EDP-related activities is a challenge;
    • Differences in the terms and overall management of the fund from region to region;
    • Many organizations no longer have loan funds, so their commitment to the EDP may be lacking as a result of the depletion of the loan fund; and
    • Reluctance to disclose disabilities. Many interviewees identified a reluctance to disclose disabilities as a barrier to providing more direct service to entrepreneurs with disabilities.

Achievement of Outputs and Outcomes of the EDP

Key informants indicated that activity levels and the types of supports for the EDP vary across components of the EDP and this variation occurs between the rural and urban delivery agents of the program. As a result, the EDP meets needs, achieves outputs and outcomes to a different degree in different locations. From the key informant interviews, it was estimated that about half of Community Futures Organizations have had very little or no activity related to the EDP and just 1 of the 7 Urban Delivery Agents has had a significant level of EDP specific loan activity. Key informants from the Community Futures Organizations indicated that the original vision of the EDP of providing a loan portfolio that would be self-sustaining has not been achieved for those organizations that have depleted their EDP loan fund.

Key informants also indicated that the reporting and accountability systems currently in place do not support ease of analysis and reporting on the results of loan funds. Many of the delivery agents have largely integrated the EDP loan funds with their core loan portfolios. The success stories of the EDP were viewed as significant in their impact, however relatively few in number by key informants. The incremental impact of the EDP overall is viewed as mainly in the area of pre and post care by providing extra counselling/advisory support, and working with other community organizations to support entrepreneurs with disabilities.

When asked about attribution, or what would have occurred without the EDP, interviewees describe clients who have been successful, and who have benefited greatly from the services provided as a result of the program, primarily in the area of pre and post care. Key informants reported that it is a challenge to attribute success to the provision of the EDP loan fund, as many of the businesses that were funded were not successful, and the loans were not repaid.

The comparison with other initiatives (Table 4–1) revealed the following relating to the EDP and its achievement of objectives:

  • Low awareness of the EDP. Overall, the programs and services that target people with disabilities appear to have a low awareness of the EDP. This would limit the EDP’s impact on the community, and its outreach.
  • Pockets of innovation. Some other programs and services appear to be more innovative in their approaches, for example in the usage of the Internet, or in reaching out to new markets (geographic reach). Examples are innovative websites offered by some of these initiatives which include courses specifically for entrepreneurs.
  • Lack of and inconsistent information, definitions, and program design features. The lack of information makes comparison, and overall evaluation, challenging.
  • Performance targets not in alignment; not met. Performance targets, for example relating to loan activities, may not align with needs or priorities for specific organizations.

 

Number of Clients Served

Within the evaluation period, the EDP provided services in all areas to 15,471 entrepreneurs with disabilities8. Table 5–1 provides a breakdown of the number of clients served by the program as captured in departmental and CF databases.

Table 5–1 Number of Entrepreneurs with Disabilities Served
Rural - CF British Columbia* Alberta Saskatchewan Manitoba Total
2006-07 1,913 152 393 510 2,968
2007-08 1,697 139 723 178 2,737
2008-09 1,105 219 196 832 2,352
2009-10 1,985 231 370 337 2,923
2010-11*** 1,296 289 185 312 2,082
Total 7,996 1,030 1,867 2,169 13,062
Urban British Columbia Alberta Saskatchewan Manitoba** Total
2006-07 77 252 116 - 445
2007-08 191 168 104 - 463
2008-09 104 305 125 193 727
2009-10 103 227 158 316 804
2010-11*** 73 145 122 135 415
Total 548 1097 625 644 2409
Grand Total 8,544 2,127 2,492 2,813 15,471

Source: TEA System of Community Futures (Rural) and departmental project database (Urban)
*Number of entrepreneurs for BC includes numbers from other initiatives. **Data for Manitoba’s Urban delivery Agent was absent in the departmental database for 2006 and 2007. ***Data includes the first three quarters of Fiscal 2010-2011

Number and Value of Loans

Table 5–2 and 5–3 below indicates the number of EDP loans issued over the period covered by the evaluation.

Table 5–2 Number and Value of Loans Issued by the Community Futures Organizations from 2006-2010
Year British Columbia Alberta Saskatchewan Manitoba Number of Loans Value of Loans
2006-07 42 30 5 8 85 $2,880,164
2007-08 39 20 5 17 81 $2,602,594
2008-09 37 28 3 19 87 $2,742,192
2009-10 45 20 5 15 85 $2,799,544
2010-11* 39 20 4 6 69 $2,088,474
Total 202 118 22 65 407 $13,112,988

Source: TEA System of Community Futures Organizations
* Data includes the first three quarters of Fiscal 2010-2011.

Table 5–3 Number of Loans Issued by the Urban Delivery Agents from 2006 to 2010*
Year British Columbia Alberta Saskatchewan Number of Loans Value of Loans
2006-07 33 2 1 36 $479,385.35
2007-08 4 1 1 6 $133,748.82
2008-09 40 12 0 52 $838,245.06
2009-10 46 8 2 56 $679,655.00
2010-11** 16 2 0 18 $82,150.00
Total 139 25 4 168 $2,213,184.23

Source: Departmental project database
*These do not include numbers from Urban Delivery Agent in Manitoba. It was not possible to reconcile and generate the data on an annual basis from the information in the department’s project database.
** Data includes the first three quarters of Fiscal 2010-2011.

Overall, in the period covered by the evaluation, the delivery agents issued a total of 575 loans (excluding the Urban Delivery Agent in Manitoba) from 2006 to 2010. From the Community Futures Organizations database, as at the end of 2010, the net cash on hand for the EDP after all loan commitments in 2010 was estimated to be about $7,165,049. About 41 (45%) of all Community Futures Organizations reported no loan activity in 2009. The cash on hand for the Urban Delivery Agents in British Columbia, Alberta, and Saskatchewan was estimated to be about $281,551 as at March 31 2010. Most of the Urban Delivery Agents reported low or no loan activity in the key informant interviews. Reliable data on loan default rates was not available for the evaluation; however, key informants and focus group participants indicated a high default rate of about 60%. Only 39% of survey respondents expressed a positive opinion that the EDP prevents clients from failing at their business ventures.

 

Impact on Employment

The EDP had jobs created/maintained/expanded as an indicator to help measure the impact of the program on employment. The following table indicates the number of jobs created by the EDP lending within the evaluation period for both urban and rural delivery agents.

Table 5–4 Numbers of Jobs Created by EDP Lending
Year Urban Rural
2006-07 82 364
2007-08 33 171
2008-09 129 179
2009-10 170 265
2010-11* 66 155
Total 480 1,134

Source: Urban Numbers from departmental project database and Consultations. Rural numbers from the Community Futures Organizations TEA System
* Data includes the first three quarters of Fiscal 2010-2011.

Overall, the EDP was estimated to have created 1614 jobs over the period covered by the evaluation. The client survey indicated that the EDP increased the number of people with disabilities who are self employed full time and reduces numbers who are unemployed and looking for work. The EDP also has the ancillary effect of creating additional employment for other individuals. The majority of respondents who started a business reported employing at least one person in addition to themselves, with an average of 3.4 employees. The majority of those who expanded their business were also able to employ at least one additional person, with an average of 4.9 employees.

Impact on Businesses

The urban delivery agents reported on an indicator for the number of business created/expanded within the period from 2007-2010 (Table 5 – 5). This indicator was part of the indicators reported by the rural delivery agents in their database.

Table 5–5 Number of Business Created/Expanded
Year Urban
2007-08 33
2008-09 87
2009-10 132
2010-11* 21
Total 273

Source: Numbers from departmental project database and Consultations.
* Data includes the first three quarters of Fiscal 2010-2011.

The client survey indicated that over 40% of respondents were unemployed or derived their income from other sources prior to accessing the EDP. Following participation in the EDP, roughly half of respondents indicated they either started or expanded on a business which continues to operate.

Survey respondents indicated the EDP played a significant role in allowing them to reach their business goals. 37% of respondents indicated the EDP had a significant impact on their business with a total of 63% assigning a positive value of 4 or higher (on a scale of 1-5) on the level of impact (Table 5 – 6).

Table 5–6 Responses to Question on Impact of EDP on Businesses
  (n=81) Percentage
1 - No impact at all 15 19%
2 3 4%
3 10 12%
4 21 26%
5 - Significant impact 30 37%
Don't know/No response 2 2%

Figure 5-1 shows the description and percentage of impacts on the business of client survey respondents. There were three areas in particular where the EDP had a particularly notable impact on the client:

• Providing moral support and encouragement (34%);
• Increasing opportunities and removing barriers, including market entry (33%); and
• Providing business planning and education (34%).

 

Figure 5-1 Impact of EDP on Client

In this figure, the percentage of respondents indicate the areas the EDP impacted on their lives.

Text Version: Impact of EDP on Clients

Most Significant Impact of EDP

The “pre and post care” services offered by the EDP was viewed by almost all key informants as a very valuable aspect of the EDP supported advisory/training/ business services. This service was described as being very much needed and highly beneficial. Focus group participants supported this and further stated that as a result of this service, a high number of clients that applies for loans and goes through with their applications do receive the loans. The approval rate for loans, as indicated by client survey respondents was quite high at 91%.

Data from the databases indicate that quite a significant amount of time is spent by the delivery agents in “pre and post care” services for clients. The total number of coaching hours recorded by the Urban Delivery Agents from 2007 to the third quarter of 2010-11 was 13,525 hours. The number of events sponsored in support of this “pre and post care” for the Urban Delivery Agents was 494 from 2007 to 2010. The Community Futures Organizations did not report results on these indicators.

As a result of the pre and post care services of the EDP, client survey respondents reported strong positive opinions regarding most aspects of the program. One a scale of 1 to 5 where 1 represents “Strongly Disagree” and 5 indicates “Strongly Agree”, over 50% of respondents assigned a positive value of 4 or higher when asked to rate the following features of the EDP:

  • Letting people know what the program offers (66%);
  • Providing confidence and motivation needed to start a business (58%);
  • Facilitating access to financing (59%);
  • Cooperating with other related programs and services (53%);
  • Providing people with disabilities specialized services not provided by other organizations (50%);
  • Supporting their preference to access business-related services specialized for people with disabilities (70%); and
  • Providing better quality of service relative other individuals and organizations (60%).

Performances (Demonstration of Efficiency and Economy)

Efficiency in Program Delivery

According to key informants, the EDP does not appear to consume significant departmental resources for oversight or program management. The Association model and the Community Futures organizations were seen as efficient; the urban delivery was viewed as less efficient in relation to the total expenditures relative to perceived benefits. Efficiency could potentially be increased by improving coordination between urban and rural models. Some interviewees felt that more proactive program management of the part of departmental staff would strengthen the EDP. Key informants from the delivery agents indicated that the EDP funding fits well with their overall operations and economies of scale and efficiencies were gained.

Key informants mentioned the existence some overlap in programming and services, both on the loan side and on the business services side as a result of the integration of the EDP services with other core services. Therefore, there is an opportunity for consolidation of the EDP with some of their core services. However, key informants strongly supported a separate EDP funding stream as necessary for providing that incremental support and enhanced access to loan funding for entrepreneurs with disabilities.

Program Design

The EDP was designed to be delivered in the rural and urban areas. Key informants indicated that the best feature of the program design was its flexibility in providing services. The most frequently identified challenges of the program design by key informants included:

  • Lack of clarity around expectations; performance management framework; definitions; and the need for more proactive departmental management of the contracts.
  • Inconsistency of delivery from region to region.
  • Overlap in what is developed and delivered, and opportunities for more coordination and sharing of resources (example for coordination in web-based resources).

In comparing the EDP to other initiatives, the program was seen as a small program, with an overall budget of approximately $600,000 for the four provincial associations, and another $1 million for the seven urban delivery agents. The comparative analysis resulted in the following conclusions about the efficiency of the EDP:

  • Limited resources. Other organizations that serve entrepreneurs have access to significant resources compared to the EDP. The EDP essentially enables an enhancement to existing services, by increasing the capacity of those organizations to serve entrepreneurs with disabilities.
  • Centralized funding is efficient. The funding provided to the Provincial Community Futures Associations to support the Community Futures Organizations in their respective provinces, appears to be an efficient usage of funds.
  • A fragmented service delivery network. The service delivery network of the EDP is fragmented. In BC, two different Credit Unions provide EDP-related service in the same community, along with the Community Futures Organizations. An urban delivery agent may serve a large population, but may not have loan funds available compared to a Community Futures Organizations that serves a much smaller population. Some provincial associations have dedicated resources to the EDP while others do not have such resources. Management of EDP funding varied among Urban Delivery Agents; Urban Delivery Agents also managed EDP funding differently than the Provincial Community Futures Associations and the Community Futures Organizations. The following delivery scenarios of EDP services were described by key informants from the Community Development Organizations:
     
    • Provide neither business coaching nor loan funding due to lack of resources.
    • Provide business coaching but not loan funding (funding for business coaching provided by Provincial Community Futures Associations).
    • Provide business coaching and loan funding; and funding for business coaching is provided by the Provincial Community Futures Associations).
    • Provide business coaching, loan funding and additional funding and services, such as training and funding for adaptive equipment, coordinated through the Provincial EDP Coordinator employed by the Provincial Community Futures Associations.
  • The EDP has become highly integrated with related services. Compared to other program/service areas, the EDP does not have a strong identity as an independent program within most of the delivery agents, but rather is most often presented as an enhancement to related programs and services. For example, the British Columbia Credit Unions include the EDP as part of the Loan and Investment Program, and the program cannot be separated from the core activities of the Community Futures Organizations. This integration impacted the reporting of results for the program.

These challenges to the design of the EDP indicate the need for the department to examine the design of the program to improve upon program delivery.

Client Satisfaction with Services

On a scale of 1 to 5 where 1 represents “Very Dissatisfied” and 5 indicates “Very Satisfied”, over 75% of respondents assigned a value of 4 or higher in all service areas except mentoring. While still demonstrating strong levels of satisfaction and earning 61% favourable responses, mentoring exhibited a small opportunity for improvement. Figure 6 – 1 indicates the client satisfaction rate as against services assessed.

Figure 5-2   Client Satisfaction with Services Offered

In this figure, key informants indicate their satisfaction in different EDP services offered to clients.

Text Version: Figure 5-2: Client Satisfaction with Services Offered

Economy of the EDP

The EDP provides services for entrepreneurs with disabilities across the four western provinces for approximately $1.5 million/year. EDP funding includes approximately $500,000 for the provincial CF associations and approximately $1 million for the urban delivery agents, with minimal departmental overhead provided for project management and oversight. This approach was indicated as being economical in the comparative analysis from these perspectives:

  • It capitalizes on the existing departmental network of Community Futures Organizations, enhancing their capacity to serve people with disabilities by way of the support and funding provided to their provincial associations.
  • In British Columbia, it ensures that the department leverages existing programs such as the Loan and Investment Program to deliver the EDP.
  • In Alberta, it enables a strong urban delivery agent to strengthen and tailor its business services to people with disabilities including an active and successful loan portfolio.

The use of operating funds to support EDP activities was generally viewed as an effective use of resources by key informants. Due to the integration of EDP-funded resources and activities with related operational areas, it was difficult to assess the use of the EDP operating funds. To some extent, the EDP funding may be supporting other activities that have synergies with the EDP and its goals, such as funding assistive devices, or promoting employment of people with disabilities.

Recommendations

The following recommendations have been made based on the findings from all lines of evidence utilised in the evaluation.

Recommendation 1: The department should determine the extent and focus of any future renewal of the Entrepreneurs with Disabilities Program.

  • While there exists a need for programs that target entrepreneurs with disabilities in western Canada, evidence from the evaluation suggest that there an ongoing need for business services and a declining need for loan services.
  • Any renewal of the EDP should consider whether the current fragmented service delivery network is still appropriate.

Recommendation 2: If the department plans to renew the Entrepreneurs with Disabilities Program, then the department should develop clear quantitative performance indicators for the program.

The improvements should ensure:

    • Measurement of the non-loan, service support components of the program;
    • A clear link between performance indicators and current departmental Program Activity Architecture outcomes (results);
    • The use of the same indicators across all service delivery agents;
    • That separate EDP performance data is collected and entered into the departmental database; and
    • Consistent definitions of indicators across regions.

Footnotes

[1] This number excludes Urban Delivery Agents in Manitoba who could provide only aggregate data for time periods that did not match those of the evaluation time period.

[2]. Community Futures Organizations in each province come together to form a provincial association.

[3] Urban Delivery Agents in Saskatchewan received $100,000 each

[4] TEA (The Exceptional Assistance) database was developed and used by Community Futures Organizations to report on results related to the Community Futures Program.

[5] 2010 Federal Disability Report – The Government of Canada Annual Report on Disability Issues. Published by The Department of Human Resources and Skills Development of the Government of Canada.

[6] Total number is skewed by some of the Community Futures Organizations reporting numbers from other initiatives as part of EDP clients. As an example, the CF South Fraser, and the CF North Fraser reported significant number of clients served in their provincial Employment Program for People with disabilities (EPPD) as part of EDP clients in the EDP database.

[7] Advantage Canada is an economic plan document published by the federal government. The plan is designed to build a strong Canadian economy and make Canadian quality of life second to none through competitive economic advantages. http://www.fin.gc.ca/ec2006/pdf/plane.pdf

[8] Total number is skewed by some of the Community Futures Organizations reporting numbers from other initiatives as part of EDP clients. As an example, the CF South Fraser, and the CF North Fraser reported significant number of clients served in their provincial Employment Program for People with disabilities (EPPD) as part of EDP clients in the EDP database.

Date modified: