Western Economic Diversification Canada
Symbol of the Government of Canada

Common menu bar links

Evaluation Approach, Design and Methodology

The logic model underlying this theory-based evaluation was developed by program managers as part of the program’s Performance Measurement Strategy. The evaluation effort, in relation to both design and methods, was calibrated to reflect the evaluation’s low risk status. A non-experimental research design with multiple lines of evidence were considered appropriate and sufficient to meet the study objectives. The lines of evidence and amount of data were reduced as much as possible without compromising data reliability and sufficiency. A consulting firm contracted by Western Economic Diversification (WD) gathered data from key informant interviews, surveys and focus groups; the evaluation unit analysed and integrated findings from all lines of evidence into a final report. To maximize the objectivity and relevance of the conclusions, the evaluation was conducted in consultation with senior program staff.

Evaluation Study Activities

Preliminary Consultations

Preliminary consultations were conducted with regional departmental staff and the Chief Executive Officers of each Women’s Enterprise Initiative (WEI) Organization. The purpose of the preliminary consultations was to discuss the evaluation methodology and request comprehensive lists of clients and key informant interviewees. The evaluation framework was reviewed by senior program staff and senior department management. Through these consultations, preliminary evaluation information was obtained.

Documents and Literature Review

Four groups of documents were reviewed as part of the evaluation:

  • General Background documentation (e.g., Treasury Board Submissions, the program Terms and Conditions, the program Performance Measurement Strategy, websites, documents that describe the program’s rationale, history and theory);
     
  • Departmental reports;4
     
  • Program reports and files (e.g., departmental databases and website, regional files, regional quarterly status reports, performance reports); and
     
  • Literature on needs and best practices related to women entrepreneurs.

File Review

WEI Organizations

The evaluation reviewed all information contained in the department’s databases (Project Gateway and the GX financial system) and all paper files on the WEI organizations between April 2008 and the time the file review was completed in June 2012. The Performance Measurement Strategy listed two sub-programs (formerly called sub-activities): 1) improve business productivity and 2) access to capital.

The WEI Organizations also received additional project-based funding to undertake activities (or projects) outside those covered under their operating funding. Four projects were approved between April 1, 2008, and June 2012, accounting for a total of $700,550 in departmental funding. Three of the projects, all approved for funding in April 2008, addressed the “Improve Business Productivity” sub-program and are now complete. The three complete projects included: 1) a one-day educational initiative targeting growth of women-owned businesses; 2) a “women’s business blaster” conference; and 3) the publication of “new pioneers” profile of 100 British Columbia women entrepreneurs. The fourth project, approved in March 2010, is still underway at the time of this evaluation and delivers a diversity certification and global supply chain access program for women entrepreneurs.

Comparative Analysis

The comparative analysis consisted of a literature review of similar programs and best practices in support of female entrepreneurs and input from formal evaluations or impact assessments. Specifically, the following sources contributed to the discussion of best practices and lessons learned:

  • Best Practices for Women Entrepreneurs in Canada:A sample of Canadian programs designed to support the establishment and growth of women-owned businesses in Canada”. The sample included eleven programs or agencies.5
     
  • Best Practices in Supporting Women’s Entrepreneurship in the United States: A Compendium of Public and Private Sector Organizations and Initiatives:Profiles 24 selected organizations or initiatives that provide outstanding support for women-owned businesses and that have a wide scope throughout the United states”.6
     
  • Good Practices in the Promotion of Female Entrepreneurship:Provides an overview of specific actions and support measures adopted by national administrations in the Member States of the European Union and the European Free Trade Association/European Economic Area countries in order to promote female entrepreneurship, particularly in the fields of start-up, funding, training, mentoring, information, advice and consultancy, and networking”. The overview consisted of 132 specific actions and support measures adopted by the European Union and the European Free Trade Association/European Economic Area.7
     
  • Evaluation of the Atlantic Canada Opportunities Agency’s Entrepreneurship and Business Skills Development Program Sub-activity: This evaluation included several initiatives that support business skills development including the Women in Business Initiative.
  • Impact Assesment of the Women’s Enterprise Initiative (2008).

Key Informant Interviews

The consultants developed and pre-tested the questionnaires and then conducted individual key informant interviews by telephone. Most key informants were selected based on their familiarity and level of involvement with the program. The consultants completed 28 key informant interviews including:

  • four departmental staff and management (three regional managers and one manager from headquarters). Two of the participants had been involved in the WEI for almost 10 years. All eight departmental managers who were most involved and familiar with the program were contacted for interview;
     
  • nine staff and board members of WEI organizations. Interviewees included the Chief Executive Officer and at least one randomly chosen board member from each region. All participants had been involved in the WEI for at least five years;
     
  • nine representatives from the Western Canada Business Service Network. The participants were from all four regions and affiliated with the Canada Business Service Centres (three participants), the Community Futures Development Corporations (four participants), or the Francophone Economic Development Organizations (two participants). Interviewees were familiar with the WEI through working arrangements such as partnerships, shared space or referral of clients;
     
  • two representatives of other government departments. One representative was from another federal department and one was provincial. Both participants were very familiar with the WEI.
     
  • two funding partners. Both partners were chosen based on their familiarity with the program and involvement with the program; and
     
  • two other representatives/experts in women’s entrepreneurship. Both were identified by departmental staff as being knowledgeable about the program and issues faced by women entrepreneurs in Canada.

Return to the top of this pagetop of page

Client Survey

Complete lists of names and contact information for clients receiving advisory or training services and loans were obtained from the WEI organizations. Although the list contained a total of 17,403 clients, contact information was inaccurate in some cases. The consultants successfully contacted 584 of the 649 loan clients on the list; 124 (21 percent) loan clients completed a survey. Of 2,000 randomly selected non-loan clients (500 from each province), 1,651 were contacted and 145 (8 percent) completed a survey. In total, 269 (12 percent) clients completed a survey; by province, response rates were 7 percent in Alberta, 11 percent in Saskatchewan, 12 percent in Manitoba and 16 percent in British Columbia. The questionnaire was pre-tested and then posted on a dedicated website. The questionnaire was also available for completion by email, fax or mail. To increase response rates, nonrespondents were followed up a minimum of three times by email and telephone; greater emphasis was placed on following up with loan clients because because loans tend to increase economic activity more than other services such as training. We excluded clients making information requests because these were usually one-time contacts that lacked the depth of experience with WEI organizations necessary for an informed response.

The sample also included 22 clients representing businesses that were not approved for loans. Of businesses not approved for loans, eight responded to an online survey and 14 were interviewed by telephone. Nine businesses were from British Columbia, five were from Saskatchewan and four were from each of Alberta and Manitoba. The response rate for businesses that were not approved was 15 percent.

Focus Groups

The consultants conducted one focus group in each of the four regions in April 2013. The participants were selected based on their knowledge and involvement in the program and familiarity with respect to the needs of small and medium-sized enterprises in the region. In total, 39 representatives participated in the four focus groups including four departmental representatives, 16 representatives of WEI organizations (staff and board members), and 19 representatives of regional partners and other organizations. Each focus group contained between eight and 11 participants.

A consultant presented the field research findings at the focus groups and then facilitated group discussions. The objectives of the focus groups were to review and validate the preliminary findings.

Limitations of the Methodology

Case Studies

Case studies are nonprobability samples of projects chosen for indepth study for a particular purpose. This evaluation chose to forego case studies in favor of surveying a larger group of clients and conducting focus groups.

Key Informant Interviews

The responses of some interviewees may be biased. To minimize bias, we used skilled interviewers that: 1) communicated the purpose of the evaluation, its design and methodology, and strict confidentiality of responses clearly to participants; 2) conducted interviews by telephone; and 3) asked respondents to provide a rationale for their ratings including a description of specific activities. The findings of the interviews were further validated in focus groups. Also, half of the departmental interviewees were new to managing regional files and unfamiliar with program operations because of recent departmental changes to the management of the WEI. Departmental staff were therefore asked to speak more generally about the impact and anticipated challenges facing the program in their region. Finally, although most key informants were able to name other organizations and programs similar to the WEI, some had difficulties commenting on differences or similarities between the programs due to the lack of familiarity with particular services provided by these other organizations. In this instance, the interviewers reminded participants of the overall objectives of the WEI and other regional programs.

Proponent Interviews and Surveys

The completion rate for the survey varied across questions. The financial questions, particularly revenue questions (e.g., revenues generated by the business in the year before receiving assistance, revenues last year, or revenues in the current year) had the lowest response because clients hesitated to share financial data (particularly online) or could not recall financial data for past years. Of the 107 loan clients with currently active businesses, 74 provided full financial data (e.g., revenues, employment, wages and exports). Of the 67 non-loan clients with currently active businesses, 50 provided full financial data. There is also the potential for non-response error: clients who are currently operating, who received more significant services (e.g., larger loans) or who were more satisfied with the services may be more likely to respond to the survey. The consultants managed this challenge by following up with non-respondents including responses of those who were not approved for loans under the program. The loan clients who provided financial data are generally representative of the overall population of loan clients in terms of loan values; for example, these clients received an average of $61,378 in loans which is similar to the average loan amount of $56,498 provided by the four organizations over the past year. Furthermore, the financial data collected on loan clients is comparable to that collected during the 2008 impact assessment of the WEI, which included a larger survey of clients.

Focus Groups

The focus groups are used to validate and interpret evaluation findings. The focus group discussion reflects the opinion of participants and may not be representative of all those involved in the program.

Efficiency and Economy

Detailed costing information required to undertake a more comprehensive analysis of efficiency and economy was unavailable. For example, the The department provides the WEI organizations with a block of funding to deliver services and loans; cost breakdowns by category of activity (service or loan) is unavailable. As a result, some of the observations are estimated or opinion-based.

Attribution

Measuring the program’s long term impacts on strengthening small and medium-sized enterprises or promoting greater gender equality is challenging because there is a myriad of factors working together towards these outcomes. This evaluation uses contribution-focused analysis to estimate the program’s contribution to its long term outcomes such as developing and diversifying the western Canadian economy.

 


[4] Several reports informed this evaluation including: 1) Evaluation Study of the Women's Enterprise Initiative. Jim August Consultants/Management Exchange, Inc. (1998); 2) An Analysis of the Impact of the Women's Enterprise Initiative. Equinox Management Consultants Ltd. (2002); 3) Evaluation of the Women's Enterprise Initiative. Goss Gilroy Inc. (2004); 4) Impact Assessment of the Women Enterprise Initiative (WEI). Ference Weicker & Company Ltd. (2008); 5) Audit WEI report. KPMG (2008).

[5] Foundation of Canadian Women Entrepreneurs, "Best Practices for Women Entrepreneurs in Canada", May 2004.

[6] The National Women's Business Council, "Best Practices in Supporting Women's Entrepreneurship in the United States: A Compendium of Public and Private Sector Organizations and Initiatives", 2004.

[7] Austrian Institute for Small Business Research, "Good Practices in the Promotion of Female Entrepreneurship". 2002.