Western Economic Diversification Canada
Symbol of the Government of Canada

Common menu bar links

Section 5: Performance (Achievement of Expected Outcomes)

This section presents evidence to answer evaluation questions regarding the success of the department’s investments in economic development projects within the VA, more specifically, in terms of whether the projects realized their outputs, the achievement of immediate and intermediate outcomes, and agreement goals, factors that constrain or facilitate achievement of program results, and the resulting impacts of the program.

Achievement of Objectives

The logic model developed for the evaluation was based on the WDP logic model and the VA Strategic Plan summarizes the activities, outputs, and intended economic impacts of the Agreement.

The major activities of the initiative involved increasing demand for DTES goods and services, strengthening supplier capabilities, and increasing employment opportunities for local residents. The key outputs generated from these activities included the provision of relevant services to businesses and individuals such as business advisory services, employment training, procurement initiatives, and financing. The initiative also promoted community planning and strategies to meet the goals of the VA. The immediate departmental outcomes were to increase the level of investment, total employment, participation of residents in workforce, and broadening suppliers’ goods and services in the DTES area. The intermediate departmental outcomes were: increased community participation; viable and growing community economies; creation of mechanisms for community and government coordination and collaboration; and enhanced capacity of VA communities to implement strategies that promote sustainable development. The final outcome was to create increased economic activity that improves the viability, prosperity and quality of life in Vancouver.

Participants in the focus group recognized that trying to use a normal evaluation framework to prove cause and effect for the projects undertaken within the VA is problematic. The Agreement was seen as a dynamic process that involved many agencies, individuals, and programs. It was a process that affected the outcomes of many initiatives, some of which were part of the agreement and some that were not. Measuring outcomes and impacts was also hampered by the ad hoc nature of many activities, and the wide range of factors that contributed to the outcomes. While the VA was viewed as an appropriate mechanism to deal with the problems in the DTES, the agreement was indicated as needing a longer lifespan to have a major lasting impact. Many of the effects of the VA were seen as non-quantifiable. Some of these non-quantifiable effects are listed below:

  • People involved at the community level stated that the VA changed people’s attitudes and improved the atmosphere among agencies in the DTES. It was mentioned that before the VA, there was a great deal of animosity between various groups in the area. However, by working together on VA projects, these groups improved their attitude towards each other. Groups such as businesses, residents, police, prostitutes, and drug users had a different and more positive view of each other after their experiences in the VA.
  • The VA created a significant amount of spin-off activities in its legacy such as the establishment of events, agencies and community efforts that will continue on after the initiative is over. One example given in the focus group session was the Heart of the City Festival, which has found a way to sustain itself beyond VA support. The Festival was described as a “flag ship activity” that changed many people’s opinion of the neighbourhood and created employment.
  • The VA left a legacy of cooperation. The VA process brought people together who were previously unaware of each other’s efforts and responsibilities. Focus group participants reported being surprised at the extent of other government activity and level of staffing for VA projects. These relationships are likely to continue after their involvement in the VA.

As a result of these factors, key informants mentioned that it will be a challenge to prove a direct correlation between the VA and positive impacts experienced through the DTES. WD’s investments in the VA was viewed as contributing to positive impacts in the DTES such as a reduced property crime rate by 14% between 2000 and 2005, an increased employment income by about 32% for men and 29% for women from 2004 to 2007, a reduced death rate due to HIV/AIDS and suicides, and an improved collaboration among all levels of government.

Factors that Facilitated/Impeded Achievement of Desired Outcomes

The constraints mentioned most often by the key informants dealt mainly with the nature of the governance. A major constraint to success was the lack of a horizontal accountability mechanism that outlined roles and responsibilities for the partners, and provided a strategic framework to measure results. This constraint was also observed in the November 2005 OAG Report. Other constraints to achieving VA results mentioned by key informants from the focus group, key informant interviews, and comparative analysis included:

  • Inconsistent political support (elections changed priorities and support levels);
  • Bureaucratic nature of the VA/decision layers/delays in approvals;
  • Personnel turnover which resulted in removing VA champions;
  • Territoriality/individual mandates among the participating agencies;
  • Low involvement of the community in the decision-making process;
  • The scope of the Agreement which was mentioned as too large; and
  • The joint-decision making process which was at times challenging and slow.

The success of the agreement was attributed mainly to the collaboration and the coordination within the VA and the dedication of key people in the various government departments. Factors that facilitated the achievement of program results mentioned by key informants from the focus group, and key informant interviews included:

  • The high level of commitment of the people involved/champions;
  • The resources dedicated to the VA effort;
  • A focus on innovative programming/risk taking;
  • Initial political support;
  • The relationships that were developed; and
  • The high level/seniority of the people involved.

In the comparative analysis, key informants from the other UDAs supported the factors above as contributing to the achievement of results in their jurisdictions. Other factors mentioned by these key informants included:

  • The building of community capacity to effectively address socio-economic problems; and
  • The quality and number of information sharing opportunities and the ability to bring key decision-makers together to identify and work on priorities.

Outputs Generated by the Department’s Projects under the VA

Most of the department’s projects within the VA involved creating and assisting small businesses in the DTES, developing courses and training local residents for employment. The key program activities were the development and funding of projects that increase demand for goods and services, support local supplier capabilities, and increase employment for local residents. These activities led to outputs in: business advisory services, employment services training, loans and grants, and procurement services; community viability and increased participation in the local economy; adjustment to economic realities; and community planning and strategy.

All agencies and organizations with projects funded by the department under the VA believe their projects were successful to very successful in achieving their outputs. Key informants from project partner organizations rated the success of their projects very high on the success measure. The average success rating was 4.4 on a scale of 1 to 5 (Table 5 – 1).

Text Version (Link 1): Table 5-1: How Successful do you think the project has been in achieving its objective?

Table 5-1

When asked for major accomplishments of their project, project partners mentioned a range of positive outcomes generated by their projects including:

  • Leveraged departmental funding with funding from other sources;
  • Many best practices;
  • Planning for future projects;
  • Innovative programming;
  • Increased capacity for future endeavours;
  • Long lasting partnerships with other agencies; and
  • Increased client satisfaction.

Case study respondents indicated that their projects were either successful to very successful in generating their outputs. Some of the outputs mentioned as being generated include loan portfolios, assistance to business, training programs and employment services.

Key informants provided many examples of how the department’s projects under the VA performed against its range of outputs. Table 5-2 illustrates some of the examples of how the projects performed against the type of output.

 

Table 5 – 2: Examples of VA Performance against its Outputs
TYPE OF OUTPUT VA PERFORMANCE
Result in business advisory services, employment services, training, loans and grants, and procurement contracts

Building Opportunities with Business (BOB) Organization

  • BOB Loan program
  • Social Purchasing Portal
  • Business plans for social enterprises
  • BOB employment/training programs
  • BOB business mentoring
  • BOB Job Recruitment Service

Eastside Movement for Business and Economic Renewal Society (EMBERS)

  • EMBERS training
  • Expansion of Business Development Centre
  • EMBERS Workshops/coaching for self employment
  • EMBERS micro-business and self-employment instruction

BC Institute of Technology (BCIT)

  • BCIT Business Education Development
  • BCIT DTES Business Development Program
Create community viability and increased participation in the local economy
  • Creation of Women’s economic institute/business incubator
  • DTES Business Database
  • Community Arts Calendar
  • Social housing initiatives
  • Safe Injection site
  • Community Court
Help the VA region adjust to economic realities
  • Business leaders forum
  • Awareness raising for business executives
  • Community Benefits Agreement
Promote community planning and strategy
  • Chinatown Marketing Strategy
  • Marketing and planning for Native Youth Centre
  • Chinatown Tourism Development Strategy
  • Strathcona Marketing Strategy
  • Strathcona Branding Strategy and Marketing Campaign
  • Capacity and business plan for emergency food program
  • Business Retention and Expansion Strategy
  • Arts and Culture Strategic Framework
  • Children of the Street Society Business Planning

Some factors were mentioned by key-informants as factors that impeded the achievement of projects outputs. These factors include:

  • Delays in accessing funding which slowed momentum and reduced the effectiveness of products;
  • Lack of sufficient promotion of service or activity reduced the level of activity;
  • Lack of sustained capacity building; and
  • Lack of continued funding which limited the impacts.

Achievement of Performance Indicators

Most projects began before the establishment mandatory reporting into WD’s Project Gateway database for projects that ended after 2007. These projects were not reported in Project Gateway. The database could therefore not be used to assess whether targets for indicators have been met or not. A review of the database noted that some departmental performance indicators linked to the outputs in the logic model. Table 5 – 2 below summarises the results from these indicators.

Table 5 – 2 WD PAA Indicators for the Agreements
PAA Indicator Name Number of Projects Using as Indicator Target Reported Outcome in Database

Number of instances of increased community stability

5

16

Not available

Number of businesses created/maintained/expanded

5

113

356

Number of jobs created

2

151

Not available

Number of people trained

6

750

469

Number of training courses developed

3

11

4

Analysis of results from the file review, case studies and key informant interviews indicated that results exceeded targets set for each of the indicators.

Outcomes identified in the evaluation were based on the performance measurement framework of the WDP. These outcomes are based on a broad framework and present challenges in the ability of the evaluation to directly attribute project results to outcomes. In order to improve on its data collection, the department should implement strategies that would result in the development of performance measurement frameworks for individualised initiatives that would support reporting to the database.

Contribution towards the Department’s Immediate Outcomes

Ratings indicate the department’s economic development projects within the VA had some positive impact on economic revitalization. Key informants were asked to rate impacts in a variety of economic revitalization categories. The ratings were above average for all economic revitalization activities (Table 5-3).

Text Version (Link 2): Table 5-3: Impact ratings related to various types of economic revitalization activities in the region

Table 5-3

Project partners rated the achievement of immediate outcomes higher than other groups of key informants (overall rating of 4.1 on a scale of 1 to 5). This is likely due to the fact that project partners were able to measure the impacts of their projects on the micro-level as compared to other groups of key informants who will be measuring the aggregate impacts of all projects. Case study respondents who are project partners indicated that their projects were successful to very successful in realising their immediate outcomes. The overall rating was 4.6 on a scale of 1 to 5.

From the file review, case studies and key informant interviews, several VA partners were profiled as having achieved very significant results in each of these economic revitalisation outcomes. Two of such partners are profiled below.

Building Opportunities with Business Inner City Society (BOB): BOB is a non-profit organization that supports local business development and revitalization without displacement in the DTES. The goals of BOB are to: strength the inner city’s community capacity; identify and capitalise on untapped business opportunities in the inner city; improve employment opportunities for residents; and increase investment and visitors to the DTES. BOB targets unemployed and underemployed inner city residents. The VA funded the establishment of BOB and has provided over $7 million from 2002-2010 for BOB activities. The funding from the VA contributed to BOB achieving the results below:

  • 320 people found employment through BOB programs since 2006, with the goal to connect a total of 410 people with employment by 2010.
  • $26.5M in purchases from inner city suppliers or supporters since 2007, well exceeding the goal of $16.9M.
  • 161 businesses assisted through BOBs Business Development and HR Services Programs since 2007 with the goal to assist 170 businesses by 2010.
  • The establishment of the Community Benefits Agreement, a partnership between the City of Vancouver, BOB and the Millennium Development Corporation to ensure inner city residents and businesses benefit economically from the development of the 2010 Olympic Athletes Village. An evaluation of this agreement indicated that it resulted in $42 million in procurement from the inner city and employment for 124 individuals.
  • The establishment of the Social Purchasing Portal to facilitate the procurement of services and goods by providing a list of participating suppliers in the DTES.

Eastside Movement for Business and Economic Renewal Society (EMBERS): EMBERS is a non-profit agency that facilitates community-based business development by helping individuals and groups start small businesses and social enterprises. EMBERS targets low-income, under-served entrepreneurs who cannot afford to start a business; part-time/causal workers starting business; individuals with disabilities; individuals recovering from addictions; new immigrants; long-term unemployed single parents; and existing microenterprises. From 2002-2009, the VA provided over $600,000 in funding for four projects with EMBERS which included: entrepreneurship training; business management and incubation; a business development support project; and an expansion of EMBERS’ microenterprise development program. VA funding contributed to the EMBERS results outlined below.

Description Projected Outputs Actual Outputs

# of participants that received training

90-100

161

# of Training workshops presented

# of new training sessions developed

13 sessions (96 workshops)

3

15 sessions (116 workshops)

4

Business plans completed

20 – 30

25

Businesses started/maintained/expanded

 

20-25

89

 

New businesses launched or ready to launch

16-25

29

Business networking workshops / events

12

14

 

The activities of BOB and EMBERS were seen by most key informants as having real impacts on the lives of people involved in the agreement. Projects from these partners which include job training, entrepreneurial support, the Community Benefit Agreement and the Social Purchasing Portal were all seen as highly successful.

Contribution towards the Department’s Intermediate Outcomes

As noted earlier, departmental funding for the projects under the VA was under the authority of the WDP. The performance measurement framework of the WDP provided outcomes in: increased community participation; viable and growing community economies; creation of mechanisms for community and government coordination and collaboration; and enhanced capacity of VA communities to implement strategies that promote sustainable development. These outcomes contribute to the long-term outcome of improving the quality of life of residents in Vancouver. The contribution of results of projects towards these intermediate outcomes was assessed as part of the evaluation.

Most key informants believed that the department’s projects within the VA have made progress towards achievement of the intermediate outcomes. Key informants were asked to rate a number of outcomes. The overall response rate was 3.2 out of a scale of 1 to 5 with each intermediate outcome scoring at least a 3 out of 5 (Table 5-4).

Text Version (Link 3): Table 5-4: success ratings regarding the achievement of various intended outcomes

Table 5-4

Note: Project partners were not included in the question regarding building viable and growing community economies

The key informants indicated their success ratings were somewhat low because of the challenge to prove direct correlation between the VA and its outcomes due to factors as outlined in section 5.1 and others such as many funded projects being on a small scale level. It was also mentioned that the VA took many other initiatives under its wing when it was established which, made it difficult to distinguish the impacts of these initiatives from the VA.

Case study respondents indicated that their projects were successful to very successful in contributing to their intermediate outcomes. The overall rating was 4.3 on a scale of 1 to 5. The projects were said to have contributed greatly to increased community participation (rating of 4.3), and the creation of mechanisms for community and government coordination and collaboration (rating of 4.8).

Key informants mentioned several reasons as examples of success or lack of success for the VA in contributing to their intermediate outcomes. Some of these reasons include:

Reasons for success of the projects

- Increasing community participation

  • Establishment of community forums as a result of VA projects.
  • The VA working closely with other social groups in the DTES.
  • Many projects employing a large number of volunteers from the community.

- Building viable and growing community economies

  • The work of BOB and EMBERS seen as contributing to Community Economic Development.

- Creating mechanisms for community and government coordination and collaboration

  • The committee structure and the task forces of the VA brought people together who never would have collaborated otherwise.
  • Funded VA projects brought many people and agencies together.

- Enhancing the capacity of VA communities to implement strategies that promote sustainable development

  • The training and jobs that people received through the VA projects had a sustainable impact.
  • An indication that some of the new businesses established would be sustainable over time.
  • The VA adding capacity to business organizations to serve their membership more effectively.

Reasons for lack of success in the projects:

- Increasing community participation

  • While there was considerable awareness of the VA among community organizations, there was confusion as to whether it was a funding or coordinating body.
  • The lack of insufficient community consultation due to the community not having a direct role in decision-making.

- Building viable and growing community economies

  • Hard to determine whether VA was the cause of some of the improvements as other factors also influence this outcome.
  • Much of the economy of the DTES being out of the control of the VA.

- Creating mechanisms for community and government coordination and collaboration

  • While there is a legacy of coordination among those who participated, there has been no formal structure created that will outlast the VA.

Contribution towards Other Outcomes in the VA Strategic Plan

The VA Strategic Plan provided for other goals in housing, public safety and health. Departmental funding for VA projects was targeted in the economic revitalisation priority area. Most key informants indicated that results from their projects indirectly contributed to results in the other goals. Table 5-5 indicates the response of key informants.

Text Version (Link 4): Table 5-5: Success ratings regarding the achievement of the VA's other goals

Table 5-5

Major Departmental Outcome

The VA, with its horizontal management structure focused on revitalization, was recognized by all key informants as a groundbreaking strategy for the three governments involved. Federal representatives participating in the Vancouver Agreement described it as a ‘living experiment in governance.’

Key informants indicated that the VA may not have succeeded in establishing a permanent formal structure for government cooperation but did leave a legacy of cooperation between individuals and groups that didn’t exist previously. Relationships were built between individuals in different agencies and government departments that continue to support successful collaboration. Case study respondents and focus group participants indicated the creation of mechanisms for community and government collaboration and coordination were the major outcome of the VA.

The collaborative approach of the VA was widely acclaimed as a leading example of successful partnership working in urban governance. The VA was recognized for its innovative approach by a number of international agencies, resulting in three major awards:

  • The Institute of Public Administration of Canada’s Award for Innovative Management for Horizontal Collaboration in 2004;
  • The United Nation’s Public Service Award for “improving transparency, accountability and responsiveness in public service in 2005; and
  • The Association of Professional Executives of the Public Service of Canada Partnership Award in 2005.

Unintended Impacts

56% of the key informants reported unintended results generated by the VA. The unintended positive impacts that were noticed by the key informants included:

  • The extent of attitude change by government people involved in the project;
  • The close relationships that have developed between government staff;
  • The level of attention it brought to the problems of the DTES;
  • The participation of the private sector in a significant way; and
  • Groups that seemed unlikely to be able to work together were able to (e.g. sex trade workers, police, and private sector).

Examples of unintended negative impacts noticed by the key informants included:

  • The unrealistic expectations of people regarding the extent and speed of changes; Negative media attention regarding extensive funding and a lack of results; and
  • The shift from cooperation to competition among the agencies when the funding program began.