Why measurement must be flexible

The Common Approach worked with 114 social enterprises in Ontario to map their indicators. We gathered 1110 indicators. We identified the indicators that real social enterprises in Ontario are using to refer to SDG-2 on Zero Hunger [Sustainable Development Goals]. Zero hunger is not a great shorthand for SDG-2, it’s really sustainable food systems. It is much more complex than hunger. It has got to do with nutrition, as well as agricultural practices and that kind of thing.
We took indicators from Ontario social enterprises that speak to sustainable food systems and decent work. Some of these indicators, although they’re different, are speaking to the same idea. They are similar-enough indicators that can be aggregated.

The aggregating of similar-enough indicators allows us to not be forced to measure in the same way. We create an approximation. And then we do ‘the collective shrug’. It’s good enough. The idea is that there are lots of different indicators or articulations, or definitions of more common ideas.

Above is a conceptual presentation of what this looks like for SDG8 on decent work. At the very bottom of the diagram, you have examples of different indicators that we pulled from real social enterprises in Ontario. These are short-hand actually, for measured skills: the number of people who acquired a new skill, attended a workshop, attended a training, participated in training, the number of training sessions delivered.

The indicators also talk about certain behaviours. People demonstrating cleanliness, or improved hygiene or punctuality. Different behaviours are being measured. As well as acquiring experience through internships. A whole host of different work-integration social enterprises – working with different populations — have different measures of what they feel is a sign, an indication, that a person is progressing on a path to employment.

We can count all of these measures and do a kind of collective shrug. And this is all a story of a number of people progressing on a path to employment. Then we can count the number of people acquiring good employment.

IRIS [Impact Investment and Reporting Standards] is a menu of indicators recommended for the impact investing community. One of the things IRIS struggled with and even the SDGs struggle with, is defining what it means to have a good job. SDGs call it decent work.

Based on the ILO’s [International Labour Organization] work, the things that constitute decent work are almost the minimum requirement in Canada – that the work not be life-threatening and have reasonable hours and appropriate pay. And in Canada, we have a minimum wage, and we have health and safety standards. And so any legal job in Canada is going to mostly meet the ILO’s condition for decent work.

IRIS has really struggled to define a good job in terms of objective things like the number of hours and amount of pay. But one thing we know from talking to social enterprises is that a good job for someone who has arrived recently in Canada is different from someone who has been in Canada all their life and has a long term cognitive impairment, which in turn is very different from someone who has recently been out of work. You have a very different set of people for whom a good outcome can’t be measured in the number of hours and pay. Some people need flexible work. Some people need full-time work with benefits.

One person from a social enterprise we spoke with said, “a good job is a job that connects your past to your present to your future.” One way we can create a flexible standard is we can allow each organization to define for themselves “What’s a good job for my population?”

And then we can start counting the number of people who were helped to find a good job without a rigid definition of ‘what is a good job’. This is an example of the kinds of thing we can do with a flexible standard.

You can see from the diagram that we have permitted a lot of flexibility in indicators. We are mapping indicators to common ideas that allow us to tell a collective story without being rigid in defining how organizations do their measurement.

When we initially tried to map the 1110 indicators from these 114 social enterprises, only 25% of these indicators could be represented by the SDG global indicator framework. Which meant that 75% of what these enterprises did would be erased or rendered irrelevant or silent or outside the SDG measurement tool. But by using this flexible approach, we were able to get 100% of that work connected to the SDGs. You lose a little bit of rigour because the number you have at the end is composed of all sorts of different measures.

For the flexible standard, we will use technology that will allow people to drill down and see the details. So that you don’t lose the quality of information. But what you’ve gained is an ability to tell an overall story.

This is what financial accounting does. Financial accounting gives organizations a number of places where there’s a choice, or customization of the standards for a particular context. For example, you can measure inventory one way, or you can measure inventory another way. We do a collective shrug. We get it, it’s inventory. It is good enough.

I think one of the reasons why we struggle to have a standard in social impact measurement is we’ve been really fixated on the idea that it has to be measurement based-equivalence, rather than the idea of ‘these are just different ways of getting at the same idea; let’s focus on that idea”. Let’s focus on what it means to have a good job and let definition and measurement be flexible so it can be the most appropriate for that particular organization.

Watch or listen to the presentation below.