Pathfinder Pilot learnings

Reflections from the first year working with the Network for Optimizing Food Impact

The Network for Optimizing Food Impact, co-convened by 10C Shared Space and Colleaga, was announced as the first network to join the Pathfinder Pilot in late October 2022. The social purpose organizations (SPOs) in this network are committed to bringing more food to communities in need and strengthening local food systems through community-driven innovation.

In the first year of the Pathfinder Pilot, we set out to support all the network members to adopt the Common Foundations and Common Impact Data Standard. The Common Foundations are adopted by confirming all five essential practices of measurement are in place. The Common Impact Data Standard is adopted by choosing (and using!) software aligned to the Common Impact Data Standard to track indicators, outcomes, and impact data.

Common Approach conducted an interview with each network member at the outset to understand their starting point with respect to their measurement practice and software use. A focus group was held to capture the network members’ history of working with each other. We also held regular check-in meetings with the network coordinators and met with network members as needed to support their efforts and collect learnings. The network coordinators have provided hands-on support to all of the network members throughout the project so far.

Over the course of the first year, 10C and Colleaga, the network conveners, provided a comprehensive set of strategy and evaluation workshops that were open to all network members. These were designed to help those network members who did not yet meet the Common Foundations to fill gaps in their impact measurement practices. Thank you, Emily, at 10C and Dan, at Colleaga!

The first year also focused on software adoption. Of the eighteen social purpose organizations (SPOs) that started with the network, fourteen have successfully onboarded to software (7 with Riddl, 7 with Sametrica), two have substantially completed bringing their own in-house software into alignment with the data standard, and two have put their participation on hold due to competing priorities.

“Good enough for what” and building measurement capacity

In our baseline data collection, we learned that four of the 18 network members met or almost met the Common Foundations; nine were likely to need moderate support to meet them, and the remaining five were likely to need significant support.

The Common Foundations articulate a minimum standard of impact measurement; the Pathfinder Pilot required only that network members meet this minimum. However, the network coordinators felt they could achieve better results by offering a cohort-based approach for improving measurement strategies to all network members, including the members that already met the minimum. In the spirit of being community-driven, we followed their lead.

We planned for software selection to take two weeks; it took more than two months. In their application to the Pathfinder Pilot, the network members described themselves as comfortable with digital technology and willing to use new software. In the initial self-assessment, 17 of the 18 starting network members responded “yes” to the statement “we have a place to start and manage the information we collect”, and 13 of 18 said “yes” to the statement “Our data is stored in ways that are useful and accessible to those who are learning about and improving the organization.”

As the software selection and adoption process unfolded, we learned that these “yes” answers were not good predictors of whether or not alignment to the Data Standard would feel easy. We learned early on that half of the members had structured data that already included the required level of detail about indicators, outputs, and impact themes but that they collected data in software that was not suitable for alignment with the Data Standard directly (e.g. Hubspot, Mailchimp, or SM Apply). Others had data that was not yet structured in a way that it could readily be entered into an aligned software (e.g. “raw data” like survey responses not yet aggregated into reports on specific indicators).

As a result, the Network for Optimizing Food Impact shifted from a plan where each member would use their own software to a model similar to how a bookkeeper uses specialized accounting software on behalf of their client. The network conveners purchased licenses for Sametrica and Riddl and shared seats in these licenses to network members. They configured the software and entered impact data on their members’ behalf. We are curious to see how this approach works in comparison to other Pathfinder Pilot networks where participants use measurement software directly.

To our surprise, the easiest software adoptions were those with in-house custom softwares. Two members of the Pilot cohort (Be One to Give and A Friendlier Company) already had their own in-house software or databases in place prior to joining the Pathfinder Pilot. They have both successfully aligned their in-house software and have exported JSON-LD files with their impact data by following the Common Impact Data Standard documentation for software developers with support from Common Approach’s Data Standards team.

At the end of the first year, all 16 network members are using software and have exported a JSON-LD report containing the indicators, outcomes, and impact themes that they are tracking. Two organizations export their reports directly, and fourteen export reports from the shared software managed by the network coordinators.

A preliminary review by the network coordinators suggests that 11 of the 14 network members using the shared licenses are now managing their own structured data in something like a database, three are still managing spreadsheet-based and/or manual systems of data collection, and two are in process of evaluating software options. This relatively small shift—from 8 to 11—tells us how challenging software adoption can be.

Narrative and qualitative data

We learned from network members the importance they place on narrative and qualitative data to convey the impact of their work—and that it’s challenging to track or show this qualitative data in many impact measurement software platforms.

Nearly all the members of the Network for Optimizing Food Impact can tell clear and compelling stories about why their work is impactful and how it contributes to overall goals related to food security or circular economy. Fanjoy Culinary + Wellness Centre, Guelph Neighbourhood Support Coalition, OASIS Food Hub, Shelldale Centre and Farm Park are just some examples of network members that are front-line service providers in the Guelph and/or the Greater Toronto Area. They collect a significant amount of information from interviews, conversations, and stories with community members and partners that they serve. This rich qualitative data is very valuable for telling the story of their impact, but it can be challenging to summarize this information using the type of numerical indicators supported by many impact measurement software platforms.

As they prepared their theories of change, indicators, and outcomes for entry into the Logic Model Builder module of their chosen software, one of the network coordinators reflected that it was hard to see the “why” of an SPO’s work when it was broken down into the discrete elements of the logic model. This is a potentially unwelcome side effect of preparing their data for input into software. We observed that impact measurement software does not necessarily help users tell a more powerful or compelling story, particularly if it requires them to reduce stories to numbers. The data loses the nuance and emotional resonance of the story behind the work.

We also learned that for some SPOs, their data collection is not only narrative but also visual. This was especially true for Winterhill Farm & Garden which records their impact by taking photographs of their land from time to time. Their reasoning: they don’t need to use soil chemistry tests or other quantitative measures which are more time or resource-intensive. They regularly spend time outside observing the changes to the land. Photographic records is sufficient for their needs. The Data Standard does not currently have the means to record data of this type. This is an area for further research and development for the Data Standard.

The entity question: nesting dolls

Two network members (Colleaga and Ontario Co-operative Association) are themselves networks. They found it challenging to manage the dual impact narratives of the network’s aggregate impact, and of their own impact as network hub or facilitator. This is the challenge that Common Approach’s standards are intended to solve. However, the software platforms that are aligned with the Common Impact Data Standard are not yet ready to support network-level analysis. This is something Common Approach is still developing in partnership with the Centre for Social Services Engineering at the University of Toronto. The existing software platforms could not readily support these networks to capture the impact of a network.

What’s to come

To sum up our key takeaways from the past year:

1. While many network members had measurement practices that were “good enough” at the start, the opportunity to achieve more than the minimum required by the Common Foundations was preferred by the network. While it improved results, this took more time than expected.
2. We learned that our intake process didn’t predict what network members would need to adopt the Data Standard easily, and that software adoption can be complicated and difficult. So, we “learned by doing” and mitigated some of the risks of adopting new software for these SPOs with an “intermediary” approach.
3. We learned from network members how important narrative and qualitative data are to conveying the impact of their work—and that it’s challenging to track or show these with the software tools available to SPOs. This is a gap that needs to be addressed.

One year in, all active members of the Network for Optimizing Food Impact have chosen software and are recording their impact data. This sets us up to get started on sharing and aggregating data, which we believe will lead to more insights into how the Common Approach Standards create opportunities for flexible collaboration among SPOs using dissimilar indicators.

We look forward to continuing to learn with and from this network as the Pathfinder Pilot continues!

Published January 15, 2024

More like this