by Gianni Zappalà & Mark Lyons, CSI BP No.6, 2009. Summary by Gianni Zappalà.
There is a growing interest in the measurement of social impact. In some countries, there are moves towards making the use of some form of social impact measurement framework or model compulsory for those Third Sector organisations that receive government funding. Three such social impact measurement approaches are gaining traction in Australia:
- Social Accounting and Audit (SAA);
- Logic Models such as LogFrame; and
- Social Return on Investment (SROI).
These frameworks undoubtedly have benefits but are equally laden with costs such as the resources, skills and time required to undertake them. Before considering making any one approach compulsory, the role of policy should first be to raise the profile, skills and benefits of using such frameworks. This paper contributes to this issue by examining each of the three major approaches in turn.
1. SAA originated in the 1970s as a way to compensate for the focus of traditional financial accounting on shareholders to the exclusion of a wider range of stakeholders and as a way to document and ‘account for’ the social impact that organisations have. Social accounting has been defined as the ‘systematic analysis of the effects of an organisation on its communities of interest or stakeholders, with stakeholder input as part of the data that are analysed for the accounting statement’. A key advantage (and attraction) of SAA is that it enables organisations to build on existing information and documents which they may keep for monitoring, reporting and evaluation purposes, but also place this information within a broader process and framework. It is a way an organisation can prove its value and improve its performance. In this way SAA is different to an external evaluation as it is the organisations themselves who identify their values, their social, environmental and economic objectives and then report the extent to which they are meeting them based on stakeholder views.
2. LogFrame also emerged in the 1970s as a response to the shortcomings of many program evaluations that were being conducted. A key problem with evaluation was (and in many cases still is) that it is seen as an ‘end of pipe’ task, something that is done at the conclusion of a project or program. This led to many large-scale and well funded programs going off-course and not achieving their desired goals and objectives. The focus of program assessment tended to be on ‘outputs’ rather than ‘outcomes’ and evaluation was not built into the project design process. The advantage and attraction of LogFrame is that it provides a framework which enables organisations to embed evaluation and performance assessment into the design and life cycle process of the program. LogFrame is a systematic way of identifying the elements of a project and the linkages between them to provide a logical, concise and objective analysis of the project design. It also provides a way of setting out the design elements of a project or program that clearly articulates to all stakeholders a program’s causal logic or theory of change, as well as how a program’s activities, output, objectives and goals will be measured (the indicators of success), where and how those indicators will be obtained, and the critical assumptions that must hold for the program to achieve its longer term impact. Logic models such as LogFrame can be seen as complementary to frameworks such as SAA and SROI. Like SAA, LogFrame is a framework to assist in thinking about, collecting and presenting information about a project or program. Also like SAA, LogFrame does not prescribe a specific set or type of indicator. Indicators may be either qualitative or quantitative, although there are guidelines for how best to develop and use indicators. Also like SAA, LogFrame encourages the engagement and involvement of stakeholders in the LogFrame design process.
3. SROI is a process and method to understand how certain activities can generate value and, importantly, a way to estimate that value in monetary terms. Like Return on Investment (ROI) it is also a way to gauge the magnitude or quantity of the value created compared with the initial investment. SROI has evolved through several iterations (from the REDF model in 2000) to the ‘SROI Network’ version currently being promoted by the Office of the Third Sector in the UK. The key assumption of SROI analysis is that there is more to value creation than purely economic value. The value creation process should be thought of as a continuum with purely economic value at one end, socio-economic value somewhere in the middle, and social value at the other end. SROI measures the value (in monetary terms) of any benefits that may be generated by a program relative to what it cost the particular organisation to achieve. For example: an SROI ratio of 7:1 suggests that an investment of one dollar delivers seven dollars worth of social value. As with any economic modeling though the problem (or skill) lies in the quality of the assumptions made. In the case of SROI, with respect to the outcomes generated and the time taken to generate them and then crucially placing a financial proxy or monetary value on those outcomes. Making these assumptions is not impossible but fraught with difficulty and risks, as making an incorrect or unrealistic assumption at any point along the process may have a significant impact on the final SROI ratio. Therein lies its biggest danger. As we are dealing with social phenomena – whose value is often intrinsic – the decisions made with respect to monetising that value will inevitably be subjective which necessarily limits the ability (and claims) of SROI to provide a means of comparing social impact across organisations within the social sector. While some of the proponents of SROI are careful to point out that the focus of SROI analysis should not solely be on the SROI ratio, the machinations and reality of public policy means that there is likely to be little focus on anything but the neatly expressed SROI ratio.
Comparing the approaches
When looking at the broad area of social impact measurement, it is important to distinguish between frameworks and methods. Frameworks provide a way for organisations to think about, design, plan, implement and embed performance measurement into a project, program or organisation as a whole. They do not prescribe a particular method or indicator to use to assess social impact or performance. SAA and LogFrame are such frameworks, with SAA being generally more applicable at an organisational level and LogFrame generally more applicable at a project or program level. In contrast, SROI should be seen as a more specific evaluation tool. There are several key differences across the approaches. First, SAA is an organisation-wide framework that enables an organisation to assess and outline its overall social (and economic and environmental) impact. So in addition to the outcomes and impact of any particular project or program it may run, SAA also examines organisational processes and capacities in areas such as strategic planning, human resources, governance and accountability, as well as financial management and sustainability, environmental and economic impact.
Second, and as a result of this, SAA is a framework best suited to capturing the social impact at an organisationwide level, while LogFrame works best when used at an individual project or program level. A third key difference relates to the requirement for external assurance. There is no requirement for Logic models to be externally verified as are social accounts and more recently SROI reports. Finally, while SAA, SROI and LogFrame require stakeholder input and engagement, the first two are more firmly predicated on stakeholder engagement principles. In fact, it is not possible to produce a set of social accounts without stakeholder involvement, whereas a LogFrame can technically be drafted by one person sitting at a desk (which unfortunately does occur).
SAA and LogFrame are broader and more flexible frameworks that can be used and applied across a wider range of Third Sector organisations, especially those in arts and culture, advocacy, and small community-based organisations where social value creation is more intangible and difficult to quantify. They provide a way to think about, design, plan, and embed evaluation into a project or program but do not prescribe a particular method or indicator to use to assess performance.
With SAA and LogFrame, as long as indicators are developed using best practice principles they can be quantitative or qualitative. Even though some of its proponents refer to SROI as a process or framework, it is in fact a more specific method or evaluation tool. It is not surprising therefore that SROI has almost exclusively been applied to nonprofit organisations in the social assistance field or social enterprises that operate as intermediate labour markets.
Of the three approaches examined, there is relatively more experience in Australia with LogFrame, with only less than a handful of organisations having gone through the SAA process and fewer still with SROI. Key reasons for the low uptake of these frameworks and methods in Australia include their relatively low profile until recently, but more importantly the fact that all these approaches are resource-intensive for nonprofit organisations to implement in terms of the time taken and the money required to either divert existing internal staff (if they have the required skills) or employ external specialists to assist them through the process.
To access the full article please visit the CSI website: www.csi.edu.au/publications/