Service evaluation by outcomes

Service evaluation by outcomes

Granted the prevailing funding climate, there is a widespread interest in service evaluation, if only in terms of ensuring value for money; and in any case, both service providers and commissioners will want to find evidence that their efforts have been effective.

Despite the complexity and the entrenched nature of the difficulties, such costs-led local evaluations of particular services all too often tend to depend upon a relatively narrow set of relatively short term 'outcomes' for the client. Yet short term results to long-term approaches are rarely deeply rooted.

Meanwhile some have criticised the overall direction of such services, arguing that pre-ordained outcomes can encourage 'gaming the system' and distorting the responsiveness of the services: and they give little scope for personalisation, and so can actually disempower the service 'user'.

A far wider challenge to such approaches to evaluation has lately been coming from a number of sources, in a critique of New Public Management; and in attempts to identify an alternative.  Couched in the framework of complexity theory, these critics argue that all supposedly quantifiable 'outcomes' are the result of many, complex interactions over time, and cannot be simply attributed to one agency's immediate efforts. These suggest that a whole systems approach is needed for any real advance.

A more psychologically informed approach to outcomes

The PIEs approach would generally encourage services at least to think more broadly about the overall needs of the service user group, and what you are trying to achieve; and to consider not just the evaluated outcomes, but how to work with all the issues in a service user's life that may be most relevant to them.  It may help to argue that unless those broader needs are addressed, short term change is likely to be less sustainable.

Nevertheless, the suggestion that a more systemic approach to service commissioning is clearly necessary; and the PIES 2 framework, by broadening the concept of the environment to include and encourage the engagement with local systems and pathways, means to encourage this. The Pizazz assessments include these as areas for a PIE to assess; and the PIE Abacus makes sharing such feedback on gaps and barriers more possible.

 

Further reading, listening and viewing

1: On evaluation per se

  • PIE assessment - what is the point?  (PIELink page) :  HERE
  • Service evaluation by outcomes (PIELink page) : HERE
  • Evaluations of specific interventions (PIELink page) : HERE
  • Whole systems evaluation (PIELink page) : HERE
  • Formative vs summative evaluation (PIELink page) : HERE
  • Outcome and process assessments (PIELink page) : HERE

2: On complex needs evaluation and research issues generally

  • Toby Lowe on Exploring New Insights (Human, Learning, Systems video)   : HERE
  • Annie Danuco, on formative vs summative evaluation: HERE
  • Becky Rice and Juliette Howe on person-centred research for complex needs: HERE
  • Grant Everitt on the range and sheer complexity of data in work with complex needs: HERE
  • Stephanie Barker and Nick Maguire on the lack of studies researching peer support: HERE
  • Sophie Boobis on researchers learning from a dialogue with evolving practice (video): HERE
  • McDonald & Tomlin: on mindfulness evaluation with young people, with cautions over a premature preference for meta-analysis: HERE
  • Emma Belton: on the challenges in researching behaviour change in young people; and the search for alternative evaluation approaches: HERE
  • Mental Health Foundation: Progression Together, a report with honest comments on difficulties with evaluation studies: HERE
  • Robin Johnson: 'Do complex needs need complex needs services? (Pts 1&2):  HERE
  • Zack Ahmed on using Participatory Appraisal in involving users in local area needs research: HERE
  • Collaborate/Newcastle University Business School on complexity and a new paradigm HERE and (excepts): HERE
  • Sophie Boobis: Evaluation of a Dialogical Psychologically Informed Environment: HERE
  • Brett Grellier: report on a mindfulness programme in three homelessness hostels:  HERE
  • Sophie Boobis on evaluation of facilitated PIEs training: HERE
  • Robin Johnson (in conversation) on outcomes measurement: HERE

 

3: On PIEs assessment specifically

  • The Pizazz as a research tool: HERE
  • The iAbacus team on the IAbacus process - developing the questions: HERE
  • 'Useful questions' the Pizazz process handbook: HERE