Service evaluation by outcomes

Service evaluation by outcomes

Granted the prevailing funding climate, there is a widespread interest in service evaluation, if only in terms of ensuring value for money; and in any case, both service providers and commissioners will want to find evidence that their efforts have been effective.

Despite the complexity and the entrenched nature of the difficulties, such costs-led local evaluations of particular services all too often tend to depend upon a relatively narrow set of relatively short term 'outcomes' for the client. Yet short term results to long-term approaches are rarely deeply rooted.

Meanwhile some have criticised the overall direction of such services, arguing that pre-ordained outcomes can encourage 'gaming the system' and distorting the responsiveness of the services: and they give little scope for personalisation, and so can actually disempower the service 'user'.

A far wider challenge to such approaches to evaluation has lately been coming from a number of sources, in a critique of New Public Management; and in attempts to identify an alternative.  Couched in the framework of complexity theory, these critics argue that all supposedly quantifiable 'outcomes' are the result of many, complex interactions over time, and cannot be simply attributed to one agency's immediate efforts. These suggest that a whole systems approach is needed for any real advance.

A more psychologically informed approach to outcomes

The PIEs approach would generally encourage services at least to think more broadly about the overall needs of the service user group, and what you are trying to achieve; and to consider not just the evaluated outcomes, but how to work with all the issues in a service user's life that may be most relevant to them.  It may help to argue that unless those broader needs are addressed, short term change is likely to be less sustainable.

Nevertheless, the suggestion that a more systemic approach to service commissioning is clearly necessary; and the PIES 2 framework, by broadening the concept of the environment to include and encourage the engagement with local systems and pathways, means to encourage this. The Pizazz assessments include these as areas for a PIE to assess; and the PIE Abacus makes sharing such feedback on gaps and barriers more possible.

 

Further reading, listening and viewing

The New Public management approach : HERE

'Joined up thinking' : HERE

360 degree appraisal : HERE

 

PIEs assessment

PIEs assessment - what's the point? : HERE

The Pizazz on paper and on screen : HERE

The Pizazz - what is it NOT? : HERE

The PIE Abacus - an on-line Pizazz (summary) : HERE

 

On evaluation per se

  • Service evaluation by outcomes: HERE
  • Evaluations of specific interventions: HERE
  • Whole systems evaluation: HERE

 

Forums

Beyond Outcomes - a discussion with Joy MacKeith : HERE

 

Library items

  • Formative vs summative evaluationHERE
  • Outcome and process assessmentsHERE

On complex needs evaluation and research issues generally

  • Toby Lowe on Exploring New Insights (Human, Learning, Systems video)   : HERE
  • Annie Danuco, on formative vs summative evaluation: HERE
  • Becky Rice and Juliette Howe on person-centred research for complex needs: HERE
  • Grant Everitt on the range and sheer complexity of data in work with complex needs: HERE
  • Stephanie Barker and Nick Maguire on the lack of studies researching peer support: HERE
  • Sophie Boobis on researchers learning from a dialogue with evolving practice (video): HERE
  • McDonald & Tomlin: on mindfulness evaluation with young people, with cautions over a premature preference for meta-analysis: HERE
  • Emma Belton: on the challenges in researching behaviour change in young people; and the search for alternative evaluation approaches: HERE
  • Mental Health Foundation: Progression Together, a report with honest comments on difficulties with evaluation studies: HERE
  • Robin Johnson: 'Do complex needs need complex needs services? (Pts 1&2):  HERE
  • Zack Ahmed on using Participatory Appraisal in involving users in local area needs research: HERE
  • Collaborate/Newcastle University Business School on complexity and a new paradigm HERE and (excepts): HERE
  • Sophie Boobis: Evaluation of a Dialogical Psychologically Informed Environment: HERE
  • Brett Grellier: report on a mindfulness programme in three homelessness hostels:  HERE
  • Sophie Boobis on evaluation of facilitated PIEs training: HERE
  • Robin Johnson (in conversation) on outcomes measurement: HERE

 

On PIEs assessment specifically

  • The Pizazz as a research tool: HERE
  • The iAbacus team on the IAbacus process - developing the questions: HERE
  • 'Useful questions' the Pizazz process handbook: HERE