#7 Multi-level design and evaluation
Using systems thinking for design and evaluation
Welcome back
This week, I’ve been having lots of conversations.
And yes, I do mean lots of meetings - the dangers of virtual scheduling is the ability to end up in back-to-back-to-back meetings!
Anyway, today I had the pleasure of presenting on a recent project that incorporated systems thinking, design thinking and evaluative… thinking? Whatever - you get the point. Three different fields of work that we attempted to bring together in a fairly cohesive and actionable way.
What I took away from the experience though was in the ‘how’ of applying those forms of thinking to the project. Unintentional in the early stages, it is now apparent to me that each had a clear form and function in the project.
This project involved the development of a systems map on the barriers and enablers to women and children remaining Safe at Home after incidents of family violence. If you want to read more about this important work you can head to the McAuley website (our client).
What I think is particularly fascinating about this work is the recognition of the need to understand the context (system) that is helping or hindering Safe at Home. The process we went through - no different to other social research projects in many ways - was done in recognition of the multi-faceted steps that needed to be undertaken.
Importantly, through a systems-lens, the project recognised the complex and interconnected nature of the barriers and enablers. For me, it has become a great example of how different modes of thinking for different purposes can help transcend some of the past disciplinary boundaries I’ve felt (and thought) in my own practice.
Here’s my thoughts on each of the three.
Systems thinking
My ‘newest’ area of interest, but also one that just makes sense. Everything is connected (no, don’t ‘use the Force’), but actually it IS. How we then understand and make sense of things really appeals to me - maybe it’s my underpinning Anthropology undergrad degree.
Regardless, translating some of these key principles into an approach that subsequently meets the needs of the project was immensely satisfying. Particularly when coupled with my somewhat pragmatic attitude when it comes to figuring out what to do with a complex problem - we have to do ‘something’ - so why not figure out what the most effective ‘something’ you can do is?
Design thinking
Design was and is a very topical area of fascination for non-designers. Whether you’re ‘thinking’, ‘human-centred’, or ‘co-designing’, I don’t think you can deny the prevalence of the basics across most content areas.
However, in this project we actually used the five base stages of Design Thinking as a way of project planning on how to meet our clients needs. It really helped with clarifying ‘what’ success needed to be for the project (and the client, and their stakeholders), but also the role of design across levels (more on that shortly).
Evaluative thinking
Who doesn’t love to question assumptions? I think this flows through most of my work, but also the idea of how change is considered, measured and communicated. I frame this as ‘thinking’, in the sense that for this project it was both mindset and capability.
We need to establish a process and way of understanding how change will be measured, but also maintaining the questioning of assumptions. I think this is a valuable fit for systems thinking, and that the two really do go hand-in-hand.
Where are you going with this?
Bear with me. For the purposes of my presentation today, the focus was (broadly) on the idea of the intersection of design and evaluation. When preparing some of my thinking I - as I often do - started to question what I was preparing. In the end, I decided I wanted to finish off my spiel (which basically reflected the process we went through, and the ‘so what’ of it, with presenting a broader model.
So - here goes - when thinking of systems, design and evaluation - and after so many conversations this week - I have settled on the fact that we really saying the same thing, just differently.
In my own attempt to delineate thoughts, and using this Safe at Home project as an example, I have settled on the following broad steps in a cognitive model of how to approach design and evaluation across levels (i.e. systems).
System design - the basics of systems thinking, clarification, boundaries, definitions and connections. While you might call it 'systems mapping’, to some extent that draws my mind back to cartography. The world is not objective, rather what we map is how we make sense of complex topics, so to some extent maybe ‘systems design’ is a way to think of it?
System segment - the next step in my pragmatic attitude. You get presented with a big complex map and it’s overwhelming, too complex and even a bit paralysing. So, work through a process of identifying a system segment, snippet or sub-system. Bite-sized chunk might be a bit too casual? But hey, have to start somewhere. The reason I like this idea of a ‘segment’ is that it’s still scalable - have an abundance of budget, time and staff? Take a bigger bite!
Intervention design - next, you figure out how you’ll make change. Enter the ‘traditional’ design forms (e.g. co-design). We’re basically using our system segment to step ‘down’ a level and make it all a bit more manageable (i.e. less overwhelming). You would still do all your basic evaluation planning bits and pieces (e.g. Theory of Change, logic model, indicators, etc, etc).
Intervention evaluation - implement evaluation. A no-brainer, and again - being pragmatic, what are the effective approaches or forms you can implement? Be realistic, but also match it to the size of your segment (and intervention).
System design and evaluation (maybe more accurately system re-design) - I won’t call it circular. Partially it was because I couldn’t make the below figure pretty enough, but actually I think this also makes sense. Circularity - to some extent - feels a bit like a treadmill. Whereas - in theory - if you’ve intervened (and not to mention the external / contextual factors you have no control over) - the system needs to be re-mapped, rather than just ‘questioned’ as to how it changed (though that’s a part of remapping).
Here’s my summary graphic:
Hopefully it resonates - but maybe it doesn’t - and I’m OK with that. My next goal is to work through this further with more examples, and I really think we can start to unpack ‘bigger’ integrated approaches across disciplines, but also levels.
That’s how we get change.