I like this conversation between Daniel Kahneman and Gary Klien, and think that although it is ostensibly about corporate decision making, it can apply to a lot of charities or social enterprises in how they design their work and think about the impact of that work.
Too often there is a tendency to go too far one way or the other on a spectrum of: "I know that what we're doing is good", through to, "I need lots of high quality data to prove to me that what we're doing is good". Neither of those poles are helpful, because really what you need is enough information to make the right decision and to learn.
Intuition is a great way into this for charities, as front line workers tend to trust intuition and economists/ consultants/ commissioners tend to not. They want data, and often in the pursuit of that data, forget why. What this debate highlights is that there are certain times when we know that intuition is exactly the right metric for experienced staff (and the bean counters should back off), and there are times when the intuition of staff is just a bias, and we need a different metric or measure. That, to me, is an interesting place for people who are designing and evaluating the effectiveness of services to start.
First, there needs to be a certain structure to a situation, a certain predictability that allows you to have a basis for the intuition. If a situation is very, very turbulent, we say it has low validity, and there’s no basis for intuition. For example, you shouldn’t trust the judgments of stock brokers picking individual stocks. The second factor is whether decision makers have a chance to get feedback on their judgments, so that they can strengthen them and gain expertise.