The Nonprofit Guide to Testing and Optimization
“Creative experimentation propels our culture forward. That our stories of innovation tend to glorify the breakthroughs and edit out all the experimental mistakes doesn’t mean that mistakes play a trivial role. As any artist or scientist knows, without some protected, even sacred space for mistakes, innovation would cease.” – Evgeny Morozov
Our final blog post in our series on innovation focuses on the critical topic of experimentation and testing. Many nonprofit marketers argue that the inability to test and measure results is possibly the biggest problem facing today’s nonprofits. Many nonprofits seem incapable of setting clear, testable goals and then creating experiments that validate their hypotheses. As a result, organizations waste thousands of hours on activities that don’t directly impact the cause, or the impact they make isn’t measurable.
There are several ways to address this problem, but the basic testing principles used by startups is a great place to start. Most startups use this simple framework for testing and measuring results that can be applied at any level of a nonprofit. By using this 4-step process, nonprofits can quickly test and validate new ideas – or, just as importantly, validate and optimize existing initiatives.
Step 1: Hypothesis
Start by making a clear hypothesis (a best guess at solving a problem). The hypothesis can address both big and small problems at the organization.
For example, a hypothesis might be, “we think that unemployment in Charlotte is the number one cause of increased incarceration,” Or, “we think that sending paper receipts doesn’t create a positive financial ROI for our organization.”
Step 2: MVP
A minimum viable product (MVP) is a development technique in which a new product is developed with sufficient features to satisfy early adopters. It is the minimum amount of work that needs to be done to test a hypothesis and determine whether a concept/product will work or not. Once the hypothesis is clear, the MVP will help validate it with the smallest amount of time and money possible.
For example, “let’s set aside 10% of our donors over the next 4 months to get digital-only receipts with similar copy, then measure the ROI (cost vs. revenue) for those donors over the period.”
Step 3: Test
Set up a valid test and validate results.
Testing is simply a formal method for gathering data and validating a hypothesis. To create a valid test, consider the following:
- Make sure to get a statistically significant set of data points (typically over 30). (e.g. 30 donors who stop getting paper receipts)
- Create an A/B split test to compare the current solution against the new solution. A/B testing is a way to isolate and test ONLY the hypothesis while avoiding other variables.
- Be very skeptical of soft, personal stories. Try to rely on hard data over the opinion of a few.
Step 4: Pivot
Fail fast!!! On a recent Modern Fundraiser Podcast, Jon Burgess encouraged our listeners to reward failure. Yes, you heard that right, REWARD failure. Unless your team feels the freedom to fail you can never truly innovate. Fear of failure will always cause your team to drift to consensus thinking and fall back to “tried and true” methods of the past rather than taking risks and pushing toward innovation.
Most leading startups would rather prove a hypothesis wrong right away than meander in the wrong direction for years. A pivot is just a change in direction based on what has been learned. Getting clarity when it’s time to make a change can be harder than you think. In fact, I’ve seen nonprofits knowingly walk down the wrong path for months without clarity on why they should change. Testing and data-driven analysis allows a nonprofit to learn what’s wrong quickly so that it doesn’t keep making the same mistakes repeatedly. When data either validates or invalidates your hypothesis, then it’s time to be brave. Pivot!
If you’re ready to start building testing into the culture of your nonprofit make sure to download our Testing Calendar Template