At Conveners.org, we know that when thoughtfully designed, conferences and related events can be transformational experiences that help attendees understand key challenges, encourage stakeholders to collaborate, and mobilize decision-makers to take action. However, the great question many organizers face is, “How do we best measure the impact of our convenings?” Last month, we hosted a webinar with a group of diverse conveners to examine this very topic. The conversation unearthed multiple best practices, as well key questions.
Best Practice 1: Define the “why” behind your convening.
Measuring the impact of your convening first requires you to clearly define your event’s objectives. You should ask yourself:
- What are the primary goals of the event?;
- In light of these goals, what does success look like?; and
- What kinds of return on investment are aligned with these objectives?
Attendees have diverse reasons for attending conferences and related events. It is critical that as an organizer you understand these motivations, and that your success indicators reflect them. During the webinar, participants identified some of the impact indicators they find helpful to use. They included:
- Connecting social impact leaders with one another;
- Disseminating knowledge;
- Highlighting sector successes;
- Increasing investment;
- Networking; and
- Promoting cross-sector collaboration.
Ultimately identifying the right indicators is an essential first step towards developing an effective impact measurement strategy.
Best Practice # 2: Consider your desired impact’s timeline
Each desired impact has a different timeline for when they should be measured. For instance, creating new connections between individuals or organizations can be easily tracked during or immediately after an event. However, realizing greater levels of resource investment or seeing increased cross-sector collaboration requires a longer term view. True partnership building and resource sharing can require months or years to come to fruition. Ultimately, the timeline for your intended outcome should determine when you can begin to measure impact and what type of survey design is most appropriate to do so.
When your impact timeline is short, a survey can be deployed during or immediately after the convening. One example shared previously by SVN, was placing brief paper surveys on attendees’ chairs before a session and collecting them as people exited the room. Setting aside a minute at the end of the session, specifically to fill out the survey, greatly increased participation.
When your intended impact requires a longer timeline, longitudinal surveys can be useful. A survey where you ask the same questions every six to twelve months can reveal complex outcomes (particularly related to change that requires the participation of multiple stakeholders to be effectively realized.) With a longer measurement horizon, it can be helpful to offer participants incentives to providing feedback. Ideas might include raffling off a gift card or free conference registration for next year for those who complete a survey. In addition, it can help to explain to convening participants how the reported data will be used to improve user experience in the future.
Best Practice #3: Look beyond just the numbers
It is important to collect both qualitative and quantitative feedback in your survey. Qualitative questions can provide excellent anecdotes to complement data and provide greater context when reporting findings to attendees and stakeholders. Providing options and space for an open-ended comment (either after each question and/or at the end of a survey) provides space for people to tell their personal stories about the longer term impact they benefited from by participating in your event.
“We have a robust surveying system. In one of the questions we ask “If you have been to the forum before, what results have occurred as a result of your attending?” We have 10 different options for them to check off and an open-ended comment. … We got back three solid pages of single-spaced impact stories about how the forum has impacted them in the long run.”
– Jill Ultan, Skoll World Forum
Best Practice #4: Tools for measuring impact
Among our webinar participants, the most commonly used tool for online surveys was Survey Monkey. For advice on crafting a post-event survey, you can read our Acing the Feedback Survey best practice article.
For immediate feedback, digital tools such as SummitSync and RFID tag codes were mentioned as tools organizers use to both track attendee participation at specific seminars, as well as to get targeted feedback. Onsite paper surveys are also an excellent low-tech option.
Our webinar participants also highlighted that convening organizers can monitor social media for anecdotal reports. Clearly established hashtags can help you do this.
Regardless of what type of tools conference designers reported using, the most successful techniques for measuring impact were intentionally integrated into the convening’s planning process versus as an afterthought following the conclusion of an event.
Challenge #1: Survey fatigue
We have all felt this before – either too many organizations have sent you a survey, or the survey takes 30 minutes to complete, and you get burned out. Survey fatigue is real and unfortunately this negatively impacts conference survey response rates. Our webinar participants struggled with how to overcome this challenge. The best suggestion was to shift the dynamic from asking participants to take time out of their schedules later on to complete surveys, to instead making time onsite to collect feedback. This communicates to participants that their feedback is so important, that you are taking time out of your schedule instead of theirs, to collect it. This also increases both the quality and quantity of responses.
Challenge #2: Reliance on self reporting of collaborations and connections
“We are dependent on people self-reporting collaborations and connections they’ve made. The hardest thing is a way to gather that information in a reasonable manner. Self-reporting is extraordinarily flawed. There should be some tech fix or framing to improve that.”
– Mischa Delaney, B-Labs
We could not agree more. Yet while self-reporting is deeply flawed, we have yet to see a great technical fix. Unfortunately many newer technical solutions (like RFID Badges) are prone to their own challenges and additional costs. Some conveners have considered web crawling (or spiders) which are automated bots that scan the web for key words and aggregate the data into a database. Unfortunately, for most event organizers, the technical sophistication of these solutions prohibits their adoption and effective use. For now, surveys, and high-touch phone calls or conversations are the best methods we have for collecting data on the impact of our convenings.