Monthly Archives: June 2012

Measuring Success

The topic of how to measure success in museums is one that has enduring interest. Back in 2004, GLI commissioned Maxwell Anderson (at the time a Research Fellow with Princeton University, and now Director of the Dallas Museum of Art) to write a provocative piece for our Compleat Leader online resource library, entitled, “Metrics of success in art museums.”  The piece endures and still attract readers to this day.

A recent email we received, triggered upon reading the metrics article, asked us which process should come first when measuring success: reviewing best practice, undertaking evaluation, benchmarking or increasing business intelligence?  I think it depends. If you want to compare your museum’s success to that of other similar institutions then focusing on best practices and benchmarking makes sense—and such comparisons are no doubt of immense importance to potential funders and trustees. But, it should not be forgotten that each organization has its own organizational processes and characteristics that make “what you do and how you go about doing it” somewhat unique, so spending some energy on designing metrics for evaluating the success of your innovations/innovative practice is also important.

One couldn’t write about museum successes without paying homage to Stephen Weil, who in his article Success/Failure Matrix for Museums (published by AAM in Museum Jan/Feb 2005) emphasized the need to look beyond the numbers and seek qualitative success measures:

“Museums operate on a different time scale. To the extent that a museum’s public programs have a positive impact on the lives of its direct and indirect audiences, that impact may be subtle, diffuse, intermixed with the impact of other organizations, and not always immediate. There is probably no more important task in the museum field today than trying to establish some middle ground—something less than a numerical scale, but also something more than blind faith—between those funders (and others) who demand that museums provide them with hard evidence about their effectiveness and those members of the museum community who argue that the work of museums is of such self-evident value that no justification of that work is necessary.”

Obviously, before designing any evaluation methodology to measure success, you first have to define what success means for your museum, and this clearly should be closely related to your organization’s mission and values. Consider Phil’s recent post about dual rationalities and emerging leadership models. In this, he suggests that good leadership is both a combination of “aesthetic/creative judgment and organizational efficiency.” It requires leaders to 1) maintain distance from the issue at hand while also connecting with stakeholders and hearing their concerns, 2) become a deep subject experts while maintaining enough breadth to stay innovative, 3) protect and conserve while also taking risks, and 4) view issues at hand within the context of the past as well as envisioning all possible futures. A tall order, but no doubt your museum fits in somewhere along each continuum. Success therefore for one institution may in fact look very different from another. Really, what we want to know is “how well do we do what we say we do, and what impact does it have?”

Good museums don’t just have clarity around mission and values, but also communicate this well, internally and externally. They are transparent and accountable.  IMA (another prior leadership role for Mr. Anderson), really embraced the idea of transparency and created Dashboard as a way to openly collect and share measurements about the various aspects of the museums’ performance on its website. (The software is I believe available for implementation by other museums.) An AAM Future of Museums’ blog post from 2009, considered various ways to measure accountability within museums (discussing traditional metrics such as attendance figures, dwell time, number of exhibits, collections acquisition as well as more complex ones such as measuring outcomes, ROI, etc.) and concluded that, “In a future shaped by an expectation of greater accountability, it behooves museum practitioners to choose measures that are appropriate and can be implemented without disproportionate investment of resources.”  With limited resources, some choices have to be made; one cannot collect and measure everything!

Many organizations (not just museums) focus most of their energies on measuring outputs, but as highlighted by Weil, to really understand whether our activities have impact we also need to measure outcomes. Colleen Dilenschneider, in her blog Know Your Own Bone, explains, “Measuring solely outputs in a museum environment (especially in regards to community engagement, provides an immediate advantage and a long-term disadvantage in attracting donors… reporting the output (50 people) may look impressive to higher-level management and potential donors at the time of an annual report, [but] the knowledge of the true outcome of the program (that it altered the lives of 5 individuals in a positive way) is more impressive than the fact that 50 people merely participated.”  We are currently in the second week of our MLI program, and one of our participants today spoke of how much more compelling such impact narratives are if trustees hear them directly from external sources outside of the museum staff, e.g. from museum visitors themselves.

So, how best to measure impact, especially where the impact is not immediate? The first step is to clearly articulate objectives for the activity, i.e. what opportunities are offered. With objectives in place, we can start to identify anticipated outcomes from the activity, i.e. what the visitor will learn/do/feel/etc after the opportunity is experienced. Lastly, we can start to identify the discrete ways in which we might be able to gauge the achievement of said outcomes. When the impact is not immediate, longitudinal research will also be necessary.

The Institute of Museum and Library Services (IMLS) has some evaluation resources that might help. If your mission includes education, then also check out Museums Now’s post on measuring museums, which notes that “The evaluation model an organization uses also speaks to its relationship with its audience. Does the museum see itself as a teacher, a resource, a mentor, coach, research assistant, partner, activist, other? The roles museums take on turn out to matter a great deal.”  The post goes on to provide a great synopsis of the four different learning models—content transfer, skills building, lifelong learning and changing attitudes/behaviors—used by most museums. Barbara Soren has created an audience-based evaluation template which she shares in her audience based measures of success article and also offers two detailed case studies for measuring success from an audience perspective.

And now there is a whole new area to measure—the growth of virtual engagement with your museum as opposed to in-person visitation. Your museum is not solely a physical destination to which visitors come, but also a place from which you now need to engage with your community, wherever they may be. We could collect numerical outputs such as social media traffic, but how to measure outcomes in this context? Well, it depends upon your goals. Is the reason you are engaging virtually to extend the museums reach? To encourage visitation to the physical museum? Or something more innovative, perhaps to extend the museum’s mission into the online environment? This last one is all about measuring impact, and a starting place in designing metrics for the online activities would be to consider how you would go about measuring the impact of your museum’s non-virtual activities. Ms. Soren’s second case study (see above) offers some suggestions for measuring the impact of online activities.

The above links are certainly not meant to be comprehensive. They represent just a few of the measuring success/evaluation-related posts I’ve come across and found interesting reading.  Perhaps, with your help, we can build a more comprehensive resource? What are your thoughts on the matter? How would you define and measure success for your museum? Have you read anything useful or provocative recently on the subject? Perhaps you’re aware of some recent evaluation or benchmarking research. Do share your thoughts below.

By Leonie Fedel, Director of Learning Technologies, Evaluation and Dissemination, GLI