Measuring Success

The topic of how to measure success in museums is one that has enduring interest. Back in 2004, GLI commissioned Maxwell Anderson (at the time a Research Fellow with Princeton University, and now Director of the Dallas Museum of Art) to write a provocative piece for our Compleat Leader online resource library, entitled, “Metrics of success in art museums.”  The piece endures and still attract readers to this day.

A recent email we received, triggered upon reading the metrics article, asked us which process should come first when measuring success: reviewing best practice, undertaking evaluation, benchmarking or increasing business intelligence?  I think it depends. If you want to compare your museum’s success to that of other similar institutions then focusing on best practices and benchmarking makes sense—and such comparisons are no doubt of immense importance to potential funders and trustees. But, it should not be forgotten that each organization has its own organizational processes and characteristics that make “what you do and how you go about doing it” somewhat unique, so spending some energy on designing metrics for evaluating the success of your innovations/innovative practice is also important.

One couldn’t write about museum successes without paying homage to Stephen Weil, who in his article Success/Failure Matrix for Museums (published by AAM in Museum Jan/Feb 2005) emphasized the need to look beyond the numbers and seek qualitative success measures:

“Museums operate on a different time scale. To the extent that a museum’s public programs have a positive impact on the lives of its direct and indirect audiences, that impact may be subtle, diffuse, intermixed with the impact of other organizations, and not always immediate. There is probably no more important task in the museum field today than trying to establish some middle ground—something less than a numerical scale, but also something more than blind faith—between those funders (and others) who demand that museums provide them with hard evidence about their effectiveness and those members of the museum community who argue that the work of museums is of such self-evident value that no justification of that work is necessary.”

Obviously, before designing any evaluation methodology to measure success, you first have to define what success means for your museum, and this clearly should be closely related to your organization’s mission and values. Consider Phil’s recent post about dual rationalities and emerging leadership models. In this, he suggests that good leadership is both a combination of “aesthetic/creative judgment and organizational efficiency.” It requires leaders to 1) maintain distance from the issue at hand while also connecting with stakeholders and hearing their concerns, 2) become a deep subject experts while maintaining enough breadth to stay innovative, 3) protect and conserve while also taking risks, and 4) view issues at hand within the context of the past as well as envisioning all possible futures. A tall order, but no doubt your museum fits in somewhere along each continuum. Success therefore for one institution may in fact look very different from another. Really, what we want to know is “how well do we do what we say we do, and what impact does it have?”

Good museums don’t just have clarity around mission and values, but also communicate this well, internally and externally. They are transparent and accountable.  IMA (another prior leadership role for Mr. Anderson), really embraced the idea of transparency and created Dashboard as a way to openly collect and share measurements about the various aspects of the museums’ performance on its website. (The software is I believe available for implementation by other museums.) An AAM Future of Museums’ blog post from 2009, considered various ways to measure accountability within museums (discussing traditional metrics such as attendance figures, dwell time, number of exhibits, collections acquisition as well as more complex ones such as measuring outcomes, ROI, etc.) and concluded that, “In a future shaped by an expectation of greater accountability, it behooves museum practitioners to choose measures that are appropriate and can be implemented without disproportionate investment of resources.”  With limited resources, some choices have to be made; one cannot collect and measure everything!

Many organizations (not just museums) focus most of their energies on measuring outputs, but as highlighted by Weil, to really understand whether our activities have impact we also need to measure outcomes. Colleen Dilenschneider, in her blog Know Your Own Bone, explains, “Measuring solely outputs in a museum environment (especially in regards to community engagement, provides an immediate advantage and a long-term disadvantage in attracting donors… reporting the output (50 people) may look impressive to higher-level management and potential donors at the time of an annual report, [but] the knowledge of the true outcome of the program (that it altered the lives of 5 individuals in a positive way) is more impressive than the fact that 50 people merely participated.”  We are currently in the second week of our MLI program, and one of our participants today spoke of how much more compelling such impact narratives are if trustees hear them directly from external sources outside of the museum staff, e.g. from museum visitors themselves.

So, how best to measure impact, especially where the impact is not immediate? The first step is to clearly articulate objectives for the activity, i.e. what opportunities are offered. With objectives in place, we can start to identify anticipated outcomes from the activity, i.e. what the visitor will learn/do/feel/etc after the opportunity is experienced. Lastly, we can start to identify the discrete ways in which we might be able to gauge the achievement of said outcomes. When the impact is not immediate, longitudinal research will also be necessary.

The Institute of Museum and Library Services (IMLS) has some evaluation resources that might help. If your mission includes education, then also check out Museums Now’s post on measuring museums, which notes that “The evaluation model an organization uses also speaks to its relationship with its audience. Does the museum see itself as a teacher, a resource, a mentor, coach, research assistant, partner, activist, other? The roles museums take on turn out to matter a great deal.”  The post goes on to provide a great synopsis of the four different learning models—content transfer, skills building, lifelong learning and changing attitudes/behaviors—used by most museums. Barbara Soren has created an audience-based evaluation template which she shares in her audience based measures of success article and also offers two detailed case studies for measuring success from an audience perspective.

And now there is a whole new area to measure—the growth of virtual engagement with your museum as opposed to in-person visitation. Your museum is not solely a physical destination to which visitors come, but also a place from which you now need to engage with your community, wherever they may be. We could collect numerical outputs such as social media traffic, but how to measure outcomes in this context? Well, it depends upon your goals. Is the reason you are engaging virtually to extend the museums reach? To encourage visitation to the physical museum? Or something more innovative, perhaps to extend the museum’s mission into the online environment? This last one is all about measuring impact, and a starting place in designing metrics for the online activities would be to consider how you would go about measuring the impact of your museum’s non-virtual activities. Ms. Soren’s second case study (see above) offers some suggestions for measuring the impact of online activities.

The above links are certainly not meant to be comprehensive. They represent just a few of the measuring success/evaluation-related posts I’ve come across and found interesting reading.  Perhaps, with your help, we can build a more comprehensive resource? What are your thoughts on the matter? How would you define and measure success for your museum? Have you read anything useful or provocative recently on the subject? Perhaps you’re aware of some recent evaluation or benchmarking research. Do share your thoughts below.

By Leonie Fedel, Director of Learning Technologies, Evaluation and Dissemination, GLI

13 responses to “Measuring Success

  1. There is a recent study of the intrinsic impact of live theater that I believe could help suggest a model of evaluating museum experiences as well. They asked people to respond about how a theater experience had impacted them in 11 categories that included increasing social connectedness, captivation/absorption, producing an emotional response, feeling renewed, gaining new insight, intellectual/reflecting on own opinions. The study was absolutely huge and is published in the book Counting New Beans: Intrinsic Impact and the Value of Art, edited by Clayton Lord.

  2. Yes that study was done by WolfBrown with Theatre Bay Area. A website was created based on the findings of the study http://intrinsicimpact.org/ – “tools and tips for assessing the impacts of arts programs.”

  3. Thanks Tracey and Shiree for adding this resource.

  4. Writing “Metrics of Success in Art Museums” some eight years ago at the GLI’s invitation, I had no idea how much the topic would tap into the museum community’s concerns. Since then I have been invited to speak to museum boards across the U.S. about our collective identity challenge—are we all about attendance, membership and box office results, or are those just symptoms of a narrow part of our business performance?

    Eli Broad recently floated a novel metric when evaluating MOCA’s performance: exhibition cost per visitor: http://www.latimes.com/news/opinion/commentary/la-oe-broad-schimmel-moca-20120708,0,3244612.story. The impulse to consider exhibitions along these lines is tempting for the same reason that measuring raw attendance is: it’s easy. But it is based on a philosophical shift about the purpose of art museums that began when King Tut rose from the dead and impelled museums towards illusory box office treasure in the late 1970s. After Tut we tilted from being educational institutions to being pseudo-commercial destinations with educational activities invisibly trailing behind would-be blockbusters. If we continue to favor performance metrics keyed off our potential destination power, we will continue to lose the plot.

    If only we could measure exhibition impact per visitor as easily. That’s the Holy Grail for exhibition metrics. How much more do your visitors know and remember about issues raised in an exhibition?

    We’re notoriously inefficient at being successful destinations—unless in New York City, where wealthy international tourists swell the ranks of visitors, and distract museum boards in the other dozens of cities imagining that MoMA is the model. But my preferred comparison for big museums cited in the media is not with MoMA or with the Statue of Liberty: it’s with a top-flight research university. We care for precious things and we make their importance manifest to the audiences we welcome and invite to participate. The caring part is where most of the cost is, not special exhibitions.

    Most art museums are about much more than exhibitions. I would guess that on average, less than 20% of our budgets are directly connected with exhibitions, whereas 95% of the public talk about museums concerns attendance. We have to change course to a research, education and experiential impact focus, and away from obsessing about ‘the gate’—which represents less than 5% of our revenue nationally. It’s up to museum professionals to change the topic and measure what we know matters, not what’s easy to measure.

    When your board asks ‘how is our attendance’, you can answer: do you mean how closely does our audience match the socio-economic composition of our metro area? Because the ‘how many bodies crossed the entrance’ question should be of no more interest to a museum director than it is to the owner of a shopping mall. The question for both should be how many transactions were made with our visitors—and in our case, the transaction is artistic, intellectual, and emotional, not monetary. After all, admissions income makes up less than 5% of museum budgets nationally, so there’s a poor business case to be made for increased attendance.

    I would argue that the more an individual makes the choice to visit your museum, the ‘better’ your attendance. If a plurality of people who live in our metro area visit us only once, we should know why. Now that’s something to study, publish the embarrassing results as a baseline, and improve upon.

    Eight years later, I am still working on this topic. This is, as Leonie says, a great place to share ideas on how to do it.

  5. Greetings Maxwell;
    Six months into your tenure at the helm of the DMA I hope has found you still commited to your resolve and advancement of those ideals broadcasted in your manifesto: “A Clear View: The Case for Museum”. I personally believe that art museums should take on the mantle of art education more than anything else because in doing so it creates a validity of continuence for endorsement of relevance and aesthetics.

  6. Pingback: The sudden sexiness of museum-success metrics | Tyler Green: Modern Art Notes | ARTINFO.com

  7. Pingback: The sudden sexiness of museo-success metrics | Art News

  8. Pingback: The sudden sexiness of museo-success metrics | All About

  9. Nik Apostolides (@napostolides)

    Thank you Leonie for shining a spotlight on such an important topic for the public today. I’ve been wrestling with how to measure success in a mixed discipline museum (art and history) for the past three years. I wish I could say that I have found an easy, one size fits all solution, but I have not.

    In fact, I have found that there often is resistance from some staff even to the notion of establishing baselines and setting numerical targets, while other staff embrace the challenge of identifying output and outcome measures with the hope that increased transparency will lead to increased accountability and change. So it’s a bit of a mixed bag out here on the front lines in museums today.

    In my experience, however, progress in measuring success can be supported by maintaining a laser-like focus on the needs of the public we serve, and the role of the artist (or, more broadly, creative actor) in society.

    Our audience literally surrounds us, and they are facing unprecedented challenges today. As Max points out, the more often they return to our institutions, the more opportunities we have to meet their diverse learning needs through art and history. We must remain deeply invested in the welfare, well being, and vast potential for good that resides in our visitors. The stories we tell and the creativity we showcase are most relevant and effective when we find innovative ways to incorporate our audience’s perspectives and engage them in ways that are meaningful and resonate with their own lives today.

    And, we cannot lose sight of the unique contributions that artists and creative actors have made and are continuing to make in society today. I can’t imagine a world without great art – visual art, music, performance art, voice, literature, etc. – and our institutions have an essential role in not only preserving art but also in ensuring that today’s culture values and remains in direct contact with artists and the role they are playing in shaping our ever-changing society.

    Trying to quantify how well we engage with our local audience or how effectively we support an awareness of the role of the artist in society is quite a challenge, but it’s not hard to actually do these things. It just requires a willingness to put our visitor’s needs and living artist’s opportunities above our old institutional prerogatives, and to realign our organization’s priorities.

    In other words, it requires leadership and moral suasion to inspire our fellow staff members, volunteers, and board members to begin to measure our success by the success of our local audience and living artists. When we become a 21st century platform for their stories, for their learning, and for their creativity – the indispensable heart of their community – then we will no longer need to search for the metrics of our success. But until then, the search must continue, and I am heartened to read this blog post and the many excellent comments including of course those by Max Anderson who has mentored so many of us on this topic.

  10. William Haynes

    I’d take issue with the argument that admissions only represent 5% of revenues and are therefore not that important. Let’s get rid of the revenue part of the argument. If you have a museum and it attracts 100,000 visitors one year, and the next year attracts 200,000, then clearly your base for altering people’s lives has doubled.

  11. Pingback: How is the world different because your museum is online? « museum geek

  12. John McCANN: In about 1998 I saw an exhibition at the DMA that changed my course of study, that shifted my compass in the appreciation of Japanese Art called “Momoyama”. From 2000-2003 I had the privilege of working for Max Anderson at the Whitney Museum. He often swam against the tide of Chelsea gallerists, or “tastemakers”. He shifted my compass again, with an exhibition of Alice Neel, thoughtfully curated by Ann Tempkin,
    and was a breath of fresh air. The “harmonic convergence” of the DMA and Max Anderson are bound to bring fresh ideas to whatever he touches. As Dallas/ Ft. Worth keep rising as a hugely important and varied center of art, both ancient and current, I am confident that we will be looking at more than Jeff Koons “Balloon Dogs” in the coming years. Those ideas that Max shared
    are now bearing fruit, and ring true- I say “Bring it!”

  13. Pingback: Measuring Success | Metrics | Scoop.it

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s