Why do our measures matter?

Did you take a research methods class in library school? Are you responsible for collecting or submitting your library’s annual statistics? Do you use surveys to evaluate your library? Are you familiar with Project Outcome? Most anyone who works in public libraries could answer yes to one or more of these questions. If you count yourself among them, then you probably have given some thought to the problems inherent to evaluation, i.e., the ways we measure the value of public libraries—or, any library for that matter.

How we measure library value is a well-documented concern, especially as it relates to how we communicate our value to others. This is a concern especially in times marked by anti-intellectualism and the “death of expertise”. Traditional input and output measures can demonstrate efficiency but not effectiveness (Connaway and Powell, 2010). Moreover, these measures fail to inspire any feelings, for, say, a voter who feels they are already taxed too heavily. Joe Matthews summed up what I am trying to get at with all of this back in 2007:

Historically and today, the generally accepted folk wisdom has regarded the library as a good thing. Yet folk wisdom is no substitute for demonstrating the relevance of the library to its stakeholders, especially the mayors, city managers, county administrators, and boards who control the library’s purse strings. Ultimately, the challenge of demonstrating effectiveness is based on the need to focus on the difference the library makes in the lives of individuals and in the community itself. (p. 286)

By the way, I count this book, The Evaluation and Measurement of Library Services, as an essential! Another indispensable tool, when it comes to talking about measuring value, is the good old logic model, seen below. The logic model is brilliant because it shows the relationships between these different types of evaluation. The library’s inputs and activities, like staffing and program planning, yield outputs such as program attendance, which may result in outcomes, such as learning something new. Impacts are last in this progression because these are usually longitudinal. For example, I myself credit my some of my successes as a baby librarian to an Excel class I took at my public library. So, I wouldn’t be where I am today if it weren’t for that class. Of course, this means impact measurement is really hard, for all sorts of reasons from time to privacy. Libraries can compensate some by asking questions such as, “How do you think this program may benefit you?” But, that too is not really so great of an approach, since it’s a leading question. Putting that problem aside for another time, the bottom line is that outcomes and impact data are usually anecdotal, and thus more compelling than inputs and outputs. Yet, all are necessary, in order that libraries can demonstrate accountability and impact, and also make informed decisions about where to put their resources.

The good ol’ logic model. This example include’s the library’s mission statement. Arrows show a progression, which is fitting since, “One of the characteristics of impact measurement is that it is almost always longitudinal. What is being assessed is how people have changed over time and what the significant factors have been in bringing about this change” (Brophy, 2006, p. 58).

What about ROI measures and calculators? Yes, these can be cool, such as the viral photo of the receipt showing how much money one family saved by using their library. But, they are also problematic, insofar as the value of public libraries can’t sufficiently measured in monetary value. Some things in life are invaluable, like community connection, or beauty, or a restorative afternoon . . .

None of this is news, as I started this post off by saying. I will post a resource list below of my favorite references, three of which are by J. R. Matthews! It’s really must-read stuff for library administrators. Lately, though, I’ve been exploring how calculability and valuation are treated in literature beyond library science. And, woah, let me tell you, it is a big can of wriggly worms. As this is just a blog post, let me try to point out just two more, really interesting sources who treat these subjects.

First, there is George Ritzer, a well-known sociologist who wrote The McDonaldization of Society, a very important book first published in 1993, and since revised and republished. What is McDonaldization? Ritzer defined this as “the process by which the fast-food restaurant are coming to dominate more and more sectors of American society as well as the rest of the world” (2004, p. 1). The concept of McDonaldization is further broken down into four “pillars”: efficiency, predictability, control through non-human technologies, and . . . calculability!

George Ritzer is a distinguished Professor Emeritus at University of Maryland

Libraries exist and operate within systems, from local bureaucracies such as our boards and town councils on up to really broad social constructs like capitalism, itself, that demand accountability and in turn calculability. I write about this more extensively in a paper called, “Libraries as Bureaucracies: a SWOT analysis” (2018). And here is a sweet summary of all that I am getting at here, from a paper that I have now under review. (Is there anything wrong with doing that? I don’t know.)

As a strength, calculability enables public libraries to manage resources, plan for the future, and demonstrate accountability to stakeholders, while an emphasis on calculability also saddles public libraries with the burden of demonstrating value. Calculability is both a defining characteristic of bureaucracies/McDonaldization, and an external force effected by capitalism. It may seem that the best that public libraries might do, in response to this reality, is to comply and keep pace through responsible stewardship, planning, and reporting. Certainly, we’ve done a pretty good job in this respect, and a shift towards outcomes and impact measurement has helped to redirect the way we demonstrate value from quantifiable inputs and outputs towards qualitative stories about the ways libraries change lives and build communities. As Joe Matthews mused in 2010, “Perhaps we should consider a Return on Imagination, a Return on Innovation, a Return on Ideas, a Return on Improvement, a Return on Inquisitiveness, a Return on . . . [?]” (p. 12).
Jordan-Makely, 2019, n.p.

You see, I again end up referencing J. R. Matthews in the above! Besides Matthews and Ritzer, the other worm in this messy can of worms (yeah, that is not the best metaphor, but I don’t want to mix, and truly, this is where this idea of valuation gets really deep and radical, anyway!) is with David Graeber. I got into Graeber through my research into bureaucracy, but he is better known for his bestselling book Debt: The First 5,000 Years, or because he is a sort of celebrity anthropologist, and some of his ideas helped to shape the Occupy Wall Street movement. He published an article in the New York Review just this week titled, “Against Economics.” Somewhere along the lines in my research, I came across one of Graeber’s (relatively) older books, Toward an Anthropological Theory of Value. I’m nowhere near able to process this theory yet, especially insofar as it is relevant to public libraries, only that from an anthropological view the question is something like, how are values created and shared, and how to they shape our reality? Graeber is a joy to read, he unpacks grand theory from the social sciences in a manner that is accessible and even funny. A lot of this relates to critical librarianship as much as to valuation; for example, a short section on “marketless societies” discusses fetishism, or “people failing to recognize the degree to which they themselves are producing value—and for exploitation"—a means by which some people appropriate the surplus value generated by others?” (Graeber, 2001, p. 69).

Of course, anthropologists principle method of evaluation is ethnography, so alongside the theory are various examples of value creation and exchange in different cultures, including gift economies. This is where this little post comes full circle back to how we measure value in public libraries, because I think there is a need for more ethnographic research and story telling, as a means of both evaluation and advocacy, i.e., communicating value. The 2018 book Palaces for the People: How social infrastructure can help fight inequality, polarization, and the civic life, by Erik Klinenberg, serves as an example, especially as it received a lot of media attention, such as this review in the New York Times.

It’s not just ethnographies, interviews with library patrons could also yield really compelling stories as to how libraries help people to connect, and likewise seem like a really under-utilized research method, within the field of library science. Here I wade into personal opinion/experience. Working in roughly ten libraries in ten years, though, I can’t think of any instances where one-on-one interviews were used to collect data about user experience or outcomes or impact. But, whereas focus groups are difficult to conduct and often costly, and surveys run into the troubles of leading questions, and the inevitability of everyone responding “More books!” (joking, sort of), personal interviews allow opportunities to explore experiences in-depth. I may venture they also provide an opportunity for relationship building.

I think that like using standardized surveys, there is a space to create ethnographic research and interview toolkits for library administrators to gather anecdotal evidence as to libraries’ value. I never expect that this would replace more traditional measures. However, I do think that we have a ways to go before librarians are adept at collecting outcomes and impact stories, and in turn using this information to guide decisions at the library, and to win friends and influence stakeholders in the communities we serve.

Resources

Connaway, L. S. and Powell, R. R. 2010. Basic research methods for librarians. 5th ed. Santa Barbara, California: Libraries Unlimited.

Graeber, D. 2001. Toward an anthropological theory of value. New York: palgrave.

Jordan-Makely, C. 2019. Bigger than an elephant: Bureaucracy in, of, and all around public libraries. (Under review.)

Jordan-Makely, C. 2018. Libraries as bureaucracies: a SWOT analysis. Library Management, 40 (3): 294-304.

Klinenberg, E. 2018. Palaces for the people: How social infrastructure can help fight inequality, polarization, and the decline of civic life. New York: Broadway Books.

Matthews, J. R. 2018. The challenges of goodness and value. Public Library Quarterly, 37 (3): 229-232.

Matthews, J. R. 2007. The evaluation and measurement of library services. Westport, Connecticut: Libraries Unlimited.

Matthews, J.R. 2010. What’s the Return on ROI? The Benefits and challenges of calculating your library’s return on investment. Library Leadership and Management, 25 (1): 1-14.

Ritzer, G. 2004. The McDonaldization of society: Revised new century edition. Thousand Oaks, CA: Pine Forge Press.