Introduction

Good morning, my name is Tom Hennen. 

I have been a librarian for 2.5% of a millennium.  In those 25 years, I have visited hundreds of libraries: good ones, bad ones, excellent ones and awful ones.   How did I judge?  By instinct, professional judgment, and, of course, by whether or not the library had what I wanted when I wanted it  - just as any other librarian or library user might do. 

 

Recently, in part inspired by ratings of everything else from cities to hospitals to universities, I decided to try to take a somewhat more systematic approach to evaluating libraries.  The result was the HAPLR Index rating system.  A report on it has been published twice in American Libraries, in January and September of this year.   The haplr index web site I established has received tens of thousands of visitors since the beginning of 1999.  Hundreds of libraries have chosen to order individualized scorecard ratings as well.  There have been some objections, of course. That’s the trouble with ratings. 

 

All in all, I am glad to be doing the HAPLR Index and plan to continue for quite a while.  I hope that most in my audience today are glad to hear that. 

 

This afternoon I want to deal with five things. 

Ø      First, I want to talk about the Index and how it is constructed. 

Ø      Second, I will turn a description of the elements of a standard rating report.

Ø      Third, I will discuss the elements of a special report.

Ø      Fourth, I will discuss the need for national library standards that I believe should be adopted by PLA.

Ø      Fifth, time permitting; I will cover the 10 most frequently asked questions about the HAPLR ratings.

 

Score card ratings for 9,000 U.S. libraries.

The HAPLR Index uses data provided by nearly 9,000 public libraries in the United States to create comparative rankings.

 

The comparisons are in broad population categories with breaks at 1,000; 2,500; 5,000; 10,000; 25,000; 50,000; 100,000; 250,000; and 500,000.  The data are from the Federal-State Cooperative Service (FSCS). 

 

Does Wind Chill Factor tell you how cold it is?

I am from Wisconsin, so I don’t have to tell you that it gets cold there in winter.  When I was a kid there was no wind chill index, but on days when I walked home from school without gloves, I suspect it was just as cold as now.  It is much the same with a library-rating index.  Your libraries are just as good, or just as bad, as they were before; the HAPLR Index just makes us notice it more.

Rating Information

 

 

Input and Output measures

The HAPLR Index uses six input and nine output measures.  The measures are calculated from the Federal-State Cooperative System for public library data (FSCS). Each factor was weighted and scored.  I then added the scores for each library within a population category to develop a weighted score.  

 

Similar to ACT or SAT scores

The HAPLR Index is similar to an ACT or SAT score with a theoretical minimum of 1 and a maximum of 1,000.   About 90% of libraries in each population group scored between 260 and 730.

Percentile Comparisons

Percentiles compare a library to all libraries in the population category.  A 90th percentile rating means that the library scores higher than 90% of comparably sized libraries.

 

 

The graphs compare each library to comparable libraries throughout the country.  For each factor the graph illustrates the library's percentile score.  A 50th percentile score means half of all libraries in the group scored higher, half lower.

 

The libraries in the population category were ranked with weighted scores for each factor to produce the composite HAPLR Index score, rank and cumulative percentile noted next to the library name for each library. 

 

Percentile scores in the top (or bottom) quartile on EVERY measure are exceptionally rare in the national data.

Mapping HAPLR

The rankings are not weighted for population of library communities. 

   

 

Grading on the Curve

As mentioned earlier, most scores are between 260 and 730. 

 

 

 

 

Standard Report

The information on a library’s HAPLR score is available free on the haplr-index.com web site.  Also available for $15, is a 4-page standard report that compares the library to 10 other libraries- 5 state and 5 national closest in population size. 

 

 

Weighting of Factors

I determined the weighting of the factors.  The weightings are intended to reflect the relative value of each factor in an overall comparison.  Other librarians throughout the country were queried about their recommendations for weighting factors.

 

The relative weight assignment is among the most frequently criticized elements of HAPLR. 

 

In a Library Journal article titled “The Trouble with Hennen,” an author from Oregon laments that the weighting is not sufficiently explicit.  Yet every report sent includes the weighting and the web site has a section devoted to methodology. 

 

The Oregon author also adds some confusion by trying to explain my methodology and getting it wrong, so allow me to correct that here.  He says that all 9000 libraries are ranked on a given measure and then later sorted by population.  The opposite is true: all libraries are ranked by population first and then compared on each measure to only those libraries in their population category. 

Sample Score Cards

Next we can look at a sample standard score card and its various elements. 

 

First is a graphical The graph compares the library to comparable libraries throughout the country.  For each factor the graph illustrates the library's percentile score.  A 50th percentile score means half of all libraries in the group scored higher, half lower. The libraries in the population category were ranked with weighted scores for each factor to produce the composite HAPLR Index score, rank and cumulative percentiles noted above.  Note that all factors are ranked from high to low EXCEPT cost per circulation.  It is assumed that the best cost per circulation is the lowest cost.  See Table 1 for detail of percentiles in this graph. 

 

 

 

Sample ScoreCard – Low

Comparing this graphic to the previous graphic gives one an idea of the variations that can be seen among libraries.  Both are libraries in the same population category, but the former got a much higher HAPLR ranking than this library.

 

   

 

 

 

Sample ScoreCard – Silk purse

There are times when the input measures – mostly money are low, but the output measures are still extraordinarily high.  I call this the silk purse scenario – as in silk purse from a sow’s ear.

 

 

 

 

 

 

 

 

Detailed Data – HAPLR Measures

 

A standard report includes the specific data used to generate the measures and the graphics in the report.  There are a total of 312 libraries nationally in the population category illustrated here.

 

This table indicates rank and percentile score for each element of the HAPLR Index.

 

 

 

Percentile Comparisons

These comparisons help keep the rankings and scores in perspective.  By comparing your library data with the percentiles here, you can tell where the library is compared to other libraries in the 75^th, 50^th and 25^th percentile. 

 

 

 

 

 

 

 

Detailed Measures

Every standard report also includes comparisons to the 5 libraries closest to the library’s population size in the state and 5 in the nation.   This allows for comparison on specific measures with like size libraries.  Libraries can also ask for specific comparisons, but the costs are considerably higher because of the extra programming involved. 

 

 

 

Special Reports

 

In addition to the $15 standard report, special reports are available.  These are more expensive (starting at $100) and I provide quotes before we agree on a product.  Special reports vary.  A library may wish to set the peer libraries to which it is compared, as one example.   States have asked for comparisons to other states or national averages.  Systems or regional groupings also request reports on all their members in a geographic area.  I will demonstrate just a few selected slides from several of the types of reports.  The examples are by no means exhaustive. PowerPoint presentations to go along with the print report are also available for an additional charge. 

 

 

 

 

 

State -Combined Inputs

This report compares the libraries in each of the main population categories in the states of Tennessee and Ohio on a composite score for the 6 input measures that make up the HAPLR score.

It came as no surprise to either state that Ohio outranked Tennessee on all input measures.  Next, let’s look at the composite output scores to see if there is a similar result.

 

 

 

 

 

 

 

State - Combined Outputs

This graph compares the libraries in each of the main population categories in the states of Tennessee and Ohio on a composite score for the 9 output measures that make up the HAPLR score.  Tennessee composite output scores were much better by comparison than the input scores would seem to have predicted.   The HAPLR Score combines both input and output measures.   Special reports allow us to go into more detail on the issues.  The full report dealt with comparisons on each of the 15 measures individually.

 

 

 

 

 

 

 State – Individual Measures example

I also received a request for a comparison of North Carolina libraries to their national peers in each population category.   This graph illustrates the percent of their budgets that libraries apply to the materials budget. 

 

 

 

 

 

   

 

 

State – Individual Measures -2

Continuing with the comparison of North Carolina libraries to their national peers in each population category, here is what circulation per open hour looked like in each grouping.

 

The numerical detail is provided for each report, of course. 

I believe that reports such as this must incorporate words, numbers and graphs in order to communicate to all the types of people there are viewing the report.

 

 

 

 

Individual Special Report- 1

The next two graphically compare an individual library to the 25th, 50th and 75th percentile on each HAPLR measure. 

 

This was part of a larger report that provided other levels of comparison detail. 

 

 

 

 

 

 

 Individual Special Report - 2

 

The entire report is available on the web at:  haplr-index.com.  Follow the links to special reports, then PowerPoint demonstration. 

 

 

 

 

 

 

 

 

 

 

Regional Comparison

 

This is from a comparison done for a federated library system that wanted to see the HAPLR scores for all of its members.  The bar graph illustrates the relative scores for each library.  The full report contained the details, of course. 

 

 

 

 

 

 

 

 

 

Regional Detail

This is from a comparison done for a federated library system that wanted to see the HAPLR scores for all of its members.  This is the detail for the measures for individual libraries.

 

A regional report such as this also includes bar graphs on the 15 HAPLR measures for each library.

 

 

 

 

 

 

 

 

Individual Special Report – 3

This slide illustrates a special peer report that was requested recently.  The libraries are all in the 250 to 500,000 population category.

 

First we have four libraries each compared on their percentile score for the 6 input measures.  As you can see, the two libraries on the left have much lower input scores than the two on the right.   The second graphs illustrates these same libraries on the 9 output measures.  Libraries A and B do considerably better than their input data would lead one to expect. 

 

 

 

 

 

Standards

 

In the early 1980’s ALA abandoned public library standards that had been in use in various forms for nearly half a century.  I believe it is time to bring back national standards.  My article, “Forward to Basics” appeared in the March issue of American Libraries.

 

I will just make a few points from that article here.  The article is also expanded on the haplr-index .com web site.

 

 

 

 

 

Back to Basics Article

Although ALA abandoned standards in favor of planning processes, most states did not.

 

Objections came from both the high and low scorers when standards were considered.  Those on the high end feared they would be cut back to the average by their funding authorities. 

 

Those at the low end felt shame and despair. ALA instituted planning processes that allowed for a BYO party – Bring Your Own Standards!   If standards got in the way of good planning, that is unfortunate, but it was not and is not, enough reason to get rid of them.  It is high time that ALA reinstated national library standards and the initiative should begin at PLA, of course.

 

Let’s “Plan for Results,” but let’s all use the same standard measuring tools.

 

Major Points to Back to Basics Article

 

I believe we need minimum standards, target standards and benchmarks of excellence.

 

Minimum standards would indicate a level below which an agency cannot fall and still call itself a library. 

 

Target standards can be generated by indicating percentile levels and trying to urge everyone to meet or exceed at least some portion of that median.  Over time, this rising tide lifts all the library boats, of course, and the targets need to be re-calibrated – a 5 year renewal rate should be included for the standards.

 

Finally, we need benchmarks for excellence that the best libraries can shoot for.  With incentives and rewards we can all learn from the best rather than having them fear and object to standards that hold them back. 

Genius Grant Detail

Here is how I would like to use benchmarking tools like the HAPLR Index.

1.      Use the HAPLR Index to identify 100 (or more) candidates for library mentor grants.  Libraries so identified could then choose to enter into a grant process. 

2.      Applicants would be subjected to a peer review process that lets seasoned professional librarians rate the libraries.  This will assure professional judgment of the libraries in a process similar to that applied to library schools for certification.

3.      The applicant libraries would also be tested with a customer satisfaction inventory using a national agency such as the Gallup poll to assure that in addition to the statistics in the HAPLR Index and the Peer Review, they are also providing customer service in an excellent manner.

4.      Have libraries go through a quality assurance process similar to that used by private industry using the ISO 9000 standards.  This would assure that libraries would have the necessary documentation on planning and development to allow other libraries and library schools to study their best practices. 

5.      Enlist the agreement of Library Schools to provide field placements at the mentor libraries.  This would allow new graduates exposure, either on site or using virtual library discussions, to examine the best practices of the top libraries.  It would also, of course, provide a pool of enthusiastic graduate students to study and further refine these best practices at the mentor libraries.

6.      Establish virtual training centers using distance education technology to discuss and examine the best practices at the mentor libraries.  The distance education centers would allow library staff at all libraries to join a virtual community to examine the best practices at the mentor libraries using the Internet, Internet telephony, and distance education tools. 

7.      Provide a limited number (50, 100) Best Practices Library Grants to libraries that succeed in getting through all the steps outlined.  These would be similar to Genius Grants.  The library would receive the prestige for the award and a cash grant from government or private foundation sources.  The awards would be without any strings with the assumption that the chosen libraries would use them to define improved practices for the future. 

 

 

Frequently Asked Questions

 

 

 

FAQ 1

What led you to do the HAPLR Index?

Practically every time you pick up a magazine or newspaper there is another rating system for universities, places to work, hospitals, mutual funds, you name it.  But there was none for libraries.  Worse than that, the Money magazine listing of best places to live covered libraries by measuring only books per capita.  I was certain that a more comprehensive tool was needed. 

 

Why don’t you consider electronic measures?

I would very much like to include measures of Internet use or use of electronic databases, but the Federal data on which I base the index does not yet include such measures.  The Federal State Cooperative Service has been debating and considering what measures to consider for a number of years.  No consensus has yet emerged.  As soon as they begin to use the measure, I plan to incorporate it into the index.

 

Why don’t you consider square feet for the building?

I would very much like to include building size but the  Federal State Cooperative Service data on which I base the index does not yet include such measures.  As soon as they begin to use the measure, I plan to incorporate it into the index.

 

Isn’t it really quality of service that counts; why rate quantity only?        

Of course quality counts.  As I said in the January 1999 issue of American Libraries, “data measurement cannot capture a friendly smile and a warm greeting at the circulation desk.  Nor can data measurement alone measure the excitement of a child at story time or a senior surfing the Internet for the first time.”  But we have no accepted and nationally consistent measures of quality in library services that would allow for comparisons like the HAPLR Index.  I agree that numbers alone do not identify truly great libraries, quality counts too.  On the other hand, I do not believe that a library can be truly great with poor numbers.  As my logic professor taught me, the numbers are a necessary but not sufficient condition for defining greatness.

FAQ 2

Do you plan to continue publishing the Index?

Yes, I hope to publish it annually.

 

Are there similar rating methods for libraries?

The HAPLR Index is the first of its kind for libraries in the United States.  There are no similar programs in Canada, Australia or New Zealand.  I know that there is some interest in developing a similar index in Australia and New Zealand, because I published an article on the topic in the June 1999 issue of APLIS, the Australasian Public Library and Information Science magazine.  Great Britain is in the process of developing new standards and performance measures, although nothing has yet been published.  This is not quite the same thing as the HAPLR Index, but close. 

See their site http://www.lic.gov.uk/publications/policyreports/building/index.html

 

There is a project funded by DG13 of the European Commission within the Telematics Applications Programme. The project runs for 3 years (until the latter end of 2000).  They are using Internet communications to develop a continuously updated database of statistics about library activities and associated costs in the context of their national economies. That information may be found at http://www.libecon2000.org/introduc.htm

 

This project does not develop an index similar to the HAPLR-Index, however.   Finally, and the closest thing to the HAPLR Index is being developed in Gemany.  The project sponsored by the Bertelsmann Foundation is called "BIX  - The Library Index": the project will start in October 1999.  They hope to use some data of the official "German Library Statistic" (see http://www.dbi-berlin.de/bib_wes/zdf/zdf_00.htm). The first results will be published in April/May 2000. A further description of the project may be found at:

http://www.stiftung.bertelsmann.de/english/suche/index.htm

 

Out of date?

Anyone involved with data gathering and statistics wishes that they could be timelier, but we do what we can.  HAPLR Index II is based primarily on 1997 data.  The 1997 information was collected by the states and submitted to the Federal State Cooperative Service (FSCS).  The data are due by July of 1998.  Information is checked for internal consistency by FSCS and then published first on the Internet.  The first edition was based on early release data before FSCS imputed, or statistically estimated responses for some libraries. The second edition is based on the revised data. After the release of the data, it takes me several months to do the necessary calculations for the rankings and for individualized reports on demand for all 9,000 libraries in the country.  As states increasingly automate their data collection and allow for filing over the Internet, the data will become closer and closer to “real time,” rather than the belated information we are now working with. 

FAQ 3

How many libraries are there in each population category?

There are 8,946 library entities included in the Federal State Cooperative Service database. Library systems with multiple branches are counted as a single entity.  Just 194 libraries did not supply sufficient data to calculate the index.  The population categories used are those used by the FSCS for other comparisons, with one exception.  The FSCS data includes another category of libraries over 1 million population, but that would have provided too few libraries for purposes of the Index. 

 

 

Category          Number of libraries      

Over 500,000  72       

250,000 to 499,999     90       

100,000 to 249,999     312     

50,000 to 99,999         508     

25,000 to 49,999         861     

10,000 to 24,999         1,672  

5,000 to 9,999 1,492  

2,500 to 4,999 1,306  

1,000 to 2,499 1,582  

Under 1,000     857     

Unrated            194     

Totals   8,946  

 

Can one compare rating numbers between two or more population categories?

With care, yes, one can do so.  There are variations in the highs, lows, medians, and so forth that vary by population sizes.  So a score of say 600 may be more easily attainable in some categories than others.  But the variations are not so extreme that no comparisons across population categories are possible.

FAQ 4

How can you mix both input and output measures in the Index?

Some have criticized the HAPLR Index for including both input and output measures in the same index.  They note that inputs like how much money is spent on materials or how many periodicals the library owns are different from outputs such as circulation per capita or turnover rate.  Combining the two makes it possible to have a library with good inputs and poor outputs score moderately well.  Conversely a library shortchanged by its community on funding that manages through good management to provide excellent service outcomes may rank more poorly than a library in a rich community with only moderately good management and output measures.  I expect to evolve the index to get closer to answering the “are you getting what you paid for” type of question.  At this point, it appears to me that 70 to 80% of the output is traceable to good input levels.  The rest is probably traceable to good management or other factors that may not be measurable.  I hope to do further investigation on the correlation of input and output some day soon. (Research firms with grant money to spare, please take note J.)

 

What does a given rating number mean, and how should I interpret it?

The HAPLR Index is similar to an ACT or SAT score with a theoretical minimum of 1 and a maximum of 1,000.  Most libraries scored between 260 and 730, so scores above and below those numbers are remarkable.  Consider the chart below for the 312 libraries in the 100,000 to 249,000-population category, for a short idea of the rating methodology.  A library above the 75th percentile for expenditure per capita of $24.70 will get a higher score on this measure than one below the 25th percentile.  Expenditure per capita is weighted more heavily than percent of budget devoted to materials.  In the HAPLR Index each library is compared to all others in its population category on all 15 measures.  The combined score is then transformed into an index score so that all can be easily compared with a single number.  For more information see the next question and follow the relevant links to rating methods.

 

Where can I see the specific calculations behind the index ratings?

An index number is always an attempt to encapsulate a lot of data into a single number.  No such index number is perfect, of course. I have provided a spreadsheet and calculation explanations on this web site for those who like to watch how numbers are crunched.   It includes only the calculations for the over 500,000 category, but the calculations for all categories are similar.

Conclusion

 

An index number encapsulates a lot of data into a single number.  No index number is perfect.  

 

Was the weather better before climatologists invented the wind chill index? 

 

Does a library-rating index change things, or, like the wind-chill index, does it just seem that way?