evergreen-ils blog

I have been stewing about my maiden post on this blog for some time, so here goes.

I presented or talked about Evergreen and open source seven times at ALA Annual 2008. (Seven times, I tell you, seven times!) It was a great experience, and part of the joy came from the many moments when the heads began bobbing. (No, not nodding off to sleep… well, maybe for one tired conference-goer… but actively bobbing in affirmation.)

I would say “No hidden code” and heads would bob.

I would say “Free to use, free to download, free to examine, free in every way” and heads would bob.

I would say “Interoperability” (which open code facilitates, even if it is not a characteristic of all open source) and heads would bob.  (Including the heads of many potential vendor partners.)

I would say “Reindexing, transaction load, deduping, powerful, avenue for growth” and heads would bob.

I would say “This brings us full circle to where we were in the beginning of library automation, when we steered our own ship,” and heads would bob.

Evergreen community members can’t always see the bobbing heads. But they are there all the same.

Happy 4th! (And belated Canada Day, not to mention the 400th birthday of Quebec!)

In A Riff on Big, I dealt with the skewed nature of library distributions in the United States (although the finding is more generally applicable)—that is, there few big libraries and many small libraries. Moreover, the big libraries are really, really big and the small libraries are really, really small.

However, the critical question is not this fact but its implications for a national information policy. In a consential democracy with the foundational principle of an informed citizenry, any information policy has to consider the disparate sizes and resources of the nation’s libraries and how to ensure that those with access only to the smallest libraries can have access to greater resources—as great as is possible.

With all that in mind, Karen Schneider asked me if the size of libraries in square feet was known? Well, actually, it is, and I produced a list for her of the PINES libraries with the size square feet for each outlet running Evergreen. Hmmm. Interesting. The summary statistics, as well as sources are included below but what do they mean?

The conventional model for allocating public library services has several layers. Independent libraries may operate independently with resources available at worse through Ill. In larger settings, branches will draw on a central library or, even larger, regions of independent systems draw on the resources of the region. Until recently, that is what was possible.

The PINES system allows the libraries in all PINES libraries in the system to draw on all other member libraries. The system includes about 2 million bibliographic items and 9 million or so physical items. This arrangement of resource sharing is not common nationally. That fact may be because a consortial library system was not available until recently.

PINES, then, has made more resources more available to more smaller libraries than the conventional model could if applied in Georgia. This is an important step forward and one that has profound information policy implications.

I conclude that the conventional model has served its function as well as was possible before the development of software capable of managing resources of a large consortium. If resource sharing is important, recreating such conventional systems today would, at best, be doing the wrong thing well.

Some Data

Here are some summary stats for libraries reporting this number to the US (N=15,858) and for PINES (N=243) for FY 2005:

Summary statistics of library outlets by square feet of space

                         All                           PINES
                         U.S.                           # by
                      Quartiles                    U.S. quartiles

Fourth           Greater or equal to                     59
Quartile         12,000 sq. ft.

Third            Less than 11,999
Quartile         and greater than 5,314                  62
                 square feet

Second           Less than 5,313                         80
Quartile         and greater than 2,234
                 square feet

First
Quartile         Fewer than 2,233                        42
                 square feet

Note that the number we normally see of the PINES libraries gives the aggregated figures for each of the (now) 48 systems but this number is considerably disaggregated. Let’s look a bit more.

Smallest library outlets by square footage

                         All                            PINES
                         U.S.                             #

Smallest          less than                              20
10%               1,120 sq. ft.

Smallest          less than                               9
5%                784 sq. ft.

Smallest          fewer than                              4
1%                384 square feet

Note that these categories are not mutually exclusive and that each interval is included in the interval above.

Source of data

The data come from the FY 2005 Outlet File published originally by the U.S. National Center for Education Statistics and now available on the Website of the U.S. Institute of Museum and Library Services. The outlet file has an observation for 17,299 outlets. Appendix J of the documentation reports that 16,557 responded and 44 did not respond but had data from the year before. I found some responses were negative (a conventional way of representing such things as “not applicable” or “unknown”) and deleted those. Bookmobiles apparently do not conventionally report square footage, at least the PINES libraries do not, thus leaving 15,585 outlets (branches and central libraries) with reported figures.

That this number is from FY 2005, the latest national level data we have, and facts will have changed since that survey.

Bob Molyneux

In an earlier post, I reported on some figures measuring the then maximum transaction load levels on PINES. Two of those were from the day after Memorial Day in 2007 (that is, May 29, 2007). What happened in 2008 the day after Memorial Day (May 27, 2008)? Well, we have another busy day.

                          May 29, 2007                               May 27, 2008

Total circs                 96,326                                     100,427
per day

Maximum circs
per hour                    11,305                                      12,227
                          (11AM-12PM)                                 (11AM-12PM) 

This year’s total figure is 4.26% over last year’s.

The busiest minute in 2008 was 331 transactions while last year’s maximum figure was 548—although it was a day earlier in the month.

What was the busiest second? Why, I am glad you asked. There were three seconds during the day when 9 items circulated.

Bob Molyneux

I have long puzzled over the fact that in the golden age of the library function, there has been a systematic failure of library institutions.

By “library function” I refer to the functions and institutions that maintain the memory of the human species. This function includes those institutions and people who save, organize, and provide for eventual retrieval the human record. In spite of the fact that what we do is often ignored or disrespected, it the genetic trick as a species. It is a key aspect of what we are built to do. We are not the fastest species nor the strongest. Smartest? Roll your own wry comment on that. But we remember and we do it with many institutions I lump in the library function: libraries, archives, museums, and so on, with apologies to the archivists.

In my lifetime, the human record has starting moving from paper, stone, vinyl recordings and so on to digital forms. When I was graduated from library school in 1971, the importance of information as a thing would get a perfunctory: “yeah, yeah.” Now, everyone knows.

The failure of institutions of our field to adapt to this new information environment is a calamity for the species and one that is being remedied by bypassing the institutions that, traditionally, have done the library function. There is an old saying that “the Internet routes around trouble”–an expression that indicates a foundational principle of the Internet architecture. I believe that with so many things going on in the information world, we can see the human species routing around the traditional institutions because there are increasingly more efficient methods for remembering, organizing, and retrieving the human record—which we humans cannot not do–and which these traditional institutions do not do well. Not all people are routing around all library institutions. Not everywhere. Not everyone. But many–perhaps most. Librarians did not invent Yahoo nor Google. We made the sale about the importance of information but couldn’t consummate it. We have much to contribute to this new information world but we are largely failing.

I was forced to consider this dynamic again recently in a microcosm of the general problem I have outlined. What restarted this whole train of thought was a report that discussed integrated library systems in a specific context. I apologize for a special pleading but it was both inaccurate and sad. There was no aspect of this report that struck me as informed and the people who paid for it are in peril and they don’t know it. The recommended solution for their problem WILL NOT WORK because the recommended technology won’t do what the marketing folks said it will do and the person who wrote the report, I infer, was none the wiser. You would think after the events of the last year or so, library decision makers would have wised up. One would suppose that by now librarians would have their defenses up to resist vaporware and the blandishments of the slick.

I think one problem is that we do not have a critical mass of technically trained or technically astute people in the field nor an assessment culture. In the librarians’ house, there are many mansions…just not a big enough IT one. And the ones we have…well, you know how they are…not like us…difficult. We treat them like the Dilberts of the library world because too many library decision makers do not understand much about technology.

Years ago, Kathleen (de la Pena) Heim, in a different context, made the perceptive observation that we “need to recruit a new us.” We didn’t in information technology. As a result, we certainly do not have enough IT folks with an understanding of the library function in the field and information seekers are routing around us seeking information.

In many fields with a requisite critical mass of technically-trained people, there is an assessment community. PC Magazine, Ars Technica, and other such sites provide independent assessment of computers, hard drives, thumb drives, and so on. I remember reading magazines that discussed “hi-fi” and reviewed equipment back when I could hear. There are magazines like Consumer Reports for consumer products. These days, one has sources like the reviews and comments at sites like NewEgg.com if you are purchasing computer equipment and discussions there are often lively but usually informed.

In the library world, we have a blizzard of marketing twaddle about technical subjects and a host of people who comment but too few with the ability to pierce the marketing veil and tell folks in the library community what these various products will and will not do based on an accurate understanding of the underlying design concepts and their execution. We do not have an assessment community. How many IT failures based on librarians believing impossibilities do we have to endure?

Boswell quotes Samuel Johnson as saying: “Depend upon it, sir, when a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully.” Except in the library world.

Bob Molyneux

I have alluded in several posts to the disparities one finds in looking at distributions of library data. By “distributions” I am talking here about making observations and generalizations when one looks at all the data from a set of libraries.

I am going to discuss a fact of library life we all know about, give you a few numbers, and discuss a few implications. I take circulation figures (TOTCIR as NCES calls it) for all public libraries in the United States for fiscal year 2005. These are the latest national-level data we have from NCES. I use the data as I recompiled them in a dataset that began when I was at the U.S. National Commission on Libraries and Information Science. I have continued to update that series since then. Exhausting documentation exists on that site.

In that year in the dataset, there were 8,957 libraries and they circulated 2 billion items. Of those total circulations, 1.8 billion were reported for the highest quartile. That is, the 2,240 library systems that had the highest number of circulations accounted for 86% of all circulations. Those libraries in the first quartile, that is, those 2,240 with the lowest circulations had 13.9 million circulations or fewer than 1% of the total. This relationship—the big are awfully big and the small are really small—is an observable fact in most variables (staff, income, expenditures, holdings, and so on) and in every universal library dataset I have worked on. It is a characteristic of our national library system. The term in Statistics to describe this kind of distribution is “skew” and library distributions typically are skewed.

There are two kinds of implications, I think, to this characteristic. One deals with information policy and the second with the design of integrated library systems. Considering the information policy implications first, there are, it seems to me, three aspects that arise when we consider that the structure of library resources in this country are so disparate.

First is the effect of differential resources in a consensual democracy where an informed citizenry is a foundational element.

Second, what effects do differential resources have on the post-Enlightenment notion that was particularly important in the history of U.S. libraries: the library as the university of the common man? This question still is important because of the necessity of continuing education in an era with so many dynamic changes in our economy and where people need to retool for new kinds of jobs.

Third, given that the small libraries are so small, their staffs are also small. Andrea Neiman, of the Kent County Public Library, Chestertown, Maryland reminded me the other day when we were chatting about the implications of skewness of something important, that is, what we know about the small libraries which is not much. Consider: the top quartile of public libraries employs 85% of all public total full time equivalent staff while the bottom quartile employs fewer than 1%–numbers similar to what we saw with circulations, of course. The upper boundary of this quartile is less than 1.2 FTEs–the rest are smaller. Thus, these libraries will not likely be adequately represented at conferences nor in decision making bodies.

The policy implications are for another place and time, of course, and we all know of the halting attempts to address this problem, that is, to breakdown the information silos faced by users of libraries.

A second implication of the skewness has to do with the design of library automation systems. They have handled the fact of the distribution of library resources awkwardly and that, in turn, follows from the unsystematic way these resources have been designed traditionally. I will leave to my colleague Mike Rylander to discuss what he has explained to me about how ILSs were designed but there was market segmentation: big ILSs and small ILSs as a result of the limitations of early design decisions and capabilities of the ILSs. The influence these design limitations had on information policy would be a fascinating subject to explore.

In any case, Evergreen is, currently, unique in that its design encompasses very big to pretty small libraries and its ability to handle diverse consortia is also unique—and valuable as I have discussed in several previous posts. The PINES experience indicates that good design can address the information policy aspect of skewed library distributions.

Bob Molyneux

No, despite the title, this isn’t a belated April Fools joke.

We had around 50 people descend on the Marriott City Center during PLA for the Evergreen Birds of a Feather gathering. Evergreen stakeholders seemed to be well distributed around the room and it was just an informal gathering with food and much chatting. My only regret is that I neglected to bring a camera. I know I got a lot out of it, and it was very nice to put faces to names. I do think that next time we should also offer a few structured activities to compliment the social aspect. If anyone has any ideas or comments, feel free to throw them out here or on the mailing lists.

Thanks folks!

– Jason

Much ado has been made about Evergreen’s ability to scale up with both its service-oriented architecture and consortia-savvy interfaces, and as folks witnessed at PLA, it can also scale down to the size of a laptop. Well, now we have really done it. We have taken the n-tier concept to the extreme and have introduced fractional-tiers! Yes, you can now partition your data into quanta, and from base axiomatic principals truly grow Evergreen into a fractal framework that will help gestalt your libraries into the next millennium. It’s no longer about consortia; instead, it’s all about personal digital appliances and technological augmentation. You can take an Evergreen data seed, and install it on, say, an mp3 player or cell phone, or, if you want to really be on the bleeding edge (literally), you can plant an Evergreen data seed into a biochip and have it surgically implanted into your neocortex, freeing you forever from the confines of conventional search and discovery interfaces. Now you can carry your data with you! If you could put Google into your brain, would you? Well, this isn’t Google, but it can be your library!

Welcome to the brave new world of Evergreen!

– Jason

Yesterday I was explaining to some librarians how Evergreen relates Items (the actual physical barcoded material that circulates) to Volumes (where the Call Number lives) to Bib Records (which contains the MARC), and one person was curious and asked, “why do you do it that way?” The short answer is because it’s good design, but the question momentarily threw me for a loop because the implication is that they’re used to systems which do not do it that way.

Imagine that the structure used for items in the database for your ILS is laid out something like this:

Item Table
———-
Internal ID
Creation Date
Barcode
Call Number
…other fields

In some automation systems, that Call Number field will be a free-text label. If you want to change the Call Number for an Item, then that field will necessarily get changed. If you want to change the Call Number for a large grouping of Items, then you will have to change the field for all those items (sometimes one at a time).

In Evergreen, that Call Number field will actually be a foreign key to another table in the database, one that looks something like this:

Volume
——-
Internal ID
Owning Library
Call Number Label
…other fields

This allows a Call Number to be shared amongst a group of items, and be modified with a single edit.

This is also an example of database normalization.

Why is this important? It has to do with redundancy of information (and I’m not talking about backup storage, RAID’s, etc).

Human beings love redundancy when it comes to communication; our meanings get emphasized by our body language, and the syntax and structure of our languages, both spoken and written, encode information in multiple ways. But redundancy opens the door to discrepancy, and discrepancy leads to ambiguity. Human beings can deal with ambiguity, but computers (and most software developers, and maybe catalogers) can’t abide it. “Time flies like an arrow, but fruit flies like a banana”.

A computer might resolve ambiguity in a non-obvious way, or worse, it may resolve ambiguity in a manner that coincidentally matches a human’s expectation, only to cause trouble later when you start getting data anomalies.

Normalization is a best practice design technique for designing relational database schemas, though there are cases when you want to make a trade-off and de-normalize certain data structures. But those cases are usually optimizations that should occur after you already have a good design as your starting point.

Relational databases are powerful enough that they have sunk into the consciousness of librarians, to the extent that they put down “Must be built on a relational database” in their RFP’s. But most legacy ILS products actually use hierarchal databases, which are good for some things, but not for others. The design decisions made when you go with a hierarchal database are very different than the ones you make when you start with a relational database, and I worry about the legacy products that have “tacked on” relational databases for buzzword and RFP-compliance. Legacy automation may now have relational databases, but are they actually using them as a relational database? Some are, but I know of at least one that isn’t.

Let’s return to Evergreen for a moment. Because Call Number Volumes are represented as their own entities, you have the option of thinking of them differently. For example, you can move a set of items to a different “call number”, one which may be associated with a different Bib Record altogether (or you could move a volume itself to a new bib record). You could even change which library “owns” a volume, and suddenly all the items attached to that volume have a new owner. And you’re able to place a hold on a specific “volume”, in addition to title-level and copy-level holds (and in Evergreen, meta-record holds across editions and formats).

But you can also do what you may already be used to, and change the Call Number for an item (or a batch of items) from an Item Editing interface, and not have to know about the more flexible structures that are being manipulated underneath.

Because we already have Volumes, we’re also in a better place for adding Serials.

There are fewer widespread repercussions to changing an end-user interface than there are with changing your database schema, so that’s why it’s important to have a good database design from the beginning, so that your interfaces have more options. To steal a sentiment from Mr. Miyagi, Evergreen has strong root.

– Jason

First I’d like to welcome NewGenLib to the virtual family of FOSS ILSs. In truth, we’ve known about them for a while and have been looking at their serials interfaces during our ACQ/SER design, but now that eIFL is covering them, well… It’s great to see another entrant, and one that has already found an itch to scratch. I’m sure cross-pollination is in the stars as they seem to have an interesting system.

Next up, a pair of kudos to Koha.

Over the past weekend they added, at a mailing list member’s request, a call number browser inspired by Evergreen’s, which we call Shelf Browse. In Evergreen, because it supports a hierarchical organization of libraries, you can actually browse an entire system or even consortium as one huge virtual shelf! It’s a very nifty feature, and one that we know the PINES patrons have been making good use of (to the tune of 66,965 and counting so far this year, and about 300,000 times in 2007) since Evergreen launched in September of 2006. Now Koha will have a similar feature at the request of a small church Library! This, my friends, is Open Source at work.

By way of evidence from our users, I’ll mention that Evergreen provides call number / shelf browse as a “Quick Search” from the advanced search interface, which is useful to Evergreen users and may be useful for Koha patrons as well. In any case, good work.

I also noticed that Koha has incorporated, as of November of last year according to their source repository’s timestamps, the SIP2 code that David Fiander and Bill Erickson wrote for Evergreen. We’re glad to see the code that GPLS funded is going to good use in and inspiring other projects!

Three and a half years ago, when I first joined PINES and the Evergreen team, there was a dream and a small test server. Now we’ve written more than a quarter of a million lines of code, and that code runs the day-to-day operations of one state-wide library consortium (biggest in the world, he bragged ) with at least two more in the works, and is helping to build a province-wide consortium in Canada — and let’s not forget the Laurentian/McMaster/Windsor “Unholy Trinity.” These are amazing, and they fill my heart with a satisfaction that is difficult to describe, but none of those things, even as possibilities, are why I signed up. I joined this effort because I believe in Open Source software. I believe whole-heartedly that it is a force for positive change in an industry I love, and fits perfectly with the mission of libraries.

Again, congrats to both NewGenLib and Koha, and let’s keep the cross-pollination going.

–miker