Koha Development for CCFLS

Last updated September 27, 2014.

To fix the scalability problem caused by the MySQL bottleneck, in 2006 CCFLS wrote an LSTA Grant and subsequently hired a third-party company, LibLime, to integrate Index Data’s Zebra indexing engine into Koha. While Koha's indexing engine was being rewritten, CCFLS designed a new staff interface for Koha. After a series of weekly meetings between the implementation staff and CCFLS librarians, we rewrote templates from Nelsonville Public Library to create our own look and feel to Koha. Our on-staff developer also wrote new features for Koha to make it better suit our needs, such as a new system for calculating fines and a custom reports module.

MPL is accustomed to contractors taking longer than projected to complete projects. We had received regular updates and progress statements from LibLime, and we were aware that the vendor was some months behind on the project. However, the last progress milestone was the delivery of usable software, which was tied to the last payment. LibLime informed us that the Zebra indexing was ready, but the search functions within Koha needed refinement. LibLime suggested that we join them and other Koha developers from around the world in Marseille, France, in May 2006 for a week-long hackfest to finish the work. Since we wanted to begin the migration process, it was decided that the CCFLS management and IT Department would go to the hackfest. With less than a month's notice (and two MPL staff members needing to quickly acquire passports), we were on our way to the first Koha Hackfest.

As far as problems are concerned, spending a week in France is one that most of us would like to have. However, the Koha Hackfest was all business. Over a period of five days, 10 software developers spent nine hours a day debugging, rewriting, teaching, planning and documenting what became known as the dev-week version of Koha. Additionally, our developer received an intensive education in Koha’s codebase. We also enjoyed many fine meals – the Chinese/Vietnamese restaurant with mint egg rolls is still memorable almost a decade later.

The dev-week Koha version program integrated Index Data’s Zebra indexing utility eliminating Koha’s MySQL bottleneck. This modification significantly increased the speed of Koha for large libraries and library systems, and it allowed libraries with collections of more than 4 million plus items to use Koha. This new version, officially dubbed Koha Zoom, was first implemented in a public library in Western Provence, France in January 2007. Even though we paid for the Zebra integration, we were not the first library to use the upgraded Koha. However, implementing it in France allowed Koha’s developers to fully test and debug it before making it available to us, which we found to be an acceptable tradeoff.

Using dev-week Koha, we then loaded our data onto an old server and began running tests. Using current circulation data, we ran a number of dry runs to test the migration process and to have a fully operating ILS available for staff to train and test. During this time, Cindy developed the parameters for data migration and Kyle wrote the migration scripts. For the data that we couldn’t export in csv or MARC format, Kyle wrote scripts to extract the data from reports we ran in Winnebago. These scripts are still available at http://kylehall.info/index.php/projects/koha/koha-tools-download/. Cindy used tools such as xml2marc, MARC::Lint, yaz-marcdump and MarcEdit to clean up the MARC records exported from each CCFLS library’s Winnebago database.

At the same time, a team of CCFLS librarians and staff examined the migrated data for errors, tested the operation of Koha, and made decisions on changes to the code that needed to be made. We used the Mantis bug tracker to track issues and development. All software changes were managed locally in git and then merged with the code base from Koha dev-week. One issue discovered during this process was that Winnebago had a basic non-MARC editor for adding and modifying materials, while Koha had a full MARC editor with the possibility of using multiple frameworks. Although Winnebago's basic editor was great back in the 1990s when our rural librarians didn’t need knowledge of MARC in order to catalog, we discovered that there was little adherence to cataloging rules outside of the two larger libraries in the system. This discovery resulted in CCFLS having to do database cleanup on nine separate MARC databases plus training staff on proper MARC cataloging.

At this part of the process, we found that the amount of clean up work plus additional training of staff was going to be huge and more then what our small IT Department staff could handle all at once. One of the benefits of OSS and hosting your own solution is that you can dictate your schedule to match the staffing resources you have. By spreading the implementation schedule over a year, we took what would have been a huge job done all at once into manageable chunks. We decided to migrate Meadville Public Library first, working out any major bugs that came up before beginning to migrate the rest of the libraries one at a time.