Chicago Migration

Last updated September 27, 2014.

The Library is currently in the midst of our migration project with the goal of bringing Kuali OLE and VuFind into production in July 2014. VuFind was introduced to the public in January 2014 as a “beta” system that was connected to the production Horizon system. It was decided to wait for the OLE implementation to make VuFind the production public interface, since that implementation was pending and it is usually preferable to implement substantial changes over the summer in an academic environment. While VuFind was essentially production ready against Horizon in spring 2014, actual migration to it as the production OPAC and retirement of HIP and Lens was delayed until summer 2014 to coincide with the OLE migration.

In addition to the internal processes to migrate systems, there is a substantial amount of staff time contributed to the open source Kuali OLE project itself. In planning for staff resource allocation, it was necessary to recognize commitment to the OLE project as well as to internal activities. Appendix A lists the required staff contributions by partner sites during this phase of the project. Participation in the project required Library staff to become proficient in use of the project collaboration tools, including WebEx, Google Docs and JIRA. Some staff were trained in creation of Selenium scripts although that effort was abandoned eventually due to the difficulty of the constant changes to database structures and screen displays during the very active development cycle for version 1.5. The scripts could not be maintained until more of the development was complete and the User Interface was changing less frequently. During this part of the development the OLE central project, QA staff were charged with doing the scripting where feasible.

Functional Migration Activities

Last updated September 27, 2014.

To facilitate the Library’s migration process, the University of Chicago Integrated Library System unit (ILS) established an ILS Migration Steering Committee (IMSC) made up of key decision-makers in the Library. The IMSC reports directly to the Library’s Administrative Committee (AdCom); two members of the IMSC are also members of AdCom. The IMSC’s charge and duties are as follows:

IMSC, under the guidance and direction of ILS, exists to make decisions about any matters pertaining to the migration from Horizon and Millennium to Kuali OLE. These matters include, but are not limited to:

  • pre-migration data cleanup
  • data mapping from old to new systems
  • data archiving of historical information NOT to be migrated to OLE
  • OLE configuration settings
  • testing preliminary and final release versions of OLE
  • developing training programs for staff
  • training staff
  • scheduling
  • internal communications

IMSC is the decision-making body for the migration process as a whole and will meet as needed to discuss cross-functional concerns, hear status reports from lead members for the various functional areas, and/or resolve specific functional issues referred to it. It, in conjunction with ILS, will identify and assign migration-related tasks and projects. The IMSC will document its activities on Staffweb and/or Basecamp and provide regular status reports to AdCom (schedule TBD). AdCom provides IMSC with input on any general matters referred to it, and may raise questions, intervene on any issue, or give directives as needed.

IMSC members acting as leads for a functional area are expected to speak for all staff stakeholders in that function (regardless of work unit). As such, each lead member will form an informal subgroup of stakeholders for the purpose of soliciting input and reaching decisions on migration matters related to the functional area. Should a subgroup be unable to reach decisions on any matter, IMSC as a whole will make the decision. Subgroups will meet on their own (with the ILS liaison) as needed and may be invited to meet with the full IMSC as required.

IMSC will also coordinate its work as needed with the Discovery Tools Group as it works to implement the Library's new user interface in conjunction with OLE. The Web Program Director as an IMSC member-at-large will be the Group's liaison.

Following OLE implementation, it is envisioned that IMSC will be replaced by a group to advise ILS on implementation of new OLE releases, define and prioritize enhancements, etc.

Duties

All members will be expected to:

  • attend meetings (frequency to depend on nature/scope of pending/current projects)
  • respond to questions, make recommendations, etc., as requested by ILS concerning specific matters related to the migration
  • assist ILS in communicating major decisions to staff
  • perform and/or coordinate assigned tasks

In addition, lead members for a functional area will be expected to:

  • represent the requirements/concerns of staff stakeholders of the functional area (regardless of library work unit)
  • recruit staff stakeholders
  • lead discussions with stakeholders on migration matters and make decisions as required
  • perform and/or coordinate assigned tasks such as data cleanup, review of data mapping documents, test software, etc.
  • determine training requirements for the functional area and assist ILS in identifying trainers and developing training classes

“Lead members for a functional area” refer to those IMSC members who lead “working groups” in the following areas:

  1. Cataloging
  2. Acquisitions
  3. Circulation
  4. Serials Receiving
  5. Reporting

The working groups’ members are key staff members in their respective functional areas and work with a designated ILS staff member on any matters related to moving data and operations from Horizon and Millennium to Kuali OLE for the respective functional areas. VuFind implementation is managed by a separate group that includes technical support from ILS; insofar as a migration of VuFind from Horizon to Kuali OLE is concerned, ILS’ role is primarily technical, although the circulation working group will be involved in testing “my account” functions when these are moved to VuFind/Kuali OLE.

Most of the actual work of performing gap analyses, testing, developing training, etc. is done by members of the IMSC working groups. It is important to note that since ALL Library units use one system, we have explicitly made clear that the working groups “represent the requirements/concerns of staff stakeholders of the functional area (regardless of library work unit)”. This is an effort to make clear that while individual library departments performing the same function (e.g., the Library has four cataloging departments) may have differing policies or procedures, use of the ILS must be the same.

One of the most important tasks of the IMSC is to ensure that staff receive the necessary training in Kuali OLE that includes both the instruction on how to use the new ILS AND any changes to policies/procedures/workflows that the new system will require. The IMSC has adopted a training plan included as Appendix B.

Details on our migration – along with links to the IMSC charge, training plan and other documents – can be found at: http://www.lib.uchicago.edu/staffweb/depts/ils/kuali/index.html.

Data Migration and Integration Issues

Last updated September 27, 2014.

For bibliographic and holdings data migration, the fact that the library was coming from a non-Unicode system required additional work. Kuali OLE also introduced a new type of holding called an EHolding record. Over time the library had dealt with electronic books, journals and other resources in a variety of ways. In some cases there were 865 fields in the MARC records, and in others there were also copy or item records. When we went to represent these materials in VuFind, it was decided to take advantage of the migration to OLE to regularize how we represented these materials by always using an EHolding record in OLE.

There were some special problems in data conversion due to the fact that we were moving from two separate systems, Horizon and III. These required custom data conversion scripts developed by in-house programmers.

MARC Authorities records were not planned to be supported in Kuali OLE until after the implementation of the early adopter sites. For a number of years, the library had sent new bibliographic records to Backstage Library Works and received corrected bibliographic and new and corrected authority headings. It was necessary to adapt this practice to continue to receive corrections for bibliographic records for loading into Kuali OLE, but to load authority records only into VuFind where they are used for cross references in the browse indexes.

Certain other features were not planned to be part of the initial Kuali OLE release and required that other local custom helper applications be developed. So, for instance, the Library used the spine label printing in Horizon, although not all libraries use that feature of an ILS. Also tracking payments accepted at circulation desks had some limited support in Horizon that was not in the new system, so some supplemental helper applications were required to support local workflows. The Library had developed many customized reports by pointing MS Access databases at the Sybase Relational Database Management System (RDBMS) of the Horizon system. Gradual adaptation of these reports to the Kuali OLE RDBMS is expected to take some work by the staff involved.

In addition, there are critical integrations required for the system to be functional. Specific Kuali OLE application programming interfaces (APIs) were developed to allow integration with the Dematic Automated Storage Retrieval System used in the new Mansueto Library. It was necessary to contract with the vendor for them to rewrite their side of that integration. The Library had moved Course Reserve functions to the Atlas ARES system and written a custom integration for the “place on reserve” and “remove from reserve” functions. Kuali OLE docstore APIs will be used to replace the custom integration at the RDBMS level used for Horizon. We are working with Atlas to implement this connection. NISO Circulation Interchange Protocol (NCIP) messaging to support participation in the UBorrow and Borrow Direct projects were also necessary parts of the project and require testing with the vendor systems. Finally, the ability to continue to extract payment information and format it correctly to load to the University Comptroller’s system in order to pay our vendors was another critical integration that is required for going into production.

It was particularly critical that the VuFind Horizon connector be replaced by an OLE Connector in order to have the public catalog work correctly and provide the My Account features to allow self-service requesting and renewals. Because Villanova, where the primary VuFind development is done, was a Kuali OLE partner, they provided the OLE Connector based on the OLE APIs developed for this purpose.

A Basecamp project was used to track work on these various projects and a snapshot of that project is provided in Appendix C.

Technical Migration Activities

Last updated September 27, 2014.

Staffing

The Integrated Library Systems group included a manager, Library Systems Analyst, Systems Librarian, Database Administrator, Senior Programmer Analyst and Library Operations Assistant, who also did some web programming. To accommodate the need to support the open source systems, another entry level Programmer Analyst position was added to help support VuFind and a Senior Programmer Analyst with Java skills was hired to help support Kuali OLE. This staff would support the migration, as well as the integrations with other systems and the add-on custom applications needed for optimal use of the system. In addition, web programming staff and the Web Program Director for the Digital Development Library Center were used in the VuFind project to customize that system.

Hardware/Software

While a number of the commercial library systems are moving to “cloud-based” systems, there was no real impetus to consider such a solution at UChicago. Indeed, in a university setting there can be obstacles to such an implementation. For instance, during the course of the project, Lehigh University – another Kuali OLE partner – made a decision that university financial data should not reside in the cloud. Chicago had not made any such general policy, but major system implementations do require security and architecture reviews. Patron database information in particular would be problematic to be made available on a commercial, vended system. VuFind and OLE will be implemented on virtual servers hosted by university computing. In the future, if the university offers cloud-based hosting services, it will be possible to take advantage of that. OLE itself is being developed on equipment that is in the Amazon cloud, so it is demonstrated that it can be run in that environment. At the moment, universities have some legal reservations about agreements to run on cloud-based commercial systems. It was seen as reasonable for the Library to follow university policies in this area and not to attempt to run the library systems separately. The intention is to take advantage of the university enterprise systems for storage, backup and system administration.

Plans for migration of the library system will undergo a review by the university ITS Technical Architecture Committee and also a security review. Appendix D contains the representative list of questions for these reviews. A separate PDF is attached which contains the diagram that the Library provided for that review.

A basic difference in implementing open source software is the need to pull down source code to a development environment and to develop a process to deploy new versions and fixes. This is true for both VuFind and Kuali OLE. This required some upgrades of equipment in the library for development and testing before deployment to production environments in the university data center.