Chapter 2. Triage and Immediate Challenges

The first task in either the lead up to, or after the onset, of a crisis situation is assessment. The extent of the crisis must be assessed and, as much as reasonably possible, the implications of the crisis extrapolated. There will be things that get missed at this phase . . . it is simply impossible to predict everything, although experience with crisis management is helpful. With luck and foresight, there will be some existing disaster planning in place to use as a guide. If such is not the case but there is time to plan before direct action is required, putting together a list of considerations and a game plan is very useful. In those cases requiring a more urgent reaction, assessing the totality of issues and concerns may need to happen concurrently with triage.

Triage represents the first practical steps taken to ameliorate a crisis situation. Triage requires action to be taken on the most critical and immediate problems while other, less critical problems are then queued for later action. It is at this point that the issues that arose from an assessment of the crisis are ranked based on how immediate a problem they are, how serious a problem they are for the library overall, and how much effort will be required by the existing staff to solve them. This provides an IT department with a basis for prioritizing work. Generally, there will be immediate work, such as re-establishing certain critical systems, or an emergency issue or reissue of equipment, which will take precedence. Once the most urgent work is done, a library IT department can then proceed to tackle the rest of the queue.

Generally, of most urgent concern will be critical systems that must be restored. In those libraries I worked in, the most critical systems to recover were the website and EZproxy (or whatever proxy tool is being used to facilitate access to the library’s digital resources). However, there can be complications to standing these systems back up. At K-State, after the fire, many university systems, including identity management, were knocked out. Without an effective authentication method, it took some time to stand our EZproxy instance back up. However, this became an opportunity. At that point EZproxy was one of the few systems we had not migrated from the university data center to Amazon Web Services (AWS). Since work had to be done to reactivate EZproxy anyway, it proved to be a propitious time to migrate to an AWS virtual machine and replace the old LDAP authentication with Shibboleth authentication. By migrating our proxy instance, we were able to get it back online more rapidly than otherwise would have been the case, since the systems from identity management were among the first recovered by K-State central IT as it carried out its herculean efforts to respond to the fire. Other hosting services took longer to recover. Close coordination with acquisitions was required, however, so that vendors could make the necessary IP address change as rapidly as possible.

Another immediate concern is communications. Depending on what happens with local systems, communications can rapidly become a challenge. At K-State, once the identity management systems went down, university users lost access to e-mail and the university’s Rocket.Chat application. It became imperative to re-establish communication, using the oxygen mask model (take care of yourself first so that you can help others). Library IT first re-established intradepartmental communication via a free Slack account before standing up a copy of the internal library wiki on Amazon Web Services that could be used for both broadcast and two-way communication by employees.

For many crises immediate concerns will include equipment issues. In those cases requiring employees to change locations (as happened with both the K-State Hale Library fire and the COVID-19 conditions at Washington University Libraries), there will generally be a need to distribute, and potentially procure, equipment on an expedited basis. In the case of the Hale Library fire, many computers were rendered inoperable or inaccessible . . . and often both. The libraries purchased a large number of new laptops to be distributed to employees.

In the case of the COVID-19 crisis, the university ordered departments, such as the libraries, to send employees home within a specified time frame, but staff computers were still serviceable. Policies had to be quickly established to determine if staff could take any office or desktop equipment home, while those employees who needed them were provided checkout laptops and laptops from the now-unused computing labs for home use. In the case of both crises, however, a process needed to be put into place to rapidly reimage the computers that needed to be issued. A system of distribution was planned out as well, with appointment-based distribution proving to be the most effective model. At K-State, based on a suggestion by one of the developers, the entire department set dates to come together and assist the desktop support team with the imaging of devices in an assembly-line arrangement that had been expertly conceived and mapped out by the desktop support supervisor and her team. At Washington University Libraries, where it was a priority to send staff home as soon as possible and the volume of devices to reimage was lower, the equipment support unit ably handled the reimaging process and worked together with its reporting chain to establish scheduling for those staff identified in the organizational planning process as needing a laptop.

Due to human factors, distribution always poses at least a limited challenge. Inevitably some employees will miss, ignore, or skip appointments, causing congestion to a schedule that may already be severely compressed due to circumstances such as the required compliance time line we faced at Washington University for leaving the office. In some cases this will necessitate a backup process for distribution, in which drop-ins who miss the established schedule are required to collect their devices in the background, with any additional setup help they need being provided later through remote assistance.

As with any expedited process, the assistance of other departments is crucial to success. It is departmental leaders who will need to work with their staff to help them make appointments and comply with organizational needs. This is especially true with the accountability aspects of issuing equipment. At Washington University Libraries, the IT department didn’t have the staffing or the time to track any equipment that staff took home from the office. Department heads were tasked with tracking the disposition of this equipment using spreadsheets provided by the IT department.

Environmental factors must also be taken into account . . . especially when handling procurement. If new computing devices and peripherals are being acquired through the university, there may be trouble in having large orders filled immediately (especially if other departments have also been affected by the crisis and are simultaneously pursuing equipment procurement). Shipping delays may cause further issues (a situation seen as the COVID pandemic matured). In the case of the COVID-19 situation, while Washington University Libraries initially made use of existing computers, the IT department did acquire a small number of cellular hot spots to assist staff who did not initially have internet access. Starting in the early days of the pandemic, stocks of many devices that facilitated remote work quickly dried up, making finding the hot spots a time-consuming challenge. As staff were reintroduced to the library and the need for webcams and similar peripherals increased (since staff desktops generally did not have the webcams needed for them to participate in the now ubiquitous Zoom and Teams meetings), it became difficult to find those models of equipment that had been ordered in the past. At such times, as new models are acquired, IT staff must exercise caution. We at Washington University libraries experienced a few incidents of items proving to be of low quality or possessing baffling limitations (including an order of webcams that did not have built-in microphones). Specifications must be scrutinized carefully, preparations made for inevitable returns of some products, and pricing expectations adjusted for a shortage environment.

As new equipment is issued, staff will still require access to specialized applications they utilized prior to the crisis. This is especially the case with those employees using platforms that require old-fashioned local client applications, such as ILLiad, or some of the older catalogs. Often installing these packages can prove to be a significant time commitment. Another way to handle this issue is to set staff up to remote into their work desktops (where such desktops still exist and have power). This was the primary method used at Washington University Libraries. All library staff were added to the library VPN at an early point in the COVID crisis, allowing them to reach their work computers via remote desktop. This mechanism generally worked well, although occasional power outages and one update issue that affected a handful of computers required library IT to work with the university IT teams that were allowed to operate on campus (and later with library building services) to physically restart desktops as needed. Such remote connections always have at least some intermittent problems, however, and providing assistance to staff experiencing these transitory issues as they cropped up became a regular feature of desktop support.

In the case of Washington University, where office devices were still functional, employees were given an opportunity to take a limited amount of equipment home, as has been previously noted. This was permitted only with supervisor approval and if employees chose to accept responsibility for their equipment. Additionally, since this was a remote work environment, those individuals who elected to use their personal computing devices were encouraged to do so as per university and library policies for remote work.

One technical issue not to be overlooked is the data in the library catalog. In the case of the COVID-19 pandemic, most of the general information was unchanged, except for the temporary addition of some information through a sideways web development process, which is discussed a bit later in this report. In the case of K-State, however, the libraries were concerned that individuals might mistakenly try to enter the library in the belief that they could find the book they needed there, despite the building being shut down. Work was done to temporarily remove the Hale Library location from all physical media catalog entries and indicate that items from the Hale Library location were temporarily unavailable.