For the first part of this study, the decision was made to focus on Atrium, the Université de Montréal Library catalog, as a case study. Research was further limited to monographic titles by selecting entries found in the following MARC 21 fields: 240, 245, 246, 700, 710, 711, 730, and 740, thus excluding series titles. Time and money constraints made excluding them from the sample necessary.

To meet the first two objectives, the following research questions were formulated:

• Question 1: Which of the articles on Atrium’s exclusion list have the most entries beginning with that string of letters when not used as an article?
• Question 2: What proportion (i.e., number of records with affected titles divided by total number of monographic records in the database) of records is affected by the deficient retrieval algorithm in Atrium?

Data collection began by identifying the 41 articles in the exclusion list used by Atrium’s initial article detection algorithm. The list is reproduced in table 1.

This list was developed locally for internal purposes and for the needs of the collection. It represents only a fraction of all initial articles listed in Annex E of the Règles de catalogage anglo-Américaines.¹⁴ The local list was used since research could only be performed on the articles already in the exclusion list. It should be noted that, due to system limitations, investigating the French article “l’” was not possible. This resulted in a total of 40 articles under investigation.

To answer the authors’ first research question, each article was searched individually in browse-title mode. The title index was then systematically and thoroughly scanned in order to find all the entries beginning with a non-article word homographic to an initial article. This was done by typing the article in the search box. It should be explained here that if only the article is included in the search string (and no other words), the system does not strip the “article” and positions the user at the beginning of the title index for that word. This is how it was possible to thoroughly scan the index for each article. For each entry thus identified, the corresponding MARC record was examined to find out which field contained the problematic word. Problematic entries were recorded in a spreadsheet; those entries that resulted from the inevitable miscoding of the non-filing indicators were not retained. While examining the MARC record, the record number in field 001 was also recorded to allow the total number of affected records to be determined. This number was less than the number of problematic entries found in the index, because any given record could contain more than one title and therefore generate two or more problematic entries in the index.

To provide an answer to the authors’ second research question, the total number of affected records (those that generate at least one problematic entry in the title index) was compiled and compared to the total number of monographic records contained in Atrium at the time of research (summer 2004). It was then possible to obtain this data from the Université de Montréal Library systems office.

For the second part of the study, a controlled experiment involving real users was prepared. Given the exploratory nature of this paper, and the limited means at the authors’ disposal, the decision was made to use Atrium, the University of Toronto catalog, as a case study. This catalog was chosen because the retrieval module integrates a detection algorithm designed to detect the presence of the three English articles: “a,” “an” and “the,” and offers a search interface on which it is possible, at the first level, to select a specific search mode (browse or keywords). Atrium automatically defaults to keyword searches and this is the reason why it could not be used for this part of the study. Some transaction logs of queries entered into Atrium nonetheless were used, along with the data collected from the University of Toronto search sessions, to provide additional data for question 3.

To meet the four objectives defined for this part of the study, the following four research questions were formulated:

• Question 3: Do users usually keep the initial articles in their queries when searching the title index in browse mode or do they leave them out?
• Question 4: When users search for known titles, which mode do they usually use: “browse” or “keywords”?
• Question 5: What is the proportion (number of prob-lematic records found divided by total number of problematic records searched) of monographic titles containing a word wrongly processed as an initial article by the detection algorithm that is actually retrieved by the end users, and is this proportion the same for titles not affected by this problem?
• Question 6: Are monographic titles containing a non-article word homograph to an initial article usually harder to retrieve than other titles, in terms of time and effort?

To answer the first of these four research questions, user behavior when searching the title index of a library catalog was analyzed. The transaction logs provided by the systems office of the Université de Montréal libraries were initially examined for the searches in browse mode in the title index of the Atrium catalog for the duration of one month (October 2005). With these data in hand, checking whether users usually keep the initial articles in their title queries, or whether they leave them out, was possible.

To answer the three remaining research questions, the authors first compiled all titles containing a word that might be erroneously considered as an initial article in the University of Toronto catalog. The exclusion list used at the University of Toronto catalog consists of only the three English articles. The authors built a file of all titles that might be difficult to retrieve—in other words, the documents whose title begins with the word “a,” “an,” or “the” when this word is not an article (for example: À bout portant; An der Wegscheide; Thé ou café, Monsieur le Ministre?)—and obtained 4,384 such document titles. In order to create the data sample, only those titles were kept that were in French or in English (i.e., 1,545 titles), because participants in the study were only fluent in these two languages.

From this set of problematic titles, 24 lists were prepared, with 30 titles in each, all titles being selected at random. In order not to influence the search behavior of the participants in the study, the authors mixed different types of titles in each list. Each list of 30 titles was made up of 3 groups of titles as follows:

• Group 1: Five titles beginning with an “ordinary” word, i.e., neither an article, nor homographic to an article (for example, Out after dark).
• Group 2: Ten titles beginning with a real article (for example, A very profitable war).
• Group 3: Fifteen “problematic” titles, i.e., beginning with a non-article word homograph to an initial article (for example, À la plage).

Throughout the rest of this paper, the first two groups of titles are usually referred to as the “non-problematic” titles and the third group of titles as the “problematic” titles.

All titles included in the lists were cataloged in the University of Toronto Library catalog, and were therefore, in principle, retrievable. The order of presentation of the titles in the lists to be searched was determined randomly, and it was modified for each list to minimize the learning factor. An example of such a list appears in the appendix.

Once these lists were prepared, 24 students at the pre-university level (first or second year of Cégep [Collèges d’enseignement général et professionnel]), enrolled in the pre-university profile (in Québec, Cégep is a required step between high school and university), were asked to try and locate the bibliographic records for the titles listed on one list. Each participant received a different list so that there would be no contamination effect. The main reason for selecting college students was to have a rather homogenous group from the point of view of exposure and experience with bibliographic searching in catalogs. Each student was requested to search all titles on his or her list using one or the other of the two search options containing or starting with as shown in figure 3.

At the start of each session, the two search options were alternated, selecting the containing option initially for one half of the participants, and the starting with option for the other half, to avoid a bias in favor of either of the two search modes, at least at the beginning of the search process. The participants were completely free to use either of the two modes at any time during the search session. The title index was preselected and the participants were not allowed to change it. Each of the search sessions was recorded using Camtasia, a software application designed to record all the operations performed on screen and to create a video that reproduces the search sessions faithfully.

Once they retrieved a record, the participants had to write down the call number on the form (see appendix), which made it possible to easily ascertain the success rate. Their answers were double-checked by replaying each video. The following information was also recorded for each title:

• Starting time: the moment when the user executes his query by clicking on the Search button
• End time: the moment when the user displays the right record (if found)
• Search mode for each query: containing (keyword mode) or starting with (browse mode)
• Number of results: in the case of keyword searches, the number of results retrieved
• Initial article inclusion or omission in string search, for titles beginning with an article

Which of the articles on Atrium’s exclusion list have the most entries beginning with that string of letters when not used as an article?

Table 2 shows the total number of affected entries for each surveyed article and their origin in the record. A rapid survey of the data in table 2 clearly shows that some of the articles potentially were much more problematic than others. Almost half of the articles in the exclusion list never generated any problematic entries. Conversely, the article “a” alone generated 4,230 problematic entries in the index (66 percent of the total). The main explanation is that the article “a” is very common in English (and in other languages as well) and it is also a very frequently used preposition in French. For example, 1,205 documents with a title beginning with “À propos de …” (“All about …”) were noted. All these entries were the cause of retrieval problems in browse mode searches. It was also noted that problems occurred when the title began with the initial of a first name beginning as “A” (A.B.C. contre Poirot), and also for many acronyms (A.A.C.R., A.B.B.), or the many works beginning with A.B.C. (A.B.C. de la lecture, for example). The high proportion of French language works in the Atrium catalog, as compared to catalogs in English-speaking institutions, magnified the problem in this case. For instance, the University of Toronto catalog (being much larger than Atrium) has a lesser proportion of problematic records, quite probably because the proportion of French language resources in the former is lower than in the latter.

What proportion (i.e., number of records with affected titles divided by total number of monographic records in the database) of records is affected by the deficient retrieval algorithm in Atrium?

The data presented in table 2 indicates that the total number of problematic entries in the title index was 6,360. While matching this data with the record numbers collected while doing data collection, these entries were found to be coming from 5,111 distinct bibliographic records in the catalog. Again, one should remember that any one record can contribute more than one entry in the title index. For instance, a record might have two problematic titles, one in field 245 and one in field 740.

The total number of monographic records in Atrium at the time of the data collection was estimated to be approximately 1,318,000. It may, therefore, be estimated that the proportion of monographic records affected by the initial article detection algorithm was slightly less than 0.4 percent (table 3). This proportion concerns only those titles found in six MARC fields, and this number would probably be higher if series titles had been considered.

For the second phase of the authors’ research, a log of Atrium browse-title queries was captured for the month of October 2005. For the part involving participants, data were collected at Collège de Maisonneuve (Montréal, Canada), between January 30 and February 6, 2006. Recruiting was done through posters explaining the tasks to be performed, the estimated time required (roughly 45 minutes), and the remuneration offered ($20).

Do users usually keep the initial articles in their queries when searching the title index in browse mode or do they leave them out?

Analysis of the queries collected in the transaction log of the Atrium catalog indicated that users seemed to retain the initial article in their queries in approximately two cases out of three (table 4). Out of the 12,216 queries recorded in the transaction log, 1,468 queries (approximately 12 percent) were queries made to search works whose titles began with an article. This was estimated to the best of the authors’ knowledge by examining each query on a case-by-case basis, but it was not always possible to be 100 percent certain whether the title of the resource sought by the end user actually began with an article. From these queries, it was observed that the initial article was omitted in only 36.8 percent of all cases, leading the authors to believe that end users usually would rather leave the initial articles in their queries. Comparing these data with other catalogs where this feature is not present would be interesting, because Atrium users may have learned over time, from using the catalog, that they do not need to pay attention to the initial article. Nonetheless, in their study of known-item queries in OPACs, Kan and Poo noted the same behavior observed in this study.¹⁴

Similar proportions were observed when the video-recorded search sessions in the University of Toronto catalog were analyzed (see table 4). Out of 54 queries made in browse mode to find titles with an initial article, 37 queries (68.5 percent) contained the initial article, while the user had not included the article in the other 17 queries (31.5 percent). The authors’ analysis revealed that the queries where the initial article was omitted were more successful. Of 37 queries in which the initial article was retained, only 18 (48.6 percent) successfully retrieved the desired record. This rises to 88.2 percent when the initial article was removed from the query.

When users search for known titles, which mode do they usually use: browse or keywords?

The compilation of the total number of queries made by the 24 participants to find their 30 titles indicates that more than three quarters of the queries were issued using the keywords mode (see table 5). This proportion rises to 80.2 percent if only the first query is counted for each title. Following these observations, one might assume that the users’ preferred mode is the keywords mode, but it must be remembered that the title samples submitted to the participants consisted of 50 percent problematic titles, which is not at all representative of the proportion of problematic titles in a catalog (less than 0.4 percent, according to the authors’ previous analysis). Because of the initial article automatic detection algorithm, retrieving these titles in browse mode is nearly impossible. The authors’ data reveal that none of the 360 problematic titles (15 titles on each of the 24 lists) could be retrieved using the browse mode. Analysis for all titles reveals that the last query—the query that successfully retrieved the record—was made in keywords mode in 9 times out of 10 (table 5). It is not surprising, therefore, that users ended up choosing this mode most of the time.

A chronological analysis of the queries indicates that at the beginning of the session, users were using the browse mode more often. Seventeen out of the 24 participants (71 percent) used this mode to issue their very first query, in spite of the authors taking care to preselect the keywords mode as the starting selection for half of them. In figure 4, that behavior can be seen at the beginning of the session. For the first 5 titles, both modes scored approximately the same—they were equally used. As the session continued, users progressively abandoned the browse mode for the keywords mode (only 2 percent of the queries in browse mode for the last 5 titles searched) in spite of the fact that the browse mode is known to be more efficient for locating a known document. Affirming that users prefer the keywords mode is difficult, because the overrepresentation of problematic titles in the sample gave the participants the misleading impression that the browse index mode was less efficient. A Web catalog analysis by Halcoussis and colleagues revealed that when asked to rate the level of satisfaction regarding organization of a Web catalog based on a variety of criteria, “browse-title” ranked as one of the highest search types, with a coefficient estimate of 0.919 (subject searches being set to zero as a control category), while “keywords-in-title” ranked as the lowest search type, with a coefficient estimate of -1.711.¹⁶

What is the proportion (number of problematic records found divided by total number of problematic records searched) of monographic titles containing a word wrongly processed as an initial article by the detection algorithm that is actually retrieved by the end users, and is this proportion the same for titles not affected by this problem?

The search for a known document (known-item search) for which the end user has the exact title is one of the easiest imaginable task in any catalog. The success rate should be near 100 percent. This is what was observed for all the titles in the samples that were not problematic (with articles and without articles combined). However, for the titles considered problematic because of the presence at the beginning of the field of a non-article homograph to an article, 2 titles out of 15 were not retrieved on average (see table 6). A t test comparison of the averages obtained reveals that the differences observed are significant (p < .0005). The authors have, therefore, concluded that titles that are considered problematic because of the presence of a word erroneously treated as an initial article by the detection algorithm are more difficult to retrieve.

Are monographic titles containing a non-article word homograph to an initial article usually harder to retrieve than other titles, in terms of time and effort?

The time measured was from the moment the user pressed a key to launch his query and the moment the record displayed on screen. The time for keying-in the query was not counted, since titles can vary in length. System response time was noted to be minimal at all times; the time measured here corresponded mainly to the time it took for the user to recognize the correct record and select it. Titles that were not found were excluded from the average.

Analysis of the time necessary to find the records reveals that problematic titles have taken much more time on average (see table 7). Finding the titles containing an initial article took more time, compared to those without such an article, but the statistical analysis reveals that this difference is not significant (p = .062). Statistical analysis of problematic titles compared with the other two title groups combined shows that the differences observed are, in this case, meaningful (p < .0005).

In this study, in addition to time, two measurements were used to represent the effort invested by the participants to locate a title: the mean number of queries used and the mean size of the retrieved sets (for the queries issued in keywords mode) were measured (see table 7). On average, more queries were necessary to find the titles containing an initial article than to find those that did not, but the statistical analysis shows that the difference is non-significant (p = .489). Conversely, the statistical analysis comparing problematic titles with titles of the two other groups combined shows that the differences are significant (p < .0005).

On average, the users retrieved slightly larger sets (offering less precision) to find titles containing an initial article than to find those without an article. The statistical analysis, however, reveals that the difference is non-significant (p = .763) (see table 7). Nonetheless, the statistical analysis of the problematic titles compared with each of the 2 other title groups (with and without article) shows that the differences are significant in this case (p < .005 and p < .011 respectively). These two measurements, therefore, indicate that (on average) more time and more effort (number of queries and size of sets to browse) were necessary to locate a problematic title.