Generative artificial intelligence has become a crucial topic in the library and information science field over the past year. The use of tools that leverage Large Language Models (LLMs) is rapidly disrupting practices in content creation and information retrieval, and it is urgent to test this technology’s potential impacts in the context of libraries’ user-centred services like reference, instruction, and research support.

This session will present an overview of the work done to date on a project that investigates the issues and challenges with employing generative AI technology in academic library information services in effective and ethical ways. The presentation will include 1) a description of the development of a chatbot that we’ve named Gaby (tm), configured for delivering academic library information services leveraging AI technology, and 2) the contributions of this research in developing a protocol for framing and evaluating the implementation of services that incorporate generative AI tools.

In terms of the chatbot development, the first step has been to build a knowledgebase from the Concordia Library’s website that will serve as Gaby’s “brain”. On top of this knowledgebase, we plan to connect different language models, including proprietary LLMs and open-source Small Language Models in different ways and determine the performance of each using a series of questions that a useful chatbot should be able to answer.

In addition to Gaby, a key output of this project will be the testing protocol and evaluation framework. Given that reference questions often require nuanced responses, it’s challenging to evaluate them as simply accurate or inaccurate. This study builds on the work of Lai (2023) to develop a testing protocol that incorporates multiple dimensions of user interactions. In addition, we are operationalizing aspects of the LC Labs AI Planning Framework (Library of Congress, 2023) to define the use cases possible for generative AI in information services as well as criteria for ethical considerations.

In keeping with the spirit of the Library Research Forum, this session will present work that is still in progress and will feature the team’s Research Assistants as presenters in order to showcase their contributions.

References

Lai, K. (2023). How well does ChatGPT handle reference inquiries? An analysis based on question types and question complexities. College & Research Libraries, 84(6), 974. doi:10.5860/crl.84.6.974

Library of Congress. (2023). LC Labs Artificial Intelligence Planning Framework. https://github.com/LibraryOfCongress/labs-ai-framework