Date of Award

5-2024

Document Type

Project

Degree Name

Master of Science in Information Systems and Technology

Department

Information and Decision Sciences

First Reader/Committee Chair

Shayo, Conrad

Abstract

The rise of conversational user interfaces (CUIs) powered by large language models (LLMs) is transforming human-computer interaction. This study evaluates the efficacy of LLM-powered chatbots, trained on website data, compared to browsing websites for finding information about organizations across diverse sectors. A within-subjects experiment with 165 participants was conducted, involving similar information retrieval (IR) tasks using both websites (GUIs) and chatbots (CUIs). The research questions are: (Q1) Which interface helps users find information faster: LLM chatbots or websites? (Q2) Which interface helps users find more accurate information: LLM chatbots or websites?. The findings are: (Q1) Participants found information significantly faster using LLM-chatbots, Q2. Participants found more accurate information using LLM-chatbots. The conclusions are: (Q1) LLM-chatbots are highly efficient, and (Q2). LLM-chatbots are highly reliable for information lookup tasks. These findings highlight the potential of LLM-powered CUIs to revolutionize user experience and advocate for integrating advanced AI capabilities in future interface design. Future research should investigate: 1. LLM-chatbot interaction speed over time to measure efficiency especially with more complex questions, 2. The precision of these models over larger knowledge bases and complex questions, 3. Improvements in chatbot’s usability and its impact on user experience and human computer interaction (HCI), and 4. Gauge user preference over prolonged interactions over more complex questions.

Download

  Contact Author

Share

COinS