<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="static/style.xsl"?><OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd"><responseDate>2026-11-22T16:39:21Z</responseDate><request verb="GetRecord" identifier="oai:uwspace.uwaterloo.ca:10012/21889" metadataPrefix="dim">https://dspacemainprd01.lib.uwaterloo.ca/server/oai/request</request><GetRecord><record><header><identifier>oai:uwspace.uwaterloo.ca:10012/21889</identifier><datestamp>2025-11-06T08:12:10Z</datestamp><setSpec>com_10012_9924</setSpec><setSpec>com_10012_1</setSpec><setSpec>col_10012_9930</setSpec><setSpec>col_10012_6</setSpec></header><metadata><dim:dim xmlns:dim="http://www.dspace.org/xmlns/dspace/dim" xmlns:doc="http://www.lyncode.com/xoai" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.dspace.org/xmlns/dspace/dim http://www.dspace.org/schema/dim.xsd">
   <dim:field mdschema="dc" element="contributor" qualifier="author">arabzadehghahyazi, negar</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="accessioned">2025-06-20T16:12:11Z</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="available">2025-06-20T16:12:11Z</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="issued">2025-06-20</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="submitted">2025-05-28</dim:field>
   <dim:field mdschema="dc" element="identifier" qualifier="uri">https://hdl.handle.net/10012/21889</dim:field>
   <dim:field mdschema="dc" element="description" qualifier="abstract">The rapid advancement of information access technologies, including neural retrieval models and generative information-seeking systems, has outpaced traditional evaluation methodologies, exposing fundamental gaps in assessing their effectiveness. Existing evaluation frameworks struggle to adapt, particularly in the presence of sparse relevance labels, limiting their ability to fairly and comprehensively compare retrieval and generation-based systems. The emergence of large language models (LLMs) further complicates evaluation, as they challenge conventional assessment paradigms while offering new opportunities for automated evaluation. To address these issues, it is crucial to first identify flaws in current evaluation methodologies and then develop more robust, efficient, and adaptable assessment strategies.

This thesis begins by demonstrating that evaluation based on sparse labeling introduces substantial biases and inconsistencies in system rankings, often failing to recognize genuine improvements in retrieval effectiveness. We show that in traditional IR benchmarks,stronger models may retrieve highly relevant but unjudged documents, leading to underestimation of their performance. To mitigate this, we propose an alternative evaluation approach based on distribution of retrieved results and labeled data using Fréchet Distance. This method not only improves robustness in the presence of sparse labels but also facilitates direct comparison between retrieval-based and generative models on a common evaluation scale.

We then investigate how LLMs can be leveraged to evaluate IR systems, distinguishing between their use for evaluating retrieval-based methods and generative IR systems. A key focus of this work is the role of LLMs in automated relevance judgments. We systematically compare different LLM-based relevance assessment methodologies, highlighting the lack of standardization in evaluating these approaches. To address this gap, we propose a structured framework that evaluates relevance judgment methods based on their alignment with human labels and their impact on system rankings. Furthermore, we examine the effect of prompt formulation on LLM-based evaluation, demonstrating how prompt variations can significantly influence the consistency and reliability of assessment outcomes.

Finally, we extend our study beyond retrieval-based evaluation to assessing generated content across multiple applications. We explore retrieval-assisted methods for evaluating generative textual content, IR-inspired approaches for assessing text-to-image generation models, and a broader framework for evaluating LLM-powered applications. These contributions lay the foundation for a new generation of evaluation methodologies that keep pace with evolving information access technologies, ensuring that improvements in retrieval and generative AI systems can be accurately and meaningfully assessed.</dim:field>
   <dim:field mdschema="dc" element="language" qualifier="iso">en</dim:field>
   <dim:field mdschema="dc" element="publisher" lang="en">University of Waterloo</dim:field>
   <dim:field mdschema="dc" element="relation" qualifier="uri">https://microsoft.github.io/msmarco/</dim:field>
   <dim:field mdschema="dc" element="relation" qualifier="uri">https://ir-datasets.com/antique.html</dim:field>
   <dim:field mdschema="dc" element="relation" qualifier="uri">https://github.com/Narabzad/llm-relevance-judgement-comparison</dim:field>
   <dim:field mdschema="dc" element="relation" qualifier="uri">https://github.com/Narabzad/Language-model-selection-strategy</dim:field>
   <dim:field mdschema="dc" element="subject">Information Retrieval</dim:field>
   <dim:field mdschema="dc" element="subject">Large Language Models</dim:field>
   <dim:field mdschema="dc" element="subject">Evaluation</dim:field>
   <dim:field mdschema="dc" element="subject">Retrieval Augmented Generation</dim:field>
   <dim:field mdschema="dc" element="subject">Generative Information Retrieval</dim:field>
   <dim:field mdschema="dc" element="title">Evaluation of Information Access Systems in the Generative Era</dim:field>
   <dim:field mdschema="dc" element="type">Doctoral Thesis</dim:field>
   <dim:field mdschema="dc" element="pending">false</dim:field>
   <dim:field mdschema="uws-etd" element="degree" qualifier="department">David R. Cheriton School of Computer Science</dim:field>
   <dim:field mdschema="uws-etd" element="degree" qualifier="discipline">Computer Science</dim:field>
   <dim:field mdschema="uws-etd" element="degree" qualifier="grantor" lang="en">University of Waterloo</dim:field>
   <dim:field mdschema="uws-etd" element="degree">Doctor of Philosophy</dim:field>
   <dim:field mdschema="uws-etd" element="embargo" qualifier="terms">0</dim:field>
   <dim:field mdschema="uws" element="contributor" qualifier="advisor">Clarke, Charles</dim:field>
   <dim:field mdschema="uws" element="contributor" qualifier="affiliation1">Faculty of Mathematics</dim:field>
   <dim:field mdschema="uws" element="published" qualifier="city" lang="en">Waterloo</dim:field>
   <dim:field mdschema="uws" element="published" qualifier="country" lang="en">Canada</dim:field>
   <dim:field mdschema="uws" element="published" qualifier="province" lang="en">Ontario</dim:field>
   <dim:field mdschema="uws" element="typeOfResource" lang="en">Text</dim:field>
   <dim:field mdschema="uws" element="peerReviewStatus" lang="en">Unreviewed</dim:field>
   <dim:field mdschema="uws" element="scholarLevel" lang="en">Graduate</dim:field>
   <dim:field mdschema="others" element="access-status">open.access</dim:field>
</dim:dim>
</metadata></record></GetRecord></OAI-PMH>