Banca de DEFESA: JEAN PAES LANDIM DE LUCENA

Uma banca de DEFESA de MESTRADO foi cadastrada pelo programa.
STUDENT : JEAN PAES LANDIM DE LUCENA
DATE: 31/08/2026
TIME: 10:00
LOCAL: https://meet.google.com/oif-vmxf-gap
TITLE:

Decomposed Evaluation of a Multimodular RAG Architecture for Clinical Decision Support in Infectious Diseases and Combating Antimicrobial Resistance

KEY WORDS:

Antimicrobial resistance; Large language models; Retrieval-Augmented Generation; RAG metrics; Clinical decision support systems; Infectious diseases.

PAGES: 127
BIG AREA: Ciências Biológicas
AREA: Biologia Geral
SUMMARY:

Antimicrobial resistance ranks among the greatest threats to contemporary public health with an estimated 10 million annual deaths projected by 2050 and inappropriate antibiotic prescribing, particularly in primary care, is one of its preventable causes. This inappropriateness stems largely from overburdened health services and the pace at which therapeutic protocols are updated; consequently, large language models are being proposed to support clinical decision-making. However, there is a lack of evidence distinguishing the extent to which these tools rely on consulting official documents versus information memorized during pre-training especially regarding the Portuguese language and the protocols of Brazil's Unified Health System (SUS). This study proposes and evaluates a multi-modular Retrieval-Augmented Generation (RAG) architecture designed to support clinical decision-making in infectious diseases. The aim is to decompose the contribution of the retrieval process, assess the reliability of the model's expressed confidence, and pinpoint the origin of errors within the system's modules. Twenty official Ministry of Health documents were indexed in a ChromaDB vector repository, segmented into 1,918 chunks (up to 1,500 characters each), and retrieved using both semantic and lexical methods (top-K = 5). The process included re-ranking via a multilingual cross-encoder and a clinical guardrails layer comprising auditable rules for scope, urgency, and response grounding. The evaluation conducted on a question-by-question basis covers 135 multiple-choice questions from the REVALIDA exam (2020–2025). These were submitted to five language models using deterministic decoding across successive context conditions: no retrieval; hybrid retrieval with and without re-ranking; irrelevant context; and for a subsample of 38 questions with annotated gold-standard answers "oracle" context. The system achieved an average accuracy of 70.60% surpassing the 59% threshold required for physicians to pass the exam and hybrid retrieval raised the passage-level Recall@5 from 52.6% to 71.1% (p = 0.016). The 1.64% gain over the no-context condition was statistically indistinguishable from chance (95% CI: −2.24% to +5.52%), whereas irrelevant context degraded performance by 3.13%, demonstrating the instrument's sensitivity to context effects. When restricted to questions demonstrably covered by the knowledge base, retrieval yielded 7.37% against a 10.00% ceiling under "oracle" context, capturing 73.7% of the available gain. Standard document-level metrics overestimated retrieval quality by up to 26.3% and even reversed the ranking of strategies; self-reported confidence was consistently overconfident; and an analysis of attributable errors revealed that reasoning failures (23 of 56) predominated over retrieval failures (18). This work presents a reproducible evaluation framework conducted in Portuguese using official protocols and offers findings with practical implications: the performance ceiling is determined by corpus coverage and answer reasoning rather than the search mechanism; consequently, database curation and verifiable grounding (rather than aggregate accuracy, which is saturated by the models' parametric memory) are the true differentiators for improvements and should guide decisions on whether to answer or abstain in a clinical system.

COMMITTEE MEMBERS:
Presidente - 3884005 - PATRICK CESAR ALVES TERREMATTE
Interno - 3063244 - TETSU SAKAMOTO
Externo ao Programa - 3492344 - SILVAN FERREIRA DA SILVA JUNIOR - UFRNExterno à Instituição - Icaro Boszczowski - USP