Improving Domain-Specific Text Understanding with Large Language Models via Hybrid Fine-Tuning Strategies

Abstract

Large language models (LLMs) often show strong general capabilities but can underperform on domain-specific text understanding when terminology, style, and label definitions differ from general web text. Fine-tuning is a natural remedy, yet a single strategy rarely satisfies all practical constraints: full fine-tuning is expensive and brittle, lightweight parameter-efficient tuning may underfit, and retrieval-only methods depend heavily on index coverage.
This paper presents a hybrid fine-tuning framework for domain-specific text understanding that combines (i) continued pretraining on domain corpora, (ii) parameter-efficient instruction tuning, and (iii) task-specific calibration and evaluation. We describe a training recipe that is modular, reproducible, and designed for realistic constraints such as limited labeled data and strict compute budgets.
We provide ablations that isolate the contributions of each component and a set of anal- ysis tools for diagnosing failures related to terminology shift, long-context evidence, and label ambiguity.