Research on Efficient Fine-tuning of Large Language Model Parameters for Text Classification: A Review from Gradient Descent Basics to LoRA/QLoRA Methods

Tianyi Qu

doi:10.54254/2755-2721/2026.GU34136

Abstract

Large Language Models (LLMs) perform well in many text classification tasks, such as sentiment analysis, spam detection, and topic classification. However, adapting large language models to downstream tasks via full-parameter fine-tuning often requires significant computing power and GPU memory, resulting in high training costs, which is impractical for course assignments or resource-constrained scenarios. To address this issue, Parameter Efficient Fine-Tuning (PEFT) has gradually become a popular approach. Its idea is to freeze most model parameters and train only a small number of additional parameters, thereby reducing training costs. This paper takes text classification as an example and begins with an introduction to fundamental concepts such as stochastic gradient descent, cross-entropy loss, and confusion matrices. It first reviews the technological evolution from traditional NLP models (such as bag-of-words, Naive Bayes, and N-grams) to word embeddings, Transformers, and ultimately large language models. Then, it focuses on the low-rank adaptation concept of LoRA and the 4-bit quantization training mechanism of QLoRA, analyzing their characteristics from the perspectives of parameter count, GPU memory consumption, and training stability. This paper also discusses key representation-level issues such as word segmentation methods (BPE, WordPiece, SentencePiece), Type/Token differentiation, vocabulary size, and sequence length, explaining how these factors affect the model's generalization ability. Overall, this paper aims to answer the question of "how to adapt large models to text classification tasks at a lower cost under limited resources," and provides a clearer theoretical framework and methodological approach for course learning and introductory research.

Keywords: large language model , efficient parameter fine-tuning , LoRA , QLoRA

Article Information

Received: 24 April 2026

Published: 8 June 2026

DOI: https://doi.org/10.54254/2755-2721/2026.GU34136

Article Type: Research Article

Cite this article

Qu,T. (2026). Research on Efficient Fine-tuning of Large Language Model Parameters for Text Classification: A Review from Gradient Descent Basics to LoRA/QLoRA Methods. Applied and Computational Engineering,242,128-133.

Data availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

Disclaimer/Publisher's Note

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

About this Volume

Volume Title: ACE Vol.242

Part of Series: Applied and Computational Engineering

ISSN: 2755-2721 (Print) / 2755-273X (Online)

Copyright & License

[1]Marina Sokolova, Guy Lapalme. A systematic analysis of performance measures for classification tasks . Information Processing & Management, Vol.45(4). Publisher: Elsevier (Pergamon). 200907. 427–437. 427–437

[2]Stanley F. Chen, Joshua Goodman. An Empirical Study of Smoothing Techniques for Language Modeling. Computer Speech & Language, Vol.13(4). Publisher: Academic Press (Elsevier). 19991001. 359–394. 359–394

[3]Rico Sennrich, Barry Haddow, Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units . In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016). Publisher: Association for Computational Linguistics. 201608. 1715–1725. 1715–1725

[4]Mike Schuster, Kaisuke Nakajima. Japanese and Korean voice search . In: 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2012). Publisher: IEEE. 201203. 5149–5152.

[5]Taku Kudo, John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing . In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP 2018 Demo). Publisher: Association for Computational Linguistics. 201811. 66–71.

[6]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan AllenZhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. LoRA: LowRank Adaptation of Large Language Models . arXiv 20211016. PDF 1–26.

[7]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv. 20230523. PDF 1-26.

[8]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly. ParameterEfficient Transfer Learning for NLP . In: Proceedings of the 36th International Conference on Machine Learning (ICML 2019). PMLR, Vol.97. 201906. 2790–2799（PDF 1–10）.

[9]Xiang Lisa Li, Percy Liang. PrefixTuning: Optimizing Continuous Prompts for Generation . In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACLIJCNLP 2021). Publisher: Association for Computational Linguistics. 202108. 4582–4597.

[10]Brian Lester, Rami AlRfou, Noah Constant. The Power of Scale for ParameterEfficient Prompt Tuning . In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021). Publisher: Association for Computational Linguistics. 202111. 3045–3059.