Improving text classification: logistic regression makes small LLMs strong and explainable ‘tens-of-shot’ classifiers
Staff working papers set out research in progress by our staff, with the aim of encouraging comments and debate. Published on 23 May 2025 Staff Working Paper No. 1,127 By Marcus Buckmann and Ed Hill
Text classification tasks such as sentiment analysis are common in economics and finance. We demonstrate that smaller, local generative language models can be effectively used for these tasks. Compared to large commercial models, they offer key advantages in privacy, availability, cost, and explainability.
We use 17 sentence classification tasks (each with 2 to 4 classes) to show that penalised logistic regression on embeddings from a small language model often matches or exceeds the performance of a large model, even when trained on just dozens of labelled examples per class – the same amount typically needed to validate a large model’s performance. Moreover, this embedding-based approach yields stable and interpretable explanations for classification decisions.
Information from this article can be found here.