Deep dive: Small vs large language models for token classification

Author

Ander Steele, PhD
June 20, 2025

Check out this talk from Ander Steele, Head of AI at Tonic.ai, which was first presented at ODSC East in Boston, and explores the evolving landscape of named entity recognition (NER); specifically comparing the performance of small language models versus large language models for token classification tasks. With real-world examples, practical evaluation criteria, and insights drawn from experiments using both public web data and proprietary datasets, this presentation offers valuable guidance for practitioners working with unstructured data in domains like healthcare, customer support, and compliance. Whether you're fine-tuning models or evaluating the trade-offs between cost, speed, and accuracy, this session delivers a nuanced perspective on how to get the most out of modern NLP tools.

Full Transcript:

Let's get into talk which is about small language models versus Large Language Models (LLMs) for token classification. Do smaller models still outperform LLMs? And so the token classification task here that I want to talk about is named entity recognition, meaning let's extract named entities from unstructured texts.

A named entity is something like a name or organization or locations or specific entities in the text that need to be detected for a number of reasons. It could also be things like nationality or religious preferences; modifiers or adjectives about specific entities in the text.

Examples of Named Entities

  • Names of individuals
  • Organizations
  • Locations
  • Diseases
  • Medications

The classification task here is to start with some annotation guidelines. In other words, descriptions of what the entities are. Then take text, which we’re going to break up into tokens or – words. And from each of those tokens, we're going to predict a label.

The label can be a person, entity, or some other category of word for redaction. We ask, which entity, if any in our guidelines does that token correspond to? And from those, you can roll those up into individual entities.

Evaluation Criteria

The way we're gonna frame our evaluation here is motivated by the CoNLL conference natural language learning benchmark, which was a named entity, recognition benchmark, which posited the following evaluation criteria:

  • Evaluate predictions based on spans; meaning, an entity's start and stop location in the text.
  • Start and stop indices in the text as well as their labels.

For instance, my name is ‘G. Ander Steele’. My given name, which goes from characters 11 to 19, is ‘G. Ander’. My family name ‘Steele’ is from character 20 to 26. And in order to be scored correctly with respect to this evaluation criteria, I would have to produce two predictions with the same start and stop indices and the same labels.

Performance Metrics

This benchmark has saturated around a .94 F1 score — but there's a couple of interesting points here that are not reflected:

  1. Release of BERT in 2019, which could be used for fine-tuning.
  2. March 14th, 2023, when GPT-4 was released; we don't see a point here indicating that this benchmark has been solved by large language models.

Small Language Models

In this talk, we mean fine-tuned versions of BERT or RoBERTa. These models generate vector embeddings which are powerful because classifiers can be built on top of them.

To fine-tune a model, you need a large number of annotated samples, which often need to be internally consistent and correct.

Comparison of Data Sets

We will compare:

  • Public web data: Presumably in the pre-training corpus of these large language models.
  • ASR transcripts: Data that has not been included in the pre-training corpus of any LLM.

Challenges with LLMs

One major challenge is localizing entities in the text, where a string can have multiple meanings. For instance, the context changes whether "Washington" refers to the monument or the person.

Conclusion

Large language models have been fine-tuned for NER, but performance hasn't greatly exceeded that of fine-tuned small language models. In practice, LLMs can be used as data annotators, particularly when humans generate test sets against which these models can be reviewed.

For further inquiries, contact Tonic.

Author Bio

Ander Steele, PhD, Head of AI at Tonic.ai, specializes in building privacy-preserving data and machine learning pipelines.