Image by Author
If you have studied NLP, you might have heard about the term "tokenization." It is an important step in text preprocessing, where we transform our textual data into something that machines can understand. It does so by breaking down the sentence into smaller chunks, known as tokens. These tokens can be words, subwords, or even characters, depending on the tokenization algorithm being used. In this article, we will see how to use the Hugging Face Tokenizers Library to preprocess our textual data.
Setting Up Hugging Face Tokenizers Library
To start using the Hugging Face Tokenizers library, you'll need to install it first. You can do this using pip:
pip install tokenizersThe Hugging Face library supports various tokenization algorithms, but the three main types are:
- Byte-Pair Encoding (BPE): Merges the most frequent pairs of characters or subwords iteratively, creating a compact vocabulary. It is used by models like GPT-2.
- WordPiece: Similar to BPE but focuses on probabilistic merges (doesn't choose the pair that is the most frequent but the one that will maximize the likelihood of the corpus once merged), commonly used by models like BERT.
- SentencePiece: A more flexible tokenizer that can handle different languages and scripts, often used with models like ALBERT, XLNet, or the Marian framework. It treats spaces as characters rather than word separators.
The Hugging Face Transformers library provides an AutoTokenizer class that can automatically select the best tokenizer for a given pre-trained model. This is a convenient way to use the correct tokenizer for a specific model andcan be imported from thetransformerslibrary. However, for the sake of our discussion regarding the Tokenizers library, we will not follow this approach.
We will use the pre-trained BERT-base-uncased tokenizer. This tokenizer was trained on the same data and using the same techniques as the BERT-base-uncased model, which means it can be used to preprocess text data compatible with BERT models:
# Import the necessary componentsfrom tokenizers import Tokenizerfrom transformers import BertTokenizer# Load the pre-trained BERT-base-uncased tokenizertokenizer = BertTokenizer.from_pretrained('bert-base-uncased')Single Sentence Tokenization
Now, let's encode a simple sentence using this tokenizer:
# Tokenize a single sentenceencoded_input = tokenizer.encode_plus("This is sample text to test tokenization.")print(encoded_input)Output:
{'input_ids': [101, 2023, 2003, 7099, 3793, 2000, 3231, 19204, 3989, 1012, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}To ensure correctness, let's decode the tokenized input:
tokenizer.decode(encoded_input["input_ids"])Output:
[CLS] this is sample text to test tokenization. [SEP]In this output, you can see two special tokens. [CLS] marks the start of the input sequence, and [SEP] marks the end, indicating a single sequence of text.
Batch Tokenization
Now, let's tokenize a corpus of text instead of a single sentence using batch_encode_plus:
corpus = [ "Hello, how are you?", "I am learning how to use the Hugging Face Tokenizers library.", "Tokenization is a crucial step in NLP."]encoded_corpus = tokenizer.batch_encode_plus(corpus)print(encoded_corpus)Output:
{'input_ids': [[101, 7592, 1010, 2129, 2024, 2017, 1029, 102], [101, 1045, 2572, 4083, 2129, 2000, 2224, 1996, 17662, 2227, 19204, 17629, 2015, 3075, 1012, 102], [101, 19204, 3989, 2003, 1037, 10232, 3357, 1999, 17953, 2361, 1012, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}For better understanding, let's decode the batch-encoded corpus as we did incase of single sentence. This will provide the original sentences, tokenized appropriately.
tokenizer.batch_decode(encoded_corpus["input_ids"])Output:
['[CLS] hello, how are you? [SEP]', '[CLS] i am learning how to use the hugging face tokenizers library. [SEP]', '[CLS] tokenization is a crucial step in nlp. [SEP]']Padding and Truncation
When preparing data for machine learning models, ensuring all input sequences have the same length is often necessary. Two methods to accomplish this are:
1. Padding
Padding works by adding the special token[PAD]at the end of the shorter sequences to match the length of the longest sequence in the batch or max length supported by the model ifmax_lengthis defined. You can do this by:
encoded_corpus_padded = tokenizer.batch_encode_plus(corpus, padding=True)print(encoded_corpus_padded)Output:
{'input_ids': [[101, 7592, 1010, 2129, 2024, 2017, 1029, 102, 0, 0, 0, 0, 0, 0, 0, 0], [101, 1045, 2572, 4083, 2129, 2000, 2224, 1996, 17662, 2227, 19204, 17629, 2015, 3075, 1012, 102], [101, 19204, 3989, 2003, 1037, 10232, 3357, 1999, 17953, 2361, 1012, 102, 0, 0, 0, 0]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0]]}Now, you can see that extra 0s are placed, but for better understanding, let's decode to see where the tokenizer has placed the [PAD] tokens:
tokenizer.batch_decode(encoded_corpus_padded["input_ids"], skip_special_tokens=False)Output:
['[CLS] hello, how are you? [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]', '[CLS] i am learning how to use the hugging face tokenizers library. [SEP]', '[CLS] tokenization is a crucial step in nlp. [SEP] [PAD] [PAD] [PAD] [PAD]']2. Truncation
Many NLP models have a maximum input length sequence, and truncation works by chopping off the end of the longer sequence to meet this maximum length. It reduces memory usage and prevents the model from being overwhelmed by very large input sequences.
encoded_corpus_truncated = tokenizer.batch_encode_plus(corpus, truncation=True, max_length=5)print(encoded_corpus_truncated)Output:
{'input_ids': [[101, 7592, 1010, 2129, 102], [101, 1045, 2572, 4083, 102], [101, 19204, 3989, 2003, 102]], 'token_type_ids': [[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1], [1, 1, 1, 1, 1], [1, 1, 1, 1, 1]]}Now, you can also use thebatch_decodemethod, but for better understanding, let's print this information in a different way:
for i, sentence in enumerate(corpus): print(f"Original sentence: {sentence}") print(f"Token IDs: {encoded_corpus_truncated['input_ids'][i]}") print(f"Tokens: {tokenizer.convert_ids_to_tokens(encoded_corpus_truncated['input_ids'][i])}") print()Output:
Original sentence: Hello, how are you?Token IDs: [101, 7592, 1010, 2129, 102]Tokens: ['[CLS]', 'hello', ',', 'how', '[SEP]']Original sentence: I am learning how to use the Hugging Face Tokenizers library.Token IDs: [101, 1045, 2572, 4083, 102]Tokens: ['[CLS]', 'i', 'am', 'learning', '[SEP]']Original sentence: Tokenization is a crucial step in NLP.Token IDs: [101, 19204, 3989, 2003, 102]Tokens: ['[CLS]', 'token', '##ization', 'is', '[SEP]']This article is part of our amazing series on Hugging Face. If you want to explore more about this topic, here are some references to help you out:
- Hugging Face Tokenizer Documentation
- Hugging Face Transformers Course: Preprocessing Data
- HuggingFace: Summary of Different Tokenizers
Kanwal Mehreen Kanwal is a machine learning engineer and a technical writer with a profound passion for data science and the intersection of AI with medicine. She co-authored the ebook "Maximizing Productivity with ChatGPT". As a Google Generation Scholar 2022 for APAC, she champions diversity and academic excellence. She's also recognized as a Teradata Diversity in Tech Scholar, Mitacs Globalink Research Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having founded FEMCodes to empower women in STEM fields.
- How to Use Hugging Face’s Datasets Library for Efficient Data Loading
- How to Use Hugging Face AutoTrain to Fine-tune LLMs
- How to Use GPT for Generating Creative Content with Hugging Face…
- A community developing a Hugging Face for customer data modeling
- Top 10 Machine Learning Demos: Hugging Face Spaces Edition
- Build AI Chatbot in 5 Minutes with Hugging Face and Gradio
