Machine Learning · Chapter 37 of 40
NLP Basics
NATURAL LANGUAGE PROCESSING converts text into features a model can use.
Classic pipeline: TOKENIZE → NORMALIZE (lowercase, stem/lemmatize) → VECTORIZE (bag-of-words, TF-IDF, embeddings).
Example 1 (python)
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
X = vec.fit_transform(['ml is fun', 'ml is hard'])
print(X.shape)Output
(2, 4)TF-IDF turns text into a numeric matrix.
Example 2 (python)
# Modern NLP: use pre-trained transformers (BERT, GPT) via HuggingFaceThe state of the art in 2020s.
Key points
- Tokenize → normalize → vectorize.
- Bag-of-words and TF-IDF are classic baselines.
- Transformers dominate modern NLP.
- Text data is huge — sparse representations help.
💡 Note: For most modern tasks, a pre-trained transformer (via HuggingFace) beats hand-crafted features easily.