Data Science · Chapter 36 of 43
Intro to NLP
NATURAL LANGUAGE PROCESSING teaches machines to work with text. Basic pipeline: tokenise → normalise → vectorise → model.
Common tasks: classification, sentiment, translation, chatbots.
Example 1 (python)
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
X = vec.fit_transform(['I love it', 'I hate it'])
print(X.shape)Output
(2, 4)Turn text into numbers via TF-IDF.
Example 2 (python)
# Tokens: words or sub-words like 'un', 'happy'Modern NLP uses sub-word tokens (BPE).
Key points
- Text must be converted to numbers.
- TF-IDF and embeddings are common.
- Transformer models (BERT, GPT) dominate today.
- Tokenisation is the first step.
💡 Note: Modern NLP is dominated by transformer models — but classical TF-IDF + logistic regression is still a very strong baseline.