Text Mining, TF-IDF Vectorization & Word Embeddings

Explanation

Unstructured natural language text must be converted into numerical matrices before machine learning algorithms can process it. Simple word count approaches are skewed by ubiquitous stop words ('the', 'is', 'at'). TF-IDF resolves this by multiplying term frequency by inverse document frequency, yielding higher weights for rare, informative keywords. Modern text mining employs Word2Vec (Continuous Bag of Words and Skip-Gram models) to learn dense continuous vectors using shallow neural networks, capturing complex semantic relationships such as vector('King') - vector('Man') + vector('Woman') approx vector('Queen').

Key points

• Bag of Words (BoW): Represents text as sparse count vectors of vocabulary words, ignoring word order and grammar.
• TF-IDF (Term Frequency - Inverse Document Frequency): Penalizes common corpus words and amplifies terms distinctive to specific documents.
• Formula: TF-IDF(t, d, D) = TF(t, d) * log(N / DF(t)).
• Word Embeddings (Word2Vec, GloVe): Dense distributed vector representations where semantically related words are positioned close together in vector space.

Example

Code examples

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "Machine learning algorithms build mathematical models",
    "Deep learning is a subset of machine learning",
    "Natural language processing uses text mining techniques"
]

vectorizer = TfidfVectorizer(stop_words='english')
tfidf_matrix = vectorizer.fit_transform(corpus)
print("Vocabulary:", vectorizer.get_feature_names_out())
print("TF-IDF shape:", tfidf_matrix.shape)

Exam questions

• [8 Marks] Explain the complete NLP text preprocessing pipeline. Detail the mathematical formula and purpose of TF-IDF vectorization.
• [6 Marks] Compare Bag of Words (BoW) representations with distributed Word Embeddings (Word2Vec).