Feng Li
School of Statistics and Mathematics
Central University of Finance and Economics
In information retrieval, tf–idf or TFIDF, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.
It is often used as a weighting factor in searches of information retrieval, text mining, and user modeling. The tf-idf value increases proportionally to the number of times a word appears in the document and is offset by the frequency of the word in the corpus, which helps to adjust for the fact that some words appear more frequently in general.
Tf-idf is one of the most popular term-weighting schemes today; 83% of text-based recommender systems in digital libraries use tf-idf.
Suppose we have a set of text documents and wish to rank which document is most relevant to the query, "the brown cow".
A simple way to start out is by eliminating documents that do not contain all three words "the", "brown", and "cow", but this still leaves many documents.
To further distinguish them, we might count the number of times each term occurs in each document; the number of times a term occurs in a document is called its term frequency. However, in the case where the length of documents varies greatly, adjustments are often made (see definition below). The first form of term weighting is due to Hans Peter Luhn (1957) which may be summarized as:
The weight of a term that occurs in a document is simply proportional to the term frequency.
In the case of the term frequency $tf(t,d)$, the simplest choice is to use the raw count of a term in a document, i.e. the number of times that term t occurs in document d. If we denote the raw count by $f_{t,d}$, then the simplest tf scheme is $tf(t,d) = f_{t,d}$. Other possibilities include
Boolean "frequencies": $tf(t,d) = 1$ if t occurs in d and 0 otherwise;
term frequency adjusted for document length : $f_{t,d} ÷$ (number of words in d)
logarithmically scaled frequency: $tf(t,d) = log (1 + f_{t,d})$
augmented frequency, to prevent a bias towards longer documents, e.g. raw frequency divided by the raw frequency of the most occurring term in the document:
Because the term "the" is so common, term frequency will tend to incorrectly emphasize documents which happen to use the word "the" more frequently, without giving enough weight to the more meaningful terms "brown" and "cow". The term "the" is not a good keyword to distinguish relevant and non-relevant documents and terms, unlike the less-common words "brown" and "cow". Hence an inverse document frequency factor is incorporated which diminishes the weight of terms that occur very frequently in the document set and increases the weight of terms that occur rarely.
Karen Spärck Jones (1972) conceived a statistical interpretation of term specificity called Inverse Document Frequency (IDF), which became a cornerstone of term weighting:
The specificity of a term can be quantified as an inverse function of the number of documents in which it occurs.
The inverse document frequency is a measure of how much information the word provides, that is, whether the term is common or rare across all documents. It is the logarithmically scaled inverse fraction of the documents that contain the word, obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient. $$ {\displaystyle \mathrm {idf} (t,D)=\log {\frac {N}{|\{d\in D:t\in d\}|}}}$$ with
${\displaystyle N}$: total number of documents in the corpus ${\displaystyle N={|D|}}$
${\displaystyle |\{d\in D:t\in d\}|}$ : number of documents where the term ${\displaystyle t}$ appears (i.e., ${\displaystyle \mathrm {tf} (t,d)\neq 0}$). If the term is not in the corpus, this will lead to a division-by-zero. It is therefore common to adjust the denominator to ${\displaystyle 1+|\{d\in D:t\in d\}|}$.
Then tf–idf is calculated as
$${\displaystyle \mathrm {tfidf} (t,d,D)=\mathrm {tf} (t,d)\cdot \mathrm {idf} (t,D)}$$A high weight in tf–idf is reached by a high term frequency (in the given document) and a low document frequency of the term in the whole collection of documents;
The weights hence tend to filter out common terms.
Since the ratio inside the idf's log function is always greater than or equal to 1, the value of idf (and tf-idf) is greater than or equal to 0.
As a term appears in more documents, the ratio inside the logarithm approaches 1, bringing the idf and tf-idf closer to 0.
Let find interesting insights into How I met your mother from its transcripts and one technique that kept coming up is TF/IDF.
Let's first build a corpus.
from collections import defaultdict
import csv
episodes = defaultdict(list)
with open("data/sentences.csv", "r") as sentences_file:
reader = csv.reader(sentences_file, delimiter=',')
for row in reader:
episodes[row[1]].append(row[4])
for episode_id, text in episodes.items():
episodes[episode_id] = "".join(text)
corpus = []
for id, episode in sorted(episodes.items(), key=lambda t: t[0]):
corpus.append(episode)
len(corpus)
209
The corpus contains 209 entries (1 per episode), each of which is a string containing the transcript of that episode.
Next it's time to train our TF/IDF model via TfidfVectorizer in scikit-learn module which is only a few lines of code. The most interesting parameter here is ngram_range - we're telling it to generate 2 and 3 word phrases along with the single words from the corpus.
e.g. if we had the sentence "Python is cool" we'd end up with 6 phrases - 'Python', 'is', 'cool', 'Python is', 'Python is cool' and 'is cool'.
from sklearn.feature_extraction.text import TfidfVectorizer
tf = TfidfVectorizer(analyzer='word', ngram_range=(1,3), min_df = 0, stop_words = 'english')
tfidf_matrix = tf.fit_transform(corpus)
feature_names = tf.get_feature_names()
print(len(feature_names))
feature_names[50:70]
498229
['00 does sound', '00 don', '00 don buy', '00 dressed', '00 dressed blond', '00 drunkenly', '00 drunkenly slurred', '00 fair', '00 fair tonight', '00 fall', '00 fall foliage', '00 far', '00 far impossible', '00 fart', '00 fart sure', '00 friends', '00 friends singing', '00 getting', '00 getting guys', '00 god']
So we're got nearly 500,000 phrases and if we look at tfidf_matrix we'd expect it to be a $209 \times 498229$ matrix - one row per episode, one column per phrase:
tfidf_matrix
<209x498229 sparse matrix of type '<class 'numpy.float64'>' with 740366 stored elements in Compressed Sparse Row format>
This is what we've got although under the covers it's using a sparse representation to save space. Let's convert the matrix to dense format to explore further and find out why:
dense = tfidf_matrix.todense()
len(dense[0].tolist()[0])
498229
The first value in each tuple is the phrase's position in our initial vector and also corresponds to the phrase's position in feature_names which allows us to map the scores back to phrases. Let's automate that lookup:
episode = dense[0].tolist()[0]
phrase_scores = [pair for pair in zip(range(0, len(episode)), episode) if pair[1] > 0]
sorted_phrase_scores = sorted(phrase_scores, key=lambda t: t[1] * -1)
for phrase, score in [(feature_names[word_id], score)
for (word_id, score) in sorted_phrase_scores][:20]:
print('{0: <20} {1}'.format(phrase, score))
ted 0.2633241815084314 olives 0.19556508008332224 marshall 0.1559959550807364 yasmine 0.1521528557736951 robin 0.13081226492372183 barney 0.1247967640645891 lily 0.12330259292855121 signal 0.1037285359479745 goanna 0.09807284390611715 scene 0.09535105429648502 cut 0.0918076740694677 narrator 0.08652535148952963 flashback 0.07836273694979588 flashback date 0.07022439497247467 ranjit 0.06937564856742995 flashback date robin 0.058520329143728886 ted yasmine 0.058520329143728886 carl 0.05819191692744656 eye patch 0.0543236333564784 lebanese 0.0543236333564784
Let's find some interesting phrase in this episode. The (!) sign indicates this is not python command. It envokes a shell command to do this quickly.
! grep -h -rni --color "lebanese" data/ sentences.csv
grep: sentences.csv: No such file or directory
For now we'll extract phrases for all episodes and write to CSV so we can explore more easily:
with open("data/tfidf_scikit.csv", "w") as file:
writer = csv.writer(file, delimiter=",")
writer.writerow(["EpisodeId", "Phrase", "Score"])
doc_id = 0
print("Writing Document:", end = " ")
for doc in tfidf_matrix.todense():
print(doc_id, end = ", ")
word_id = 0
for score in doc.tolist()[0]:
if score > 0:
word = feature_names[word_id]
writer.writerow([doc_id+1, word, score])
word_id +=1
doc_id +=1
Writing Document: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208,
And finally a quick look at the contents of the CSV:
import pandas as pd
da = pd.read_csv('data/tfidf_scikit.csv')
da
| EpisodeId | Phrase | Score | |
|---|---|---|---|
| 0 | 1 | 2005 | 0.007399 |
| 1 | 1 | 2005 seven | 0.010865 |
| 2 | 1 | 2005 seven just | 0.011704 |
| 3 | 1 | 2030 | 0.010004 |
| 4 | 1 | 2030 kids | 0.004960 |
| ... | ... | ... | ... |
| 740361 | 208 | young marshall went | 0.014774 |
| 740362 | 208 | young marshall wow | 0.014774 |
| 740363 | 208 | young real | 0.014774 |
| 740364 | 208 | young real marshall | 0.014774 |
| 740365 | 209 | sentence | 1.000000 |
740366 rows × 3 columns
Related Papers by Tomas Mikolov et al from Google:
from gensim.test.utils import common_texts
common_texts
[['human', 'interface', 'computer'], ['survey', 'user', 'computer', 'system', 'response', 'time'], ['eps', 'user', 'interface', 'system'], ['system', 'human', 'system', 'eps'], ['user', 'response', 'time'], ['trees'], ['graph', 'trees'], ['graph', 'minors', 'trees'], ['graph', 'minors', 'survey']]
from gensim.models import Word2Vec
model = Word2Vec(sentences=common_texts, vector_size=100, window=5, min_count=1, workers=4)
model.save("word2vec.model")
vector = model.wv['computer'] # get numpy vector of a word
vector
array([-0.00515774, -0.00667028, -0.0077791 , 0.00831315, -0.00198292,
-0.00685696, -0.0041556 , 0.00514562, -0.00286997, -0.00375075,
0.0016219 , -0.0027771 , -0.00158482, 0.0010748 , -0.00297881,
0.00852176, 0.00391207, -0.00996176, 0.00626142, -0.00675622,
0.00076966, 0.00440552, -0.00510486, -0.00211128, 0.00809783,
-0.00424503, -0.00763848, 0.00926061, -0.00215612, -0.00472081,
0.00857329, 0.00428458, 0.0043261 , 0.00928722, -0.00845554,
0.00525685, 0.00203994, 0.0041895 , 0.00169839, 0.00446543,
0.00448759, 0.0061063 , -0.00320303, -0.00457706, -0.00042664,
0.00253447, -0.00326412, 0.00605948, 0.00415534, 0.00776685,
0.00257002, 0.00811904, -0.00138761, 0.00808028, 0.0037181 ,
-0.00804967, -0.00393476, -0.0024726 , 0.00489447, -0.00087241,
-0.00283173, 0.00783599, 0.00932561, -0.0016154 , -0.00516075,
-0.00470313, -0.00484746, -0.00960562, 0.00137242, -0.00422615,
0.00252744, 0.00561612, -0.00406709, -0.00959937, 0.00154715,
-0.00670207, 0.0024959 , -0.00378173, 0.00708048, 0.00064041,
0.00356198, -0.00273993, -0.00171105, 0.00765502, 0.00140809,
-0.00585215, -0.00783678, 0.00123304, 0.00645651, 0.00555797,
-0.00897966, 0.00859466, 0.00404815, 0.00747178, 0.00974917,
-0.0072917 , -0.00904259, 0.0058377 , 0.00939395, 0.00350795],
dtype=float32)
sims = model.wv.most_similar('computer', topn=10) # get other similar words
sims
[('system', 0.21617144346237183),
('survey', 0.044689204543828964),
('interface', 0.01520337164402008),
('time', 0.0019510634010657668),
('trees', -0.032843153923749924),
('human', -0.0742427185177803),
('response', -0.09317589551210403),
('graph', -0.09575346857309341),
('eps', -0.10513806343078613),
('user', -0.16911624372005463)]