TF-IDF¶

In information retrieval, tf–idf or TFIDF, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.

It is often used as a weighting factor in searches of information retrieval, text mining, and user modeling. The tf-idf value increases proportionally to the number of times a word appears in the document and is offset by the frequency of the word in the corpus, which helps to adjust for the fact that some words appear more frequently in general.

Tf-idf is one of the most popular term-weighting schemes today; 83% of text-based recommender systems in digital libraries use tf-idf.

Term frequency¶

Suppose we have a set of text documents and wish to rank which document is most relevant to the query, "the brown cow".

A simple way to start out is by eliminating documents that do not contain all three words "the", "brown", and "cow", but this still leaves many documents.
To further distinguish them, we might count the number of times each term occurs in each document; the number of times a term occurs in a document is called its term frequency. However, in the case where the length of documents varies greatly, adjustments are often made (see definition below). The first form of term weighting is due to Hans Peter Luhn (1957) which may be summarized as:
```
  The weight of a term that occurs in a document is simply proportional to the term frequency.
```

In the case of the term frequency $tf(t,d)$ , the simplest choice is to use the raw count of a term in a document, i.e. the number of times that term t occurs in document d. If we denote the raw count by ft,d, then the simplest tf scheme is $tf(t,d) = f_{t,d}$ . Other possibilities include[5]:128

Boolean "frequencies": $tf(t,d) = 1$ if t occurs in d and 0 otherwise;
term frequency adjusted for document length : $f_{t,d} ÷$ (number of words in d)
logarithmically scaled frequency: $tf(t,d) = log (1 + f_{t,d})$
augmented frequency, to prevent a bias towards longer documents, e.g. raw frequency divided by the raw frequency of the most occurring term in the document:
$t f (t, d) = 0.5 + 0.5 \cdot \frac{f_{t, d}}{max {f_{t^{'}, d} : t^{'} \in d}}$ $\mathrm {tf} (t,d)=0.5+0.5\cdot {\frac {f_{t,d}}{\max\{f_{t',d}:t'\in d\}}}$

Inverse document frequency¶

Because the term "the" is so common, term frequency will tend to incorrectly emphasize documents which happen to use the word "the" more frequently, without giving enough weight to the more meaningful terms "brown" and "cow". The term "the" is not a good keyword to distinguish relevant and non-relevant documents and terms, unlike the less-common words "brown" and "cow". Hence an inverse document frequency factor is incorporated which diminishes the weight of terms that occur very frequently in the document set and increases the weight of terms that occur rarely.

Karen Spärck Jones (1972) conceived a statistical interpretation of term specificity called Inverse Document Frequency (IDF), which became a cornerstone of term weighting:

The specificity of a term can be quantified as an inverse function of the number of documents in which it occurs.

The inverse document frequency is a measure of how much information the word provides, that is, whether the term is common or rare across all documents. It is the logarithmically scaled inverse fraction of the documents that contain the word, obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient.

i d f (t, D) = \log \frac{N}{| {d \in D : t \in d} |}

$\mathrm {idf} (t,D)=\log {\frac {N}{|\{d\in D:t\in d\}|}}$ with

$N$ : total number of documents in the corpus $N={|D|}$
$|\{d\in D:t\in d\}|$ : number of documents where the term $t$ appears (i.e., $\mathrm {tf} (t,d)\neq 0$ ). If the term is not in the corpus, this will lead to a division-by-zero. It is therefore common to adjust the denominator to $1+|\{d\in D:t\in d\}|$ .

Term frequency–Inverse document frequency¶

Then tf–idf is calculated as

t f i d f (t, d, D) = t f (t, d) \cdot i d f (t, D)

$\mathrm {tfidf} (t,d,D)=\mathrm {tf} (t,d)\cdot \mathrm {idf} (t,D)$

A high weight in tf–idf is reached by a high term frequency (in the given document) and a low document frequency of the term in the whole collection of documents;
The weights hence tend to filter out common terms.
Since the ratio inside the idf's log function is always greater than or equal to 1, the value of idf (and tf-idf) is greater than or equal to 0.
As a term appears in more documents, the ratio inside the logarithm approaches 1, bringing the idf and tf-idf closer to 0.

Example¶

Let find interesting insights into How I met your mother from its transcripts and one technique that kept coming up is TF/IDF.

Let's first build a corpus.

from collections import defaultdict
import csv

episodes = defaultdict(list)
with open("sentences.csv", "r") as sentences_file:
    reader = csv.reader(sentences_file, delimiter=',')
    for row in reader:
        episodes[row[1]].append(row[4])

for episode_id, text in episodes.items():
    episodes[episode_id] = "".join(text)

corpus = []
for id, episode in sorted(episodes.items(), key=lambda t: t[0]):
    corpus.append(episode)
    
len(corpus)

209

The corpus contains 209 entries (1 per episode), each of which is a string containing the transcript of that episode.

Next it's time to train our TF/IDF model via TfidfVectorizer in scikit-learn module which is only a few lines of code. The most interesting parameter here is ngram_range - we're telling it to generate 2 and 3 word phrases along with the single words from the corpus.

e.g. if we had the sentence "Python is cool" we'd end up with 6 phrases - 'Python', 'is', 'cool', 'Python is', 'Python is cool' and 'is cool'.

from sklearn.feature_extraction.text import TfidfVectorizer
tf = TfidfVectorizer(analyzer='word', ngram_range=(1,3), min_df = 0, stop_words = 'english')

tfidf_matrix =  tf.fit_transform(corpus)
feature_names = tf.get_feature_names() 
print(len(feature_names))
feature_names[50:70]

498229

['00 does sound',
 '00 don',
 '00 don buy',
 '00 dressed',
 '00 dressed blond',
 '00 drunkenly',
 '00 drunkenly slurred',
 '00 fair',
 '00 fair tonight',
 '00 fall',
 '00 fall foliage',
 '00 far',
 '00 far impossible',
 '00 fart',
 '00 fart sure',
 '00 friends',
 '00 friends singing',
 '00 getting',
 '00 getting guys',
 '00 god']

So we're got nearly 500,000 phrases and if we look at tfidf_matrix we'd expect it to be a $209 \times 498229$ matrix - one row per episode, one column per phrase:

tfidf_matrix

<209x498229 sparse matrix of type '<class 'numpy.float64'>'
	with 740366 stored elements in Compressed Sparse Row format>

This is what we've got although under the covers it's using a sparse representation to save space. Let's convert the matrix to dense format to explore further and find out why:

dense = tfidf_matrix.todense()
len(dense[0].tolist()[0])

498229

The first value in each tuple is the phrase's position in our initial vector and also corresponds to the phrase's position in feature_names which allows us to map the scores back to phrases. Let's automate that lookup:

episode = dense[0].tolist()[0]
phrase_scores = [pair for pair in zip(range(0, len(episode)), episode) if pair[1] > 0]
sorted_phrase_scores = sorted(phrase_scores, key=lambda t: t[1] * -1)
for phrase, score in [(feature_names[word_id], score) 
    for (word_id, score) in sorted_phrase_scores][:20]:
        print('{0: <20} {1}'.format(phrase, score))

ted                  0.26332418150842635
olives               0.1955650800833185
marshall             0.1559959550807334
yasmine              0.1521528557736922
robin                0.13081226492371933
barney               0.12479676406458672
lily                 0.12330259292854885
signal               0.10372853594797252
goanna               0.09807284390611527
scene                0.09535105429648319
cut                  0.09180767406946594
narrator             0.08652535148952796
flashback            0.07836273694979438
flashback date       0.07022439497247332
ranjit               0.06937564856742862
flashback date robin 0.05852032914372776
ted yasmine          0.05852032914372776
carl                 0.05819191692744544
eye patch            0.05432363335647736
lebanese             0.05432363335647736

Let's find some interesting phrase in this episode. The (!) sign indicates this is not python command. It envokes a shell command to do this quickly.

!grep -h -rni --color "lebanese" sentences.csv

with open("tfidf_scikit.csv", "w") as file:
    writer = csv.writer(file, delimiter=",")
    writer.writerow(["EpisodeId", "Phrase", "Score"])

    doc_id = 0
    print("Writing Document:", end = " ")
    for doc in tfidf_matrix.todense():
        print(doc_id, end = ", ")
        word_id = 0
        for score in doc.tolist()[0]:
            if score > 0:
                word = feature_names[word_id]
                writer.writerow([doc_id+1, word, score])
            word_id +=1
        doc_id +=1

Writing Document: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208,

And finally a quick look at the contents of the CSV:

!head tfidf_scikit.csv